AutoGen Plugin: LLM

Overview

FlotorchAutogenLLM implements AutoGen’s ChatCompletionClient backed by FloTorch LLM. It supports tools, optional structured JSON output, and usage accounting.

---

Setup

from flotorch.autogen.llm import FlotorchAutogenLLM

API_KEY = "<your_api_key>"
BASE_URL = "https://gateway.flotorch.cloud"
MODEL_ID = "<your_flotorch_model_id>"

llm = FlotorchAutogenLLM(model_id=MODEL_ID, api_key=API_KEY, base_url=BASE_URL)

---

API

FlotorchAutogenLLM

Constructor:

FlotorchAutogenLLM(
  model_id: str,
  api_key: str,
  base_url: str,
)

create(messages, tools=None, json_output=None, **kwargs) -> CreateResult

• Converts AutoGen messages and tools to FloTorch format
• Enables structured output when json_output is provided and tools are absent or tool results are present
• Returns CreateResult with content (string or function calls), finish_reason, and usage

create_stream(messages, **kwargs) -> AsyncGenerator[str | CreateResult]

Streams the text content followed by the final CreateResult.

---

Notes

• Uses FloTorch Gateway /api/openai/v1/chat/completions
• Pass the same MODEL_ID you configured in FloTorch Model Registry