Skip to content

Models

Guide: Use other models.

Model

Bases: ABC

Base class for provider adapters. Subclass it to add a provider.

Implement respond and context_window. Everything else has a working default: arespond and acompact run the sync versions on a worker thread, compact sends a summary request through respond, and mark_cache and count_tokens need no override.

Rules for an adapter:

  • Only return a reply. Do not change the State; the Agent records the reply and its usage.
  • User messages and tool results can hold Image blocks. Convert them to the provider's image format, as Anthropic and OpenAICompatible do, or raise a clear error if the provider cannot read images.
  • Call on_text with each piece of text as it streams in, if it is not None.
  • Put the model that actually answered in Reply.model: the name the provider reported, or self.name if it did not say. A Model that falls back to another model puts the fallback's name there.
  • Use no network and no credentials in __init__. Create the SDK client on the first request, so an Agent can be built without an API key.
  • Let the SDK do retries (retries is passed as its retry count). Wrap only SDK exceptions that finally fail, as RateLimitError, AuthError, ContextTooLongError or ProviderError (raise ... from e). Let every other exception propagate as is.
  • Print nothing. Report what happens outside the reply (a fallback, a retry) with on_event.
  • Send back RawBlocks whose provider matches self.provider unchanged, and drop the others.

The settings below have class-level defaults, so a subclass works without calling super().__init__.

name class-attribute instance-attribute 

name: str = ''

Model name without the provider prefix, e.g. "claude-sonnet-5".

provider class-attribute instance-attribute 

provider: str = ''

Provider id, e.g. "anthropic". RawBlocks with this provider are sent back to the model.

supports class-attribute instance-attribute 

supports: frozenset[str] = frozenset()

Features this adapter supports: "thinking", "cache", "server_compact", "vision", etc.

max_tokens class-attribute instance-attribute 

max_tokens: int | None = None

Maximum output tokens per reply. None uses the adapter's default.

temperature class-attribute instance-attribute 

temperature: float | None = None

Sampling temperature. None leaves it to the provider.

timeout class-attribute instance-attribute 

timeout: float | None = None

Request timeout in seconds. None uses the SDK default.

retries class-attribute instance-attribute 

retries: int = 2

How many times the SDK retries a failed request.

price class-attribute instance-attribute 

price: Price | None = None

Token prices used for usage.cost. Without it, cost is None.

context_window abstractmethod property 

context_window: int

Required. The context window size in tokens. agent.context_used(state) is measured against it.

respond abstractmethod 

respond(request: Request, on_text: OnText | None = None, on_event: OnEvent | None = None) -> Reply

Required. Sends one request and returns the reply.

Parameters:

Name Type Description Default
request Request

The system prompt, messages and tool definitions to send.

 required
on_text OnText | None

Called with each text chunk as it streams in. None when nobody is listening.

 None
on_event OnEvent | None

Called with a ModelEvent for anything that happens outside the reply, such as a fallback to another model. A Model that wraps another Model passes both callbacks through.

 None

Returns:

Type Description
Reply

A Reply whose message is an assistant message with blocks in the order received,
Reply

usage.requests == 1, usage.cost from price (or None), context_tokens set to
Reply

all input tokens (cached or not) plus output tokens, and model set to the model that answered.

Raises:

Type Description
ProviderError

The provider request finally failed. Use a subclass such as RateLimitError when one fits.

count_tokens 

count_tokens(request: Request) -> int

Estimates the size of request in tokens.

A utility for your own code. The framework never calls it, so overriding it does not change agent.context_used(state) or when compact_if_full fires.

compact 

compact(request: Request, instructions: str | None = None, on_event: OnEvent | None = None) -> Reply

Optional. Summarizes the context and returns the summary in reply.text.

The default works with every model: it appends a summary request (COMPACT_PROMPT plus instructions) to the context and calls respond once with tool calls disabled. Override it for providers with server-side compaction.

Parameters:

Name Type Description Default
request Request

The current context.

 required
instructions str | None

What the summary must keep.

 None
on_event OnEvent | None

As in respond.

 None

arespond async 

arespond(request: Request, on_text: OnText | None = None, on_event: OnEvent | None = None) -> Reply

Optional. The async version of respond, used by athink and aask.

The default runs respond on a worker thread and delivers on_text and on_event on the event loop. Cancelling it drops the reply, but the thread keeps running until its next callback. Override it with the provider's async SDK so a cancel also closes the request.

acompact async 

acompact(request: Request, instructions: str | None = None, on_event: OnEvent | None = None) -> Reply

Optional. The async version of compact, used by Agent.acompact.

The default sends the same summary request through arespond. If a subclass overrides only compact, that compact runs on a worker thread instead.

mark_cache 

mark_cache(request: Request) -> Request

Optional. Returns the request with prompt cache markers placed. The default returns it unchanged.

The Agent passes every request through this before respond. Request has no marker field, so adapters that cache usually place markers inside respond while converting to the provider format.

Anthropic 

Anthropic(name: str, *, max_tokens: int = 8192, temperature: float | None = None, timeout: float | None = None, retries: int = 2, price: Price | None = None, thinking: bool | int = False, cache: bool | None = None, context_window: int = 200000, api_key: str | None = None, base_url: str | None = None, supports: Iterable[str] | None = None)

Bases: Model

A Model for the Anthropic Messages API. Uses ANTHROPIC_API_KEY unless api_key= is given.

Building it uses no network and needs no API key; the SDK client is created on the first request.

Parameters:

Name Type Description Default
name str

Model name, e.g. "claude-sonnet-5". A leading "anthropic/" is removed.

 required
max_tokens int

Maximum output tokens per reply.

 8192
temperature float | None

Sampling temperature. Cannot be combined with thinking.

 None
timeout float | None

Request timeout in seconds. None uses the SDK default.

 None
retries int

How many times the SDK retries a failed request.

 2
price Price | None

Token prices for usage.cost.

 None
thinking bool | int

False turns extended thinking off, True turns on adaptive thinking, and an int turns it on with that token budget (at least 1024 and below max_tokens). 0 does not turn it off.

 False
cache bool | None

Prompt caching. None turns it on when supports has "cache".

 None
context_window int

Context window size in tokens.

 200000
api_key str | None

API key. None reads ANTHROPIC_API_KEY.

 None
base_url str | None

API base URL, for a proxy or a compatible gateway.

 None
supports Iterable[str] | None

Features the model supports. Defaults to {"thinking", "cache", "vision"}.

 None

Raises:

Type Description
ValueError

A setting the model does not support, thinking together with temperature, or a thinking budget out of range.
Example 
agent = Agent(model=Anthropic("claude-sonnet-5", thinking=True))

provider class-attribute instance-attribute 

provider = 'anthropic'

Always "anthropic".

name instance-attribute 

name: str = name

Model name sent to the API.

max_tokens instance-attribute 

max_tokens: int | None = max_tokens

Maximum output tokens per reply.

temperature instance-attribute 

temperature: float | None = temperature

Sampling temperature, or None to leave it to the provider.

timeout instance-attribute 

timeout: float | None = timeout

Request timeout in seconds, or None for the SDK default.

retries instance-attribute 

retries: int = retries

How many times the SDK retries a failed request.

price instance-attribute 

price: Price | None = price

Token prices used for usage.cost.

supports instance-attribute 

supports: frozenset[str] = frozenset(supports) if supports is not None else frozenset({'thinking', 'cache', 'vision'})

Features this model supports.

cache instance-attribute 

cache: bool

Whether prompt caching is on.

context_window property 

context_window: int

Context window size in tokens, as passed to context_window=.

respond 

respond(request: Request, on_text: OnText | None = None, on_event: OnEvent | None = None) -> Reply

Sends one request as a stream and returns the reply.

Text chunks go to on_text as they arrive. Thinking and other Anthropic-only blocks are kept as RawBlocks and sent back unchanged on later requests. If the reply was cut off at max_tokens in the middle of a tool call, that call is marked invalid so the tool is not run with partial arguments.

Raises:

Type Description
RateLimitError

The API returned 429 after the SDK's retries.
AuthError

The API key is missing or rejected.
ContextTooLongError

The request is larger than the context window.
ProviderError

Any other API error, or the connection dropped or timed out while the reply was streaming (not retried: the SDK retries only a request whose reply has not started).

arespond async 

arespond(request: Request, on_text: OnText | None = None, on_event: OnEvent | None = None) -> Reply

The async version of respond, with the same reply and errors. Cancelling it closes the stream.

OpenAICompatible 

OpenAICompatible(name: str, *, base_url: str | None = None, api_key: str | None = None, max_tokens: int | None = None, temperature: float | None = None, timeout: float | None = None, retries: int = 2, price: Price | None = None, context_window: int = 128000, supports: Iterable[str] | None = None)

Bases: Model

A Model for servers that speak the OpenAI Chat Completions API: OpenAI, Ollama, vLLM, OpenRouter and others.

Building it uses no network and needs no API key; the SDK client is created on the first request.

Parameters:

Name Type Description Default
name str

Model name as the server knows it, e.g. "llama3:8b".

 required
base_url str | None

Server URL, e.g. "http://localhost:11434/v1". None uses OPENAI_BASE_URL or api.openai.com.

 None
api_key str | None

API key. None reads OPENAI_API_KEY. With a base_url and no key anywhere, a placeholder key is sent, which is what local servers expect.

 None
max_tokens int | None

Maximum output tokens per reply. None leaves it to the server. For OpenAI reasoning models (o1, o3, ...) it is sent as max_completion_tokens.

 None
temperature float | None

Sampling temperature. None leaves it to the server.

 None
timeout float | None

Request timeout in seconds. None uses the SDK default.

 None
retries int

How many times the SDK retries a failed request.

 2
price Price | None

Token prices for usage.cost.

 None
context_window int

Context window size in tokens.

 128000
supports Iterable[str] | None

Features the server supports. Empty by default, since it varies by server.

 None
Example 
model = OpenAICompatible("llama3:8b", base_url="http://localhost:11434/v1")

provider class-attribute instance-attribute 

provider = 'openai_compatible'

Always "openai_compatible".

name instance-attribute 

name: str = name

Model name sent to the API.

max_tokens instance-attribute 

max_tokens: int | None = max_tokens

Maximum output tokens per reply.

temperature instance-attribute 

temperature: float | None = temperature

Sampling temperature, or None to leave it to the provider.

timeout instance-attribute 

timeout: float | None = timeout

Request timeout in seconds, or None for the SDK default.

retries instance-attribute 

retries: int = retries

How many times the SDK retries a failed request.

price instance-attribute 

price: Price | None = price

Token prices used for usage.cost.

supports instance-attribute 

supports: frozenset[str]

Features this model supports.

base_url instance-attribute 

base_url: str | None = base_url

Server URL, or None for the SDK default.

api_key instance-attribute 

api_key: str | None = api_key

The API key passed to api_key=, or None to read OPENAI_API_KEY.

context_window property 

context_window: int

Context window size in tokens, as passed to context_window=.

respond 

respond(request: Request, on_text: OnText | None = None, on_event: OnEvent | None = None) -> Reply

Sends one request as a stream and returns the reply.

Text chunks go to on_text as they arrive. Reasoning text some servers send (reasoning_content) is kept as a RawBlock and sent back on later requests. A tool call with arguments that are not a JSON object, or cut off by the output limit, is marked invalid so the tool is not run.

Raises:

Type Description
RateLimitError

The server returned 429 after the SDK's retries.
AuthError

The API key is missing or rejected.
ContextTooLongError

The request is larger than the context window.
ProviderError

Any other API error.

arespond async 

arespond(request: Request, on_text: OnText | None = None, on_event: OnEvent | None = None) -> Reply

The async version of respond, with the same reply and errors. Cancelling it closes the stream.

resolve_model 

resolve_model(model: Any, **options: Any) -> Model

Turns the value passed to Agent(model=...) into a Model object.

Keyword options go to the adapter it picks: resolve_model("anthropic/claude-sonnet-5", max_tokens=16_000) is Anthropic("claude-sonnet-5", max_tokens=16_000).

  • Model instance → as is. With options → TypeError (set them when creating the Model)
  • base_url= given (not None) → OpenAICompatible(model, **options) with the whole string as the model name, not split: routers such as OpenRouter use names like "anthropic/claude-sonnet-5". ollama/x is the one exception (below). For an Anthropic-format proxy, create Anthropic(..., base_url=...) directly.
  • "provider/model" (split on the first /; : is an Ollama tag, not a separator):
    • anthropic/x → Anthropic("x", **options)
    • openai/x → OpenAICompatible("x", **options) (base_url=None, i.e. the default OpenAI URL. An extension point: switches to a dedicated OpenAI adapter once one exists)
    • ollama/x → OpenAICompatible("x", base_url=OLLAMA_BASE_URL, **options); an explicit base_url wins (an Ollama server on another machine)
    • Any other prefix → ValueError: unknown provider, the known list (anthropic, openai, ollama), and the fix OpenAICompatible("model", base_url="...")
    • ValueError if the model part is empty
  • Names without a prefix: starting with claude- → Anthropic, starting with gpt- → same as openai/. Otherwise ValueError: the provider is ambiguous; shows the candidates "anthropic/{s}", "openai/{s}", "ollama/{s}".
  • Empty string → ValueError. Neither a string nor a Model → TypeError.
  • An option the chosen adapter does not accept → TypeError naming the adapter and the options it accepts. Only the names are checked here; the adapter checks the values.

No network or API key is needed at construction (adapter rule).