Models

Guide: Models.

Model

Bases: ABC

Base class for provider adapters. Subclass it to add a provider.

Implement respond and context_window. Everything else has a working default: arespond and acompact run the sync versions on a worker thread, compact sends a summary request through respond, and mark_cache and count_tokens need no override.

Rules for an adapter:

• Only return a reply. Do not change the State; the Agent records the reply and its usage.
• Use no network and no credentials in __init__. Create the SDK client on the first request, so an Agent can be built without an API key.
• Let the SDK do retries (retries is passed as its retry count). Wrap only SDK exceptions that finally fail, as RateLimitError, AuthError, ContextTooLongError or ProviderError (raise ... from e). Let every other exception propagate as is.
• Print nothing. Report what happens outside the reply (a fallback, a retry) with on_event.
• Send back RawBlocks whose provider matches self.provider unchanged, and drop the others.

The settings below have class-level defaults, so a subclass works without calling super().__init__.

name class-attribute instance-attribute

name: str = ''

Model name without the provider prefix, e.g. "claude-sonnet-5".

provider class-attribute instance-attribute

provider: str = ''

Provider id, e.g. "anthropic". RawBlocks with this provider are sent back to the model.

supports class-attribute instance-attribute

supports: frozenset[str] = frozenset()

Features this adapter supports: "thinking", "cache", "server_compact", "vision", etc.

max_tokens class-attribute instance-attribute

max_tokens: int | None = None

Maximum output tokens per reply. None uses the adapter's default.

temperature class-attribute instance-attribute

temperature: float | None = None

Sampling temperature. None leaves it to the provider.

timeout class-attribute instance-attribute

timeout: float | None = None

Request timeout in seconds. None uses the SDK default.

retries class-attribute instance-attribute

retries: int = 2

How many times the SDK retries a failed request.

price class-attribute instance-attribute

price: Price | None = None

Token prices used for usage.cost. Without it, cost is None.

context_window abstractmethod property

context_window: int

Required. The context window size in tokens. state.context_used is measured against it.

respond abstractmethod

respond(request: Request, on_text: OnText | None = None, on_event: OnEvent | None = None) -> Reply

Required. Sends one request and returns the reply.

Parameters:

Name	Type	Description	Default
request	Request	The system prompt, messages and tool definitions to send.	required
on_text	OnText | None	Called with each text chunk as it streams in. None when nobody is listening.	None
on_event	OnEvent | None	Called with a ModelEvent for anything that happens outside the reply, such as a fallback to another model. A Model that wraps another Model passes both callbacks through.	None

Returns:

Type	Description
Reply	A Reply whose message is an assistant message with blocks in the order received,
Reply	usage.requests == 1, usage.cost from price (or None), and context_tokens set to
Reply	all input tokens (cached or not) plus output tokens.

Raises:

Type	Description
ProviderError	The provider request finally failed. Use a subclass such as RateLimitError when one fits.

count_tokens

count_tokens(request: Request) -> int

Estimates the size of request in tokens.

A utility for your own code. The framework never calls it, so overriding it does not change state.context_used or when compact_if_full fires.

compact

compact(request: Request, instructions: str | None = None, on_event: OnEvent | None = None) -> Reply

Optional. Summarizes the context and returns the summary in reply.text.

The default works with every model: it appends a summary request (COMPACT_PROMPT plus instructions) to the context and calls respond once with tool calls disabled. Override it for providers with server-side compaction.

Parameters:

Name	Type	Description	Default
request	Request	The current context.	required
instructions	str | None	What the summary must keep.	None
on_event	OnEvent | None	As in respond.	None

arespond async

arespond(request: Request, on_text: OnText | None = None, on_event: OnEvent | None = None) -> Reply

Optional. The async version of respond, used by athink and aask.

The default runs respond on a worker thread and delivers on_text and on_event on the event loop. Cancelling it drops the reply, but the thread keeps running until its next callback. Override it with the provider's async SDK so a cancel also closes the request.

acompact async

acompact(request: Request, instructions: str | None = None, on_event: OnEvent | None = None) -> Reply

Optional. The async version of compact, used by Agent.acompact.

The default sends the same summary request through arespond. If a subclass overrides only compact, that compact runs on a worker thread instead.

mark_cache

mark_cache(request: Request) -> Request

Optional. Returns the request with prompt cache markers placed. The default returns it unchanged.

The Agent passes every request through this before respond. Request has no marker field, so adapters that cache usually place markers inside respond while converting to the provider format.

Anthropic

Anthropic(name: str, *, max_tokens: int = 8192, temperature: float | None = None, timeout: float | None = None, retries: int = 2, price: Price | None = None, thinking: bool | int = False, cache: bool | None = None, context_window: int = 200000, api_key: str | None = None, base_url: str | None = None, supports: Iterable[str] | None = None)

Bases: Model

A Model for the Anthropic Messages API. Uses ANTHROPIC_API_KEY unless api_key= is given.

Building it uses no network and needs no API key; the SDK client is created on the first request.

Parameters:

Name	Type	Description	Default
name	str	Model name, e.g. "claude-sonnet-5". A leading "anthropic/" is removed.	required
max_tokens	int	Maximum output tokens per reply.	8192
temperature	float | None	Sampling temperature. Cannot be combined with thinking.	None
timeout	float | None	Request timeout in seconds. None uses the SDK default.	None
retries	int	How many times the SDK retries a failed request.	2
price	Price | None	Token prices for usage.cost.	None
thinking	bool | int	False turns extended thinking off, True turns on adaptive thinking, and an int turns it on with that token budget (at least 1024 and below max_tokens). 0 does not turn it off.	False
cache	bool | None	Prompt caching. None turns it on when supports has "cache".	None
context_window	int	Context window size in tokens.	200000
api_key	str | None	API key. None reads ANTHROPIC_API_KEY.	None
base_url	str | None	API base URL, for a proxy or a compatible gateway.	None
supports	Iterable[str] | None	Features the model supports. Defaults to {"thinking", "cache", "vision"}.	None

Raises:

Type	Description
ValueError	A setting the model does not support, thinking together with temperature, or a thinking budget out of range.

Example

agent = Agent(model=Anthropic("claude-sonnet-5", thinking=True))

provider class-attribute instance-attribute

provider = 'anthropic'

Always "anthropic".

name instance-attribute

name: str = name

Model name sent to the API.

max_tokens instance-attribute

max_tokens: int | None = max_tokens

Maximum output tokens per reply.

temperature instance-attribute

temperature: float | None = temperature

Sampling temperature, or None to leave it to the provider.

timeout instance-attribute

timeout: float | None = timeout

Request timeout in seconds, or None for the SDK default.

retries instance-attribute

retries: int = retries

How many times the SDK retries a failed request.

price instance-attribute

price: Price | None = price

Token prices used for usage.cost.

supports instance-attribute

supports: frozenset[str] = frozenset(supports) if supports is not None else frozenset({'thinking', 'cache', 'vision'})

Features this model supports.

cache instance-attribute

cache: bool

Whether prompt caching is on.

context_window property

context_window: int

Context window size in tokens, as passed to context_window=.

respond

respond(request: Request, on_text: OnText | None = None, on_event: OnEvent | None = None) -> Reply

Sends one request as a stream and returns the reply.

Text chunks go to on_text as they arrive. Thinking and other Anthropic-only blocks are kept as RawBlocks and sent back unchanged on later requests. If the reply was cut off at max_tokens in the middle of a tool call, that call is marked invalid so the tool is not run with partial arguments.

Raises:

Type	Description
RateLimitError	The API returned 429 after the SDK's retries.
AuthError	The API key is missing or rejected.
ContextTooLongError	The request is larger than the context window.
ProviderError	Any other API error.

arespond async

arespond(request: Request, on_text: OnText | None = None, on_event: OnEvent | None = None) -> Reply

The async version of respond, with the same reply and errors. Cancelling it closes the stream.

OpenAICompatible

OpenAICompatible(name: str, *, base_url: str | None = None, api_key: str | None = None, max_tokens: int | None = None, temperature: float | None = None, timeout: float | None = None, retries: int = 2, price: Price | None = None, context_window: int = 128000, supports: Iterable[str] | None = None)

Bases: Model

A Model for servers that speak the OpenAI Chat Completions API: OpenAI, Ollama, vLLM, OpenRouter and others.

Building it uses no network and needs no API key; the SDK client is created on the first request.

Parameters:

Name	Type	Description	Default
name	str	Model name as the server knows it, e.g. "llama3:8b".	required
base_url	str | None	Server URL, e.g. "http://localhost:11434/v1". None uses OPENAI_BASE_URL or api.openai.com.	None
api_key	str | None	API key. None reads OPENAI_API_KEY. With a base_url and no key anywhere, a placeholder key is sent, which is what local servers expect.	None
max_tokens	int | None	Maximum output tokens per reply. None leaves it to the server. For OpenAI reasoning models (o1, o3, ...) it is sent as max_completion_tokens.	None
temperature	float | None	Sampling temperature. None leaves it to the server.	None
timeout	float | None	Request timeout in seconds. None uses the SDK default.	None
retries	int	How many times the SDK retries a failed request.	2
price	Price | None	Token prices for usage.cost.	None
context_window	int	Context window size in tokens.	128000
supports	Iterable[str] | None	Features the server supports. Empty by default, since it varies by server.	None

Example

model = OpenAICompatible("llama3:8b", base_url="http://localhost:11434/v1")

provider class-attribute instance-attribute

provider = 'openai_compatible'

Always "openai_compatible".

name instance-attribute

name: str = name

Model name sent to the API.

max_tokens instance-attribute

max_tokens: int | None = max_tokens

Maximum output tokens per reply.

temperature instance-attribute

temperature: float | None = temperature

Sampling temperature, or None to leave it to the provider.

timeout instance-attribute

timeout: float | None = timeout

Request timeout in seconds, or None for the SDK default.

retries instance-attribute

retries: int = retries

How many times the SDK retries a failed request.

price instance-attribute

price: Price | None = price

Token prices used for usage.cost.

supports instance-attribute

supports: frozenset[str]

Features this model supports.

base_url instance-attribute

base_url: str | None = base_url

Server URL, or None for the SDK default.

api_key instance-attribute

api_key: str | None = api_key

The API key passed to api_key=, or None to read OPENAI_API_KEY.

context_window property

context_window: int

Context window size in tokens, as passed to context_window=.

respond

respond(request: Request, on_text: OnText | None = None, on_event: OnEvent | None = None) -> Reply

Sends one request as a stream and returns the reply.

Text chunks go to on_text as they arrive. Reasoning text some servers send (reasoning_content) is kept as a RawBlock and sent back on later requests. A tool call with arguments that are not a JSON object, or cut off by the output limit, is marked invalid so the tool is not run.

Raises:

Type	Description
RateLimitError	The server returned 429 after the SDK's retries.
AuthError	The API key is missing or rejected.
ContextTooLongError	The request is larger than the context window.
ProviderError	Any other API error.

arespond async

arespond(request: Request, on_text: OnText | None = None, on_event: OnEvent | None = None) -> Reply

The async version of respond, with the same reply and errors. Cancelling it closes the stream.