> ## Documentation Index
> Fetch the complete documentation index at: https://docs.guild.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLMs

> Make language model calls from inside your agent.

Every agent has access to `task.llm` for making language model calls. Guild resolves the provider and model from the workspace owner's LLM settings — your agent code doesn't need to specify them.

<Note>
  LLM usage is subject to per-execution budgets: request payload size, call count, and token count. Exceeding a budget fails the call with a `429 Too Many Requests` error. See [Execution limits](/reference/limits#llm-execution-budgets).
</Note>

## Basic usage

```typescript theme={null}
const result = await task.llm.generateText({
  prompt: "Summarize this text...",
})

// result.text contains the model's response
console.log(result.text)
```

## Structured generation

Pass a Zod schema to get typed, validated output:

```typescript theme={null}
import { z } from "zod"

const result = await task.llm.generateText({
  prompt: "Extract the key details from this issue...",
  schema: z.object({
    severity: z.enum(["low", "medium", "high"]),
    summary: z.string(),
    affectedFiles: z.array(z.string()),
  }),
})

// result is typed according to your schema
console.log(result.severity) // "high"
```

## Best practices

* **Cache results.** Store the return value of `generateText()` in a variable if you need it more than once. Each call costs tokens.
* **Be specific in prompts.** Clear, detailed prompts produce better results and reduce the need for follow-up calls.
* **Use schemas for structured data.** When you need specific fields, pass a schema rather than parsing free-form text.
* **Keep prompts focused.** One clear task per call is better than a complex multi-part prompt.

## LLM preferences

Use `llmPreferences` to express an ordered list of LLM provider/model preferences for a `generateText` call. Guild honors this list only when the account brings its own keys (BYOK) and server-side model selection is enabled; otherwise the server ignores it and selects the model from policy.

```typescript theme={null}
const result = await task.llm.generateText({
  prompt: "Summarize this text...",
  llmPreferences: [
    { provider: "anthropic", model: "claude-opus-4-5" },
    { provider: "openai" },
  ],
})
```

Earlier entries take priority over later ones. Omit `model` to let the server choose a model from the matching policy.

Each entry is an `LLMPreference` object:

| Field      | Type                                            | Required | Description                                                                                    |
| ---------- | ----------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------- |
| `provider` | `"anthropic" \| "openai" \| "gemini" \| "meta"` | Yes      | The LLM provider to prefer.                                                                    |
| `model`    | `string`                                        | No       | The preferred model for this provider. Omit to let the server choose from the matching policy. |

`llmAgent` also accepts `llmPreferences` with the same semantics. See [LLM agents](/guide/llm-agents#llm-preferences).

## Configuration

The provider and model are resolved at runtime from the **workspace owner's** [LLM settings](/platform/llm-settings), not in agent code. For a workspace owned by your user account, use **Settings > LLM Settings**. For an organization workspace, organization admins configure **Settings > LLM Settings** on the organization.

This means:

* Your agent code does not include API keys or provider names
* Changing LLM settings in the console updates behavior without redeploying agents
* Workspaces with different owners can use different LLM configuration with the same agent code

## Unified LLM proxy

When your agent code already targets the OpenAI API — for example, code running inside a Docker environment managed with `task.env` — you can point it at the runtime's built-in LLM proxy instead of a provider directly. Guild routes each request to the provider configured in the workspace's LLM settings and translates the request and response as needed. Your code targets one interface regardless of the active provider.

The proxy endpoint accepts the OpenAI chat completions format and is available inside the agent task environment at the path `/runtime/services/llm/chat/completions`.

```python theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="<runtime-base-url>/runtime/services/llm",
    api_key="unused",
)

completion = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarize this text..."}],
)
print(completion.choices[0].message.content)
```

### Provider support

| Provider  | Streaming     | Notes                                                                          |
| --------- | ------------- | ------------------------------------------------------------------------------ |
| OpenAI    | Supported     | Passthrough — requests are forwarded without modification, including streaming |
| Anthropic | Supported     | SSE is translated from Anthropic's event format to the OpenAI format           |
| Gemini    | Not supported | Streaming requests return a `501 invalid_request_error`                        |

### Developer role

OpenAI's `developer` role messages are automatically folded into the `system` prompt. You do not need to adjust your message structure when switching providers.

### Error handling

Errors are returned as standard OpenAI error shapes:

| Condition               | Error type         |
| ----------------------- | ------------------ |
| Quota exceeded          | `rate_limit_error` |
| Upstream provider error | `upstream_error`   |