> ## Documentation Index
> Fetch the complete documentation index at: https://docs.superdoc.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM eval results

> Which LLMs work best with SuperDoc's Document API: tested with Promptfoo across 7 models and 360+ tools

SuperDoc's Document API exposes [360+ operations](https://docs.superdoc.dev/document-api/available-operations.md) as LLM tools. Six models were tested at Level 1, with Gemini 2.5 Pro added for Level 2 (seven total).

> [!TIP]
>
> **Recommended models**
>
> * **Best overall accuracy: GPT-5.4 (OpenAI):** 100% tool quality score with the full system prompt. Use this for batch processing, async pipelines, and any case where speed doesn't matter.
> * **Best for fast responses: Gemini 2.5 Flash (Google):** 94% tool quality score at lower latency. Use this for real-time, interactive editing where speed matters.
>
> Claude Haiku 4.5 (Anthropic) is a strong alternative for fast use cases: 90% accuracy with the smallest prompt dependency gap (-13 points), meaning it degrades least when the system prompt is trimmed.

## Tool quality results

Level 1 tests whether a model produces valid tool calls with correct arguments. No documents are executed: only the decision is evaluated. Each of the 31 tests runs twice per model: once with the full SuperDoc system prompt, once with a minimal two-line prompt.

| Model | Full prompt | Minimal prompt | Gap |
| - | - | - | - |
| **GPT-5.4** (OpenAI) | **100%** (31/31) | 74% (23/31) | -26 |
| GPT-4o (OpenAI) | 97% (30/31) | 42% (13/31) | -55 |
| Amazon Nova 2.0-Lite (AWS) | 94% (29/31) | 74% (23/31) | -20 |
| Gemini 2.5 Flash (Google) | 94% (29/31) | 61% (19/31) | -33 |
| Claude Haiku 4.5 (Anthropic) | 90% (28/31) | 77% (24/31) | -13 |
| GPT-4.1-mini (OpenAI) | 90% (28/31) | 61% (19/31) | -29 |

**Full prompt** = the complete SuperDoc system prompt with tool categories, workflow guidance, and error patterns.
**Minimal prompt** = tool schemas only, no behavioral instructions.
**Gap** = the accuracy drop when switching from full to minimal prompt.

## End-to-end results

Level 2 runs 21 tests against real `.docx` fixtures. Each test executes a full agent loop: open the document, let the LLM pick tools, execute them via the CLI, and verify the output file changed correctly.

Three models were tested end-to-end: **GPT-5.4**, **Claude Haiku 4.5**, and **Gemini 2.5 Pro**.

Tests span 10 categories:

* **Reading**: text search, node search, full extraction, counting
* **Mutation**: replace, delete, multi-step atomic edits, first-occurrence targeting
* **Formatting**: bold, highlight, underline via `format.apply`
* **Structure**: headings, paragraphs via `create` group
* **Tables**: table discovery and creation
* **Comments**: comment insertion and discovery
* **Tracked changes**: `tracked` vs. `direct` change modes
* **Lists**: list discovery and creation
* **Hygiene / efficiency**: no hallucinated parameters, undo usage, call count limits
* **Aspirational**: page margins, TOC, cell merging, hyperlinks, liability analysis

> [!NOTE]
>
> Per-category breakdowns by model will be published as the eval suite matures. The current results validate that all three end-to-end models handle the core categories (reading, mutation, formatting) reliably.

## The prompt gap

The "gap" column measures how much a model depends on detailed prompt engineering. A smaller gap means the model interprets SuperDoc's tool definitions well on its own.

* The full system prompt lifts every model by **13–55 percentage points**.
* **GPT-5.4** is the only model that reaches 100% with the full prompt.
* **Claude Haiku 4.5** has the smallest gap (-13 points): it degrades least when the prompt is sparse. This makes it the most robust choice if you need to trim the system prompt for token budget reasons.
* **GPT-4o** is the most prompt-dependent (55-point gap). It scores 97% with the full prompt but drops to 42% without it.

**Takeaway:** always include the full system prompt. Load it from the SDK or copy it from the [LLM tools reference](https://docs.superdoc.dev/ai/agents/llm-tools.md).

> [!TIP]
>
> If you strip down the system prompt to save tokens, expect accuracy to drop. How much depends on the model: Claude Haiku 4.5 loses the least, GPT-4o loses the most.

## Methodology

Results come from a [Promptfoo](https://promptfoo.dev)-based eval framework running against SuperDoc's grouped intent tool set: 9 meta-tools that cover all [360+ underlying operations](https://docs.superdoc.dev/document-api/available-operations.md).

* **Level 1 (tool quality):** 31 synthetic test cases × 2 prompts × 6 models = 372 assertions. Gemini 2.5 Pro was tested only in Level 2. A test passes when the model produces a structurally valid tool call with correct arguments. Assertion types include `tool-call-f1` (set comparison), custom JavaScript checks, and latency thresholds.
* **Level 2 (end-to-end):** 21 tests across 3 real `.docx` fixture documents. Each test runs a full agent loop and checks the resulting document content with `contains` / `not-contains` assertions.
* **Models tested:** GPT-5.4, GPT-4o, GPT-4.1-mini (OpenAI); Claude Haiku 4.5 (Anthropic); Gemini 2.5 Flash, Gemini 2.5 Pro (Google); Amazon Nova 2.0-Lite (AWS).
* **Configuration:** temperature 0, seed 42, tool\_choice required, 30s timeout (Level 1) / 120s timeout (Level 2).
* **Eval date:** March 2026.

> [!WARNING]
>
> Model behavior changes over time. These results are a point-in-time snapshot. Re-run the eval suite after model version updates to verify accuracy.

## Related

> [card] [LLM tools](https://docs.superdoc.dev/ai/agents/llm-tools.md)
>
> Tool definitions, dispatch, and the system prompt reference

> [card] [Available operations](https://docs.superdoc.dev/document-api/available-operations.md)
>
> All Document API operations by namespace

> [card] [Skills](https://docs.superdoc.dev/ai/agents/skills.md)
>
> Reusable prompt templates for common document editing tasks

> [card] [Integrations](https://docs.superdoc.dev/ai/agents/integrations.md)
>
> Connect to AWS Bedrock, Vercel AI SDK, and other providers


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.