This page explains Curator Benchmarks for the person who designs the process — analyst, coordinator, business owner. You do not need to know how to program. Whenever a screen name appears, it is explained right there, with an everyday analogy. The animations play on their own: hover to pause and click to watch again.
Think of an exam given to several candidates. The exam is the same. The questions are the same. At the end, you compare who got more right, who cost less, and who finished faster.
In Curator, the “exam” is an interpreter — the artificial intelligence recipe that reads an official letter, extracts a field, or classifies a document. Each candidate is an artificial intelligence model (GPT, DeepSeek, Kimi…). Each question is one of your real cases, with the correct answer already known — the reference answer (answer key).
The practical question Benchmark answers is: which artificial intelligence model should I use in this recipe? Instead of choosing by opinion or hype, you measure on your own document base.
After the exam, Curator shows three numbers side by side for each model:
| Score | In plain language |
|---|---|
| Accuracy | What percentage of answers matched the answer key you registered. |
| Cost | How much that model would spend, in dollars, to process that volume. It is an estimate for comparison — it does not replace the provider’s invoice. |
| Speed | How long the model took to process everything. |
At the end, Curator builds a ranking. Quality first, savings second: whoever is more accurate goes up; if they tie, the cheaper one goes up; if they still tie, the faster one goes up.
Use Benchmarks when the model choice has money or quality at stake:
These names appear in menus and reports. Worth knowing before you start:
| Name in Curator | In plain language |
|---|---|
| Interpreter | The recipe (the exam) to be evaluated, made of chained steps. |
| Step | One stage of the recipe, with its own instruction and its own answer. |
| Batch | The set of exam cases, with the answer key for each one. |
| Case | One real question — a letter, a contract, a specific document. |
| Reference answer | The correct answer expected at that step of that case. |
| Obtained | The answer the artificial intelligence actually returned. |
| Job | One benchmark run: date, models used, and result. |
| Accuracy | The percentage of correct answers. |
The batch is the folder with the exam questions. Each case brings two things: the inputs (the letter text, the PDF, what the interpreter needs to read) and the answer key (the correct answer at each step).
The same batch can be reused with different models or on different dates. That is what lets you compare results over time.
| Size | What it is good for |
|---|---|
| Fewer than 10 | Only to test whether the recipe works — not to decide on a model. |
| 20 to 50 | Usually enough to compare models on classification or extraction with reasonable confidence. |
| More than 100 | More confidence, but also more time and cost — each case runs once per model. |
The safest path is the button that generates the Excel template. Curator already creates the right columns for that interpreter: case name, one column per input, and one answer-key column per step (named after the step).
After you choose the file, the mapping screen links each spreadsheet column to the right field. Suggestions come pre-filled; you only adjust what differs. At the end, Curator shows how many cases entered and how many have an answer key.
Before starting, three conditions: the interpreter must have at least one step; the batch must have at least one case; each chosen model must have its access key configured.
All models get the same exam: same cases, same order, same instructions. One important detail: during the benchmark, the model under test runs the whole recipe, even if in production each step uses a different model. That is how you answer “what if I used only this model”.
Models run in parallel — one does not wait for the other. Inside each model, cases run one at a time. Inside each case, steps run in a chain.
The screen shows which case and which step each model is on. You can leave: the run continues in the background. Pause and stop apply to the selected model, not the whole run — useful when one model is too expensive or too slow.
A quick cost check: calls ≈ cases × steps × models. Example: 50 cases × 3 steps × 4 models = 600 calls.
Accuracy is: correct steps ÷ steps with an answer key × 100. Steps without a key stay out of the denominator. On screen you see something like “90% (135/150 steps)”.
Automatic grading ignores capitals and extra spaces. What it does not do: judge whether two texts mean the same thing. “TJSP” and “Court of Justice of São Paulo” are different texts — and automatic grading marks them as wrong.
Ranking order: (1) highest accuracy; (2) lowest cost; (3) shortest time. Models that failed go to the bottom. Charts show accuracy, cost, and speed per model — taller bar is better on the accuracy chart; shorter bar is better on cost and time.
If the batch was created with manual review, the run ends in “Awaiting review”. Accuracy is only defined after you mark each step as right or wrong.
The screen shows the expected answer, the obtained answer, and the case input. You mark it and, if you want, note the reason — useful for diagnosing later. The human mark prevails. Even in manual batches, automatic grading can pre-mark; you confirm or correct.
The Excel from the run has three sheets:
History keeps recent runs. For long-term audit, export the Excel. The number alone does not decide: use the ranking to organize, the Excel to understand the errors, and the cost of an error in your process to choose the production model.
