Why System One models are a bigger deal than you think

I built a drawing game this month. Eight people draw the same word at once. An AI judge watches every canvas and ends the round the moment it recognizes one drawing. The judge looks at every canvas every 1.5 seconds, so a full room of eight makes up to 320 judgements in a 60-second round.

I didn’t use an LLM for the judge. I used a kind of model that had no name until September.

#The job

Given a 256×256 PNG and a word, say how likely it is that the drawing is that word. That number is the player’s meter. Cross 75% and you win.

You can ask a vision LLM to do this. Attach the image, list the options, and ask for JSON with a label and a probability. Then parse the JSON, hope the label is one you listed, and hope the probability means something.

It doesn’t, and I’ll show you below. When an LLM writes "probability": 0.95, that number came out of the same next-token machinery as the word “cat”. LLMs are overconfident when you ask them to rate themselves, and the number changes with the wording of the prompt.

Here’s what the judge sends instead:

{
  "model": "clef",
  "state": "A quick doodle from a drawing game.",
  "images": ["<png>"],
  "questions": {
    "d0": {
      "type": "choice",
      "instructions": "What is it?",
      "criteria": {
        "cat": "cat",
        "dog": "dog",
        "fox": "fox",
        "rabbit": "rabbit",
        "other": "something else",
        "scribble": "scribbles or blank",
        "writing": "written words or letters"
      }
    }
  }
}

And what comes back:

{
  "answers": {
    "d0": {
      "choice": "cat",
      "probabilities": {
        "cat": 0.63,
        "fox": 0.15,
        "dog": 0.1,
        "rabbit": 0.07,
        "other": 0.03,
        "scribble": 0.01,
        "writing": 0.01
      },
      "confidence": 0.71
    }
  },
  "usage": { "input_tokens": 290, "output_tokens": 0 }
}

No text, nothing to parse, zero output tokens. The meter is probabilities.cat. At $0.24 per million input tokens that call costs $0.00007. Two players over five full rounds is about 400 calls, so a game costs under three cents, and less when rounds end early.

#What it is

The name comes from Kahneman. System 1 is the fast, automatic judgement that sees a face and knows it’s angry. System 2 is the slow reasoning that multiplies 17 by 24. Chain-of-thought LLMs are System 2 machines. Most decisions in software are System 1 questions. Which team gets this ticket? Is this a cat?

TypeSafe AI coined the term when it launched Jev on September 15, 2026. A study of its first week counted 2,170 public GitHub projects built on it, with 43,750 stars between them. Two weeks later Cloudflare shipped Clef with open weights under Apache 2.0, the same API, and image input. Ollama runs Clef, so ollama pull clef gets you one on a laptop.

The API is the idea. You send a state (text, JSON, or up to four images) and a map of typed questions:

One request can carry 64 questions, and the model answers all of them in one pass. Here is a support inbox with four questions on every message. The team is a choice, the refund a noul, the urgency a score, and the tone another choice.

Two things to notice. The “no rush” feature request lands on “no action” at 74% with technical support at 15%, which is the right amount of unsure. And urgency comes back as a number between the levels, so you can sort an inbox by it.

The model can’t pick anything you didn’t list, can’t explain itself, and can’t write a sentence. That is the design.

#How it works

Clef is a frozen Qwen3.8-27B backbone, vision encoder included, with a small transformer head on top. The backbone reads the images, the state, and the questions. The head reads the final hidden states and produces one logit per option. A softmax per question turns those into the probabilities.

The speed comes from one fact. This is a prefill-only pass. A normal LLM reads the prompt, then writes the answer one token per forward pass. Clef reads the prompt and stops. It scores every option of every question at once.

Cloudflare reports a median of 209 ms for Clef and 39 ms for the smaller Clef-flash.

The probabilities are a training target, not a side effect. Cloudflare trains the head with a Brier loss, which penalizes a probability by how far it lands from the truth, then with RLCD, Reinforcement Learning for Calibrated Decisions, the method TypeSafe introduced for Jev. The goal is that when the model says 80%, it is right 80% of the time.

If this sounds like a zero-shot classifier, it is. When someone on Hacker News said so, TypeSafe’s CEO replied “exactly right!” The same idea runs through NLI classifiers, CLIP, and the old trick of reading an LLM’s logprobs for the option letters. What’s new is the package. One general model, a schema you change per request, calibration as a training target, image input, and a price of $0.24 per million input tokens.

#Versus an LLM

The closest an LLM gets is the logprob trick. List the options as A, B, C, D and read the probability of the first output token. It mostly works, with known problems. Models prefer the letter “A” regardless of content, swapping the order changes the answer, and “giraffe” is not one token.

The last row is the cost. On Cloudflare’s own table, Clef scores 48 on GPQA Diamond and 74 on BBH, where Jev reports 78 and 93. One pass means no thinking before answering. It still scores 81% on GSM8K, so a lot of grade-school math is recognition.

#A probability versus a number that looks like one

I ran the game’s test doodles through both kinds of model on my laptop, in Ollama. Clef was the decision model. Gemma 4 12B was the vision LLM, asked for JSON with a label and a probability. Same seven options, same image, temperature 0. Both got all ten animals right.

Then I rendered the cat at 0%, 25%, 50%, 75% and 100% of its strokes, which is what the judge sees during a round.

Clef’s number climbs with the drawing. Gemma’s jumps. The moment it decides “cat”, it writes 0.95, and it wrote 0.9 or 0.95 for every finished doodle in the set. Clef’s ranged from 81% for a snake, which is a wavy line, to 97% for a giraffe, and the lowest scores went to the doodles with the closest look-alike on the list: snake and eel, flamingo and ostrich, spider and crab.

A meter needs a number that moves. A threshold needs a number that means something. Gemma’s does neither.

Locally, Gemma was faster. It took 1.5 seconds per doodle against Clef’s 3.1. Gemma is 12B in Apple’s MLX format and Clef is 27B, so the comparison favours Gemma. In production the judge runs on Workers AI, and the AI Gateway logs for one day of games show a median of 393 ms per drawing and a p95 of 1.3 s, over 102 single-drawing requests.

#Why it matters

It changes what you can afford to ask. The judge checks every canvas every 1.5 seconds because a check costs $0.00007 and returns in under a second. At LLM prices the game is a quiz you grade at the end of the round. Any product where a decision runs on every event, every message, or every frame needs this price.

Decisions become a function call. Agent code asks the same questions on every step. Which model handles this? Is this tool call safe? Is this memory relevant? Today each one is another LLM call and a regex. A decision model makes each one a typed function that returns a probability. “Could this send_email call cause harm?” is a worked example in the Ollama docs. Here is that guard on six tool calls from a coding agent.

Reading a file is a run, wiping the disk and emailing every customer are refusals, and the one in the middle is the interesting case. Deleting node_modules gets a 52% chance of harm and still an 86% “run it”. A harness would set its own threshold there, and that is a one-line change in code rather than a prompt rewrite.

The probability makes thresholds possible. A classifier that is right 95% of the time but can’t tell you which 5% can’t be automated. One with a calibrated probability can. Act above 90, confirm in the middle, escalate below. My meter is that threshold, and 75% is the blaze line.

#Where it breaks

A month of using one, with the numbers.

Wrong inside the schema. 312 of the first 431 candidate words passed on the first run. The model can only pick from your list, but it picks wrong. A nose on its own read as a hook, a screwdriver as a toothbrush, a piano as “written words”. I cut 33 words that never passed with any fair drawing, and replaced them with two new packs, so the 431 words that shipped are not the 431 I started with. Every shipped word passes.

The schema is the work. I changed 40 of my look-alike options before testing started, because each was a broader word for the target. A police car with “car” as an option is the example. The judge picked “car” for a good police car, and it was right. Type safety guarantees the format, not the answer.

Context leaks between items. I batched four drawings per request to save tokens. A drawing judged right after a similar one drifted toward it. An astronaut scored 95% judged first and 12% judged second, behind a snowman. Every drawing now gets its own request.

Prompt shape moves the score. The same doodle scored 47% alone and 89% in a batch. If you change how you ask, re-test everything.

Calibration is a claim. Neither vendor has published a reliability curve. The one independent study I found, on Jev, had to recalibrate against its own labels, which cut calibration error by 3.3×. My raw probabilities didn’t work as a meter either. A clear cat at 63% looked like the judge wasn’t sure, so I sharpen them with a temperature of 0.7 before showing them. That is recalibration by eye. And the confidence field measures how peaked the distribution is, not how often it is right.

The numbers are vendor numbers. TypeSafe’s headline, 194× faster and 445× cheaper, comes from workflows its own team wrote. Independent tests measured 5× to 25× faster, and in one of them Jev missed a planted defect that the LLM caught.

It is not small. 27B parameters, an 18 GB download, 26 GB in memory, and 3 seconds a drawing on an M2 Max. Local is for testing, not for a game.

#Which one to use

Use an LLM to design the schema once. Use a System One model to run it on every request.

If the answer space exists before you ask, and you’ll ask often, use a decision model and keep the probability. If you need prose, code, an explanation, or a chain of steps, use an LLM. Most real systems want both. The LLM decides what to do, and the decision model checks every step.

The game is at doodleblaze.bimsina.dev.