How we check

No single test catches everything, so we combine several. All of them only use the provider's normal API through OpenRouter.

Running now

• Top answer. Short prompts with one word answers, asked at temperature 0 every day. We count how often each provider's top answer equals the maker's own endpoint, and compare that with how often the other providers agree. This decides the verdict.
• Answer spread. Many sampled answers compared with the maker. It also reacts to harmless setup differences, so we show it for reference only.
• Word probabilities. Where providers return token probabilities, we compare them with the other providers. We also note providers that list them but return none.
• Pinned routing. Each test is pinned to one provider with fallbacks off, and we check who actually answered.
• Reasoning switch. We ask for no reasoning and check that the provider respects it.

Coming next

• Capability. Known answer maths and code questions.
• Tool calls. Official tool prompts replayed. We check the schema and when tools fire.
• Long context. Recall tests at long input lengths.
• Your traffic. Opt in. A small share of your requests compared with the maker.

Avoiding false alarms

• Two honest providers never match exactly, so each provider is compared with the other providers, not with perfection.
• We correct for testing many providers at once.
• A clear gap puts a provider on watch. It is marked as no match only after 2 strong failures on different days.
• A provider that is offline keeps its last verdict, marked offline, until we can test it again.
• Every result shows the sample size and how far it is from the others.
• Providers can reply on their scoreboard page.

Wording

Most failures are honest bugs like a broken tool parser or a wrong chat template. We say "does not match", never "cheating". A result is not a claim about intent.