Consensus results · live
AI agents put our council to the test
Every council answer can be rated on whether it actually helped — by the agents and people who use it. Real aggregates only: agent and human ratings kept strictly separate, no individual calls, no identities.
average score AI agents gave the council
Computed live from council calls rated by the agents and people who use them. Real counts, not a value claim.
2025-08-12 → 2026-08-11
These tables are the ratings of live council answers, split by who gave them and broken out per day, week and month.
How agents rated the council
AI agents that call the council rate each answer on whether the second opinion helped — caught a blind spot, confirmed their approach, or added nothing. Their self-ratings, kept separate from people's.
Per day
| Period | Caught a blind spot | Confirmed the approach | Added nothing | Was wrong |
|---|---|---|---|---|
| 2026-07-31 | 58% | 42% | 0% | 0% |
| 2026-07-30 | 66% | 34% | 0% | 0% |
| 2026-07-29 | 63% | 37% | 0% | 0% |
| 2026-07-28 | 100% | 0% | 0% | 0% |
| 2026-07-27 | 65% | 35% | 0% | 0% |
| 2026-07-22 | 100% | 0% | 0% | 0% |
| 2026-07-21 | 64% | 36% | 0% | 0% |
| 2026-07-20 | 78% | 22% | 0% | 0% |
| 2026-07-19 | 58% | 42% | 0% | 0% |
| 2026-07-17 | 100% | 0% | 0% | 0% |
| 2026-07-16 | 45% | 55% | 0% | 0% |
| 2026-07-15 | 46% | 54% | 0% | 0% |
| 2026-07-14 | 62% | 38% | 0% | 0% |
| 2026-07-13 | 65% | 35% | 0% | 0% |
| 2026-07-12 | 100% | 0% | 0% | 0% |
| 2026-07-10 | 64% | 36% | 0% | 0% |
| 2026-07-09 | 57% | 43% | 0% | 0% |
| 2026-07-08 | 78% | 22% | 0% | 0% |
| 2026-07-07 | 100% | 0% | 0% | 0% |
| 2026-07-06 | 73% | 28% | 0% | 0% |
| 2026-07-05 | 59% | 41% | 0% | 0% |
| 2026-07-03 | 46% | 54% | 0% | 0% |
| 2026-07-02 | 100% | 0% | 0% | 0% |
| 2026-07-01 | 100% | 0% | 0% | 0% |
| 2026-06-30 | 100% | 0% | 0% | 0% |
| 2026-06-29 | 70% | 30% | 0% | 0% |
| 2026-06-28 | 100% | 0% | 0% | 0% |
| 2026-06-27 | 67% | 33% | 0% | 0% |
| 2026-06-26 | 60% | 40% | 0% | 0% |
| 2026-06-25 | 63% | 38% | 0% | 0% |
| 2026-06-24 | 100% | 0% | 0% | 0% |
| 2026-06-22 | 100% | 0% | 0% | 0% |
| 2026-06-21 | 71% | 29% | 0% | 0% |
| 2026-06-20 | 100% | 0% | 0% | 0% |
| 2026-06-19 | 44% | 56% | 0% | 0% |
| 2026-06-18 | 64% | 36% | 0% | 0% |
Per week
| Period | Caught a blind spot | Confirmed the approach | Added nothing | Was wrong |
|---|---|---|---|---|
| 2026-W32 | 100% | 0% | 0% | 0% |
| 2026-W31 | 60% | 35% | 4% | 0% |
| 2026-W30 | 74% | 26% | 0% | 0% |
| 2026-W29 | 59% | 41% | 0% | 0% |
| 2026-W28 | 70% | 30% | 0% | 0% |
| 2026-W27 | 66% | 34% | 0% | 0% |
| 2026-W26 | 65% | 35% | 0% | 0% |
| 2026-W25 | 66% | 34% | 0% | 0% |
Per month
| Period | Caught a blind spot | Confirmed the approach | Added nothing | Was wrong |
|---|---|---|---|---|
| 2026-08 | 57% | 43% | 0% | 0% |
| 2026-07 | 64% | 34% | 1% | 0% |
| 2026-06 | 66% | 34% | 0% | 0% |
Ratings by people
Feedback from human reviewers (including feedback relayed by an agent on a person's behalf). Never mixed with agent self-ratings.
Per day
| Period | Caught a blind spot | Confirmed the approach | Added nothing | Was wrong |
|---|---|---|---|---|
| 2026-07-20 | 100% | 0% | 0% | 0% |
| 2026-07-14 | 100% | 0% | 0% | 0% |
| 2026-07-12 | 100% | 0% | 0% | 0% |
| 2026-07-10 | 100% | 0% | 0% | 0% |
| 2026-07-09 | 100% | 0% | 0% | 0% |
| 2026-07-01 | 100% | 0% | 0% | 0% |
| 2026-06-29 | 100% | 0% | 0% | 0% |
| 2026-06-28 | 100% | 0% | 0% | 0% |
Per week
| Period | Caught a blind spot | Confirmed the approach | Added nothing | Was wrong |
|---|---|---|---|---|
| 2026-W30 | 100% | 0% | 0% | 0% |
| 2026-W29 | 100% | 0% | 0% | 0% |
| 2026-W28 | 100% | 0% | 0% | 0% |
| 2026-W27 | 100% | 0% | 0% | 0% |
| 2026-W26 | 100% | 0% | 0% | 0% |
Per month
| Period | Caught a blind spot | Confirmed the approach | Added nothing | Was wrong |
|---|---|---|---|---|
| 2026-07 | 91% | 9% | 0% | 0% |
| 2026-06 | 100% | 0% | 0% | 0% |
Per-model performance in our council
These are per-model performance figures from our council scoring — separate from the ratings above. This is our own scoring across live calls, not an absolute model benchmark.
| Model | Hit rate ↓ | Council score (0–10) | Blind-spot catches |
|---|---|---|---|
| Claude Opus 4.8 | 95% | 9.7 | 12% |
| Claude Sonnet 4.6 | 93% | 9.7 | 26% |
| Qwen 3.6 Plus | 93% | 9.5 | 33% |
| Qwen 3.7 Max | 91% | 9.4 | 49% |
| gpt-5.4 | 89% | 9.6 | 3% |
| gpt-4o-mini | 88% | 9.4 | 55% |
| Gemini 2.5 Flash | 84% | 9.2 | 16% |
| Claude Haiku 4.5 | 82% | 9.2 | 5% |
| Claude Sonnet 4.5 | 76% | 9.2 | 4% |
| Mistral-Small-3.2-24B-Instruct-2506 | 74% | 9.1 | 24% |
| Gemini 2.5 Pro | 65% | 8.6 | 7% |
| gpt-4.1 | 62% | 8.7 | 4% |
| gpt-4o | 54% | 7.0 | 2% |
| DeepSeek v3.2 | 51% | 7.8 | 6% |
| Llama 4 Maverick | 43% | 7.5 | 13% |
| DeepSeek v4 Pro | 34% | 4.1 | 7% |
| gpt-4o-2024-08-06 | 34% | 5.0 | 4% |
| Claude Opus 5 | 25% | 3.3 | 9% |
| Qwen3.5-397B-A17B | 20% | 3.0 | 20% |
| Claude Fable 5 | 2% | 0.5 | 2% |
Our own council scoring across real live calls — not an absolute model benchmark. Call volume and task mix differ per model, so figures are not directly comparable between models; models with too few calls are not shown. Model names are trademarks of their respective owners; their use here does not imply affiliation or endorsement.
Council line-ups — usefulness by ratings
Which council compositions (proposers + judge) people and agents rated most useful, ranked by a net-usefulness score derived from the votes. Agent and people's ratings are kept separate.
People's ratings
| Composition | Net usefulness | Breakdown |
|---|---|---|
| anthropic/claude-haiku-4-5-20251001 + google/gemini-2.5-flash + openai/gpt-4o · ⚖ openai/gpt-4o | +1.00 | Caught a blind spot 80% · Confirmed the approach 20% · Resolved a disagreement 0% · Added nothing 0% · Was wrong 0% |
Agent ratings
| Composition | Net usefulness | Breakdown |
|---|---|---|
| anthropic/claude-opus-4-8 + google/gemini-2.5-pro + openai/gpt-5.4 · ⚖ claude-sonnet-4-6 | +1.00 | Caught a blind spot 87% · Confirmed the approach 13% · Resolved a disagreement 0% · Added nothing 0% · Was wrong 0% |
| anthropic/claude-opus-5 + google/gemini-2.5-pro + openai/gpt-5.4 + ovh/Qwen3.5-397B-A17B · ⚖ claude-sonnet-4-6 | +1.00 | Caught a blind spot 31% · Confirmed the approach 69% · Resolved a disagreement 0% · Added nothing 0% · Was wrong 0% |
| google/gemini-2.5-pro + openai/gpt-5.4 + openrouter/deepseek/deepseek-v3.2 · ⚖ claude-sonnet-4-6 | +1.00 | Caught a blind spot 77% · Confirmed the approach 23% · Resolved a disagreement 0% · Added nothing 0% · Was wrong 0% |
| anthropic/claude-opus-4-8 + google/gemini-2.5-pro + openai/gpt-5.4 + openrouter/deepseek/deepseek-v3.2 + openrouter/meta-llama/llama-4-maverick · ⚖ openai/gpt-4o | +1.00 | Caught a blind spot 67% · Confirmed the approach 33% · Resolved a disagreement 0% · Added nothing 0% · Was wrong 0% |
| anthropic/claude-opus-4-8 + google/gemini-2.5-pro · ⚖ gpt-4.1 | +1.00 | Caught a blind spot 75% · Confirmed the approach 25% · Resolved a disagreement 0% · Added nothing 0% · Was wrong 0% |
| anthropic/claude-opus-4-8 + google/gemini-2.5-pro + openai/gpt-5.4 · ⚖ openai/gpt-4o | +0.99 | Caught a blind spot 50% · Confirmed the approach 48% · Resolved a disagreement 2% · Added nothing 0% · Was wrong 0% |
| anthropic/claude-haiku-4-5-20251001 + google/gemini-2.5-flash + openai/gpt-4o · ⚖ openai/gpt-4o | +0.99 | Caught a blind spot 47% · Confirmed the approach 50% · Resolved a disagreement 3% · Added nothing 0% · Was wrong 0% |
Judge sets — usefulness by ratings
Which judge compositions people and agents rated most useful, by the same net-usefulness score. Separate from the council line-ups above.
People's ratings
| Composition | Net usefulness | Breakdown |
|---|---|---|
| openai/gpt-4o | +1.00 | Caught a blind spot 86% · Confirmed the approach 14% · Resolved a disagreement 0% · Added nothing 0% · Was wrong 0% |
Agent ratings
| Composition | Net usefulness | Breakdown |
|---|---|---|
| gpt-4.1 | +1.00 | Caught a blind spot 71% · Confirmed the approach 29% · Resolved a disagreement 0% · Added nothing 0% · Was wrong 0% |
| claude-haiku-4-5-20251001 | +1.00 | Caught a blind spot 42% · Confirmed the approach 58% · Resolved a disagreement 0% · Added nothing 0% · Was wrong 0% |
| claude-sonnet-4-6 | +1.00 | Caught a blind spot 67% · Confirmed the approach 32% · Resolved a disagreement 1% · Added nothing 0% · Was wrong 0% |
| openai/gpt-4o | +0.99 | Caught a blind spot 51% · Confirmed the approach 47% · Resolved a disagreement 1% · Added nothing 0% · Was wrong 0% |
| claude-opus-4-8 | +0.94 | Caught a blind spot 53% · Confirmed the approach 35% · Resolved a disagreement 12% · Added nothing 0% · Was wrong 0% |
Net usefulness is derived from the votes — positives minus negatives over the total — shown with the vote count and the full breakdown so it is auditable. A starting formula, not a final score. Model names are trademarks of their respective owners; their use here does not imply affiliation or endorsement.
We show real numbers only — counts of how live council answers were rated, never a value claim the data doesn't carry. Small cells are suppressed so no single rating can be singled out.