Speed analysis
Latency measured across all benchmark runs. P50 (median) and P95 (95th percentile) give a realistic picture of response speed under normal and peak load.
Quality scores
How this model compares to the rest of the field on each prompt category, from a pairwise fit over the same prompts. The raw judge score sits underneath each number.
Win rate per category: how often this model beats a field-average model on a prompt from that category. 50% is average, not a failing grade. It is not a percentage of correct answers.
Pricing
What you pay per million tokens when you use this model on Tokonomix, plus an estimate for a typical conversation.
Tokens per second
Throughput in tokens per second, derived from measured P50 latency. Higher is better; fluctuations track provider-side load.
Estimated from P50 latency × 200 output tokens — the absolute number depends on this assumption; the trend is what matters.
Capabilities
Availability
Availability
No measurements yet
We haven't recorded enough API calls to show availability stats for this model. Data appears once the model starts receiving live traffic.
Tokonomix benchmark verdicts
Quality drops 33 points to 49.5 as latency more than doubles
Mistral-Nemo-Instruct-2407 experienced a significant performance regression in this benchmark window, with overall quality declining from 82.6 to 49.5 points. The model's latency deteriorated substantially, with p50 response times increasing from 3969ms to 8587ms, representing a 116% slowdown. Category performance shows mixed results with a dramatic shift in capability distribution. Coding performance declined from 95 to 69 points, though it remains the model's strongest area. Reasoning performance dropped from previous levels to 66 points. Most notably, factual accuracy collapsed to just 14 points, indicating serious issues with knowledge retrieval and accuracy. The multilingual and creative categories, which showed strong scores of 100 and 53 respectively in the previous window, were not measured in this evaluation period. These combined regressions in both quality and speed suggest potential infrastructure issues or model serving problems affecting the OVH AI Endpoints deployment in the GRA region. Users should exercise caution with factual queries and expect significantly slower response times until these issues are investigated and resolved.
Quality
49.5
Latency p50
8,587 ms
Test runs
5