Mistral-Nemo-Instruct-2407

Mistral-Nemo quality plummets 38 points to 46.8, latency up 43%

Mistral-Nemo-Instruct-2407 on OVH AI Endpoints has experienced a severe performance degradation in the current benchmark window. Overall quality dropped dramatically from 84.9 to 46.8, representing a 38.1 point decline that affects nearly all measured capabilities. The multilingual category saw the most significant collapse, falling from 97 to just 26. Creative performance dropped from 75 to 58, while the model now scores 50 in factual tasks and 53 in reasoning. These new categories replace the previously measured coding capability, which scored 83 in the last window. Latency has also deteriorated substantially, with p50 response times increasing 43% from 3051ms to 4372ms. This combination of quality collapse and slower response times suggests either a model version change, infrastructure issues, or configuration problems at the provider level. The stability between benchmark windows has clearly been compromised. Users should exercise caution and potentially consider alternative providers or models until performance stabilizes and returns to previously demonstrated levels.