📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

Italy’s Minerva-3B, a large language model trained from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, underscoring the significant scale and investment challenges in developing effective national language models.The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research and supercomputing infrastructure, trained models ranging from 350 million to 7 billion parameters using a dataset of 2.5 trillion tokens, half of which was Italian. Despite outperforming comparable multilingual models on Italian benchmarks, Minerva-3B’s performance on the INVALSI tests was near chance, at only 4.9%. Researchers emphasize that while dataset composition is important, the overall size of data and model parameters are critical for handling complex language tasks. The empirical results challenge assumptions that larger native-language datasets alone suffice for deep language understanding, highlighting the need for even larger investments.

Implications for European Sovereign-Language AI Strategies

The results from Minerva demonstrate that even substantial investments in native-language data and model size may be insufficient to achieve true language comprehension and academic-level performance. This raises questions about the scale of investment needed for effective national AI models and suggests that European strategies must consider more aggressive scaling. The findings also challenge the narrative that training from scratch on large native datasets is a straightforward path to language mastery, emphasizing the importance of resource allocation and infrastructure. For policymakers and researchers, these insights point to the necessity of re-evaluating current approaches and setting realistic expectations about what national AI projects can achieve at given scales.

Italy’s Sovereign-LLM Development and Its Challenges

Italy’s Minerva project represents one of Europe’s most ambitious efforts to build a sovereign language model from scratch, utilizing Italy’s national supercomputing resources and a large, dedicated dataset. The project trained models up to 7 billion parameters on 2.5 trillion tokens, with significant institutional backing from Sapienza University, CINECA, and the Italian government. While Minerva outperformed multilingual models on benchmarks, its performance on the INVALSI Italian exam was remarkably low, revealing a discrepancy between benchmark success and real-world language understanding. This development follows broader debates within Europe about the feasibility and scale of sovereign AI initiatives, contrasting with approaches like Portugal’s AMÁLIA, which layers specialization onto multilingual foundations. The key lesson emerging is that scale and native-language investment are critical, and current efforts may still fall short of producing truly capable models.

“Despite the large dataset and dedicated Italian training, the model’s performance on academic tests was near chance, indicating a fundamental scale challenge.”

— Research team, Minerva project

Unresolved Questions About Scaling and Effectiveness

It is not yet clear what the precise scale of data and parameters is required for sovereign models to achieve academic-level performance. The ongoing research aims to refine these thresholds, but definitive benchmarks remain unestablished, and the impact of training methodology versus scale is still being evaluated.

Next Steps in Sovereign-Language Model Development

Researchers will continue iterating on Minerva’s architecture and training processes, with upcoming experiments focusing on larger datasets and parameter counts. Further assessments are expected to clarify the relationship between scale and language understanding, guiding future investments and policy decisions. The project team also plans to publish more comprehensive results to inform European AI strategy debates.

Key Questions

Why did Minerva score so low on the Italian academic test?

Despite extensive training, the model’s limited performance suggests that the scale of data and parameters may still be insufficient for complex language understanding tasks.

How does Minerva compare to other European sovereign LLMs?

Minerva trained from scratch on a large native dataset, outperforming multilingual models on benchmarks but revealing limitations in real-world comprehension, unlike approaches that layer specialization onto multilingual foundations.

What does this mean for future European AI projects?

It indicates that more substantial investments in data and model scale are necessary to develop truly effective national language models, challenging current assumptions.

Is the low test score indicative of overall model quality?

Not necessarily; it highlights a specific challenge in achieving deep language understanding at current scale levels, and ongoing research aims to address this gap.

Source: ThorstenMeyerAI.com