📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project built a large-scale European sovereign LLM from scratch, but it underperformed on Italian academic benchmarks. This raises questions about the scale of native-language data needed for effective country-specific models.
Italy’s Minerva-3B, a large-scale European sovereign language model trained from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, highlighting significant challenges in achieving country-specific language understanding through scale alone.
The Minerva project, led by Sapienza University of Rome, involved training models from scratch on a massive dataset, making it a notable case in European sovereign AI development. Despite its impressive technical infrastructure, including 128 GPUs on Italy’s CINECA supercomputer and open weights and data, Minerva-3B’s performance on the INVALSI test was near chance, a surprising outcome given the model’s extensive Italian data.
Researchers from the project concluded that, while dataset composition is important, the overall size of the dataset and the number of parameters are more critical for handling complex language tasks. This empirical result suggests that even large-scale native-language training may not suffice at the current parameter levels, raising questions about the necessary scale for effective country-specific models.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Fine-Tuning Large Language Models: From Custom Datasets to High-Performance AI Models Using Modern Toolchains
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-LLM Strategies
The findings from Minerva challenge assumptions that simply increasing native-language data and model size guarantees better country-specific language understanding. For European AI initiatives, this underscores the need to consider more than just scale, including data quality, architecture, and training methodology. The results suggest that current investments may still fall short of producing models with deep country-specific knowledge, influencing future policy and funding strategies across Europe.
European Sovereign LLM Development and Scaling Debates
The Minerva project is part of a broader European effort to develop sovereign language models, contrasting with projects like Portugal’s AMÁLIA, which layered specialization onto multilingual foundations. Italy’s approach trained from scratch on a large Italian dataset, leveraging national infrastructure and open data. Prior to Minerva, many believed that scale alone could overcome language complexity, but recent results indicate that this may not be sufficient at current parameter levels, prompting a reevaluation of strategies.
“The Italian project demonstrates impressive engineering, but the low exam scores highlight the persistent challenges of achieving deep language understanding through scale.”
— Thorsten Meyer, AI researcher
Unresolved Questions About Model Scaling and Performance
It remains unclear what specific factors beyond scale—such as training methodology, data quality, or architectural choices—are necessary to improve performance on complex language understanding tasks. The ongoing research may reveal whether alternative approaches can bridge the gap between data volume and real-world language competence.
Next Steps in European Sovereign AI Development
The Minerva team is continuing to refine their models, including ongoing experiments with continual training and different architectures. Future evaluations will likely focus on whether increased scale or improved training methods can enhance performance. Policymakers and researchers may also reconsider funding and strategic priorities based on these findings, emphasizing quality and efficiency over sheer size.
Key Questions
Why did Minerva-3B perform so poorly on Italian school exams?
Despite extensive native-language data, the evaluation suggests that current model sizes and training approaches may be insufficient for complex language understanding tasks like academic exams. It indicates a need for different or additional strategies beyond scale alone.
Does this mean European sovereign models are not viable?
Not necessarily. The results highlight challenges but also provide valuable insights into what is needed to improve performance. Future research may develop more effective methods to achieve deep country-specific language understanding.
What does this mean for future investments in sovereign AI?
Investments may need to focus more on data quality, architecture, and training techniques rather than just increasing scale, to produce models capable of handling complex language tasks effectively.
Is the performance of Minerva-3B typical for large-scale models trained from scratch?
While some large models perform well on specific benchmarks, Minerva’s results suggest that simply scaling up does not guarantee high performance on complex, real-world tasks, especially in languages with fewer resources.
Source: ThorstenMeyerAI.com
