🔍 Read the full analysis: A Field Test Of 71 AI Model Configurations On A News Pipeline on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer reports testing 71 AI model configurations on a news-writing pipeline from September 8 to October 13, 2026. GPT-6 Astra had the highest reported quality score, but GPT-6 Luna was chosen for production because it cost far less; two backup models were removed after producing fabricated quotes or unusable output.
Thorsten Meyer says a test of 71 AI model configurations on his news-writing pipeline found that GPT-6 Astra produced the highest-rated articles, while cheaper GPT-6 Luna was the model selected for production. The results, published October 13 on ThorstenMeyerAI.com, also led him to remove two backup models after they produced fabricated quotations or output his pipeline could not use.
The field test ran from September 8 to October 13, 2026, using the production prompt and real news sources. Meyer tested API, subscription, free-tier and locally run open-weight configurations. Each source was used to create an English and German article. The test included one longer source, a 6,340-character product filing, and one shorter, 619-character political item; Meyer says short sources are common in his workflow.
Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, scored the articles on source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with language quality also scored for German. The strongest configurations then went through a robustness run covering 10 additional articles, mostly based on short sources. Meyer reports 95% bootstrap intervals and says output counted only if it remained valid after the pipeline’s usual repair step.
Astra scored 4.00 out of 5 in both languages in the standard test, with a source-fidelity score of 5.00. In the robustness run, it averaged 3.93 across 12 articles. Meyer puts its cost at about $0.12 per article and generation time at roughly 50 seconds. Luna scored 3.67 in English and 3.68 in German in the standard test; its reported robustness average was 3.67, with source fidelity of 4.71. Meyer says Luna cost about $0.0012 to $0.0013 per article and took a median of 17 seconds in production.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Quality Versus Cost in News Production
The findings illustrate why a model leaderboard may not answer a publisher’s operational question. Astra led on measured writing quality, but Meyer judged its reported cost too high for a high-volume workflow and chose Luna instead. In his assessment, the useful production choice is the least expensive model that reliably meets a quality threshold, rather than the highest-scoring model regardless of expense.
The test also highlights a different risk from the headline model ranking: fallback systems can affect factual reliability when the primary model is unavailable or fails. Meyer reports that one backup produced invented quotations and another generated no valid articles in its four test attempts. Those are findings from his own pipeline and sample, not a general measurement of either model across other prompts or tasks.
For readers, the practical implication is that AI-written news needs controls beyond choosing a well-ranked model. Publishers must test outputs against source material, check quotation accuracy, and verify that fallback behavior preserves the same standards. The reported scores offer one example of such testing, but they do not establish how the configurations would perform for other publishers or news topics.
As an affiliate, we earn on qualifying purchases.
How the Pipeline Was Tested
Meyer framed the project as a comparison of models on the task his sites actually perform: turning a source into a finished article. The test used two sources and two language editions, followed by a larger robustness run for the strongest configurations. The short political source was included because Meyer says much of his production work begins with limited material, where unsupported detail and invented quotes are particular concerns.
His report says differences below roughly 0.3 points across the two initial test articles should be treated as noise. He describes the robustness run as a more dependable basis for ranking. The article reports that Luna’s six reasoning-effort settings did not produce a reliable improvement: the maximum setting cost about 12 times as much and took about 12 times as long as the no-effort production setting, while score differences remained within the report’s stated uncertainty.
The report also compares writing results with coding reputation. Meyer says Claude Opus 5.5, which he describes as a leading coding model, scored 3.46 in English and 3.39 in German as a news writer, below Luna in the robustness run. That comparison supports his narrower point that performance on one task does not establish performance on another; it is not an independent assessment of the coding benchmark or the model’s general capabilities.
“For a high-volume task, the right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”
— Thorsten Meyer, in the October 13 report
Limits of the Reported Results
The results come from one publisher’s pipeline, prompt and source set. The standard comparison used two news sources, and the robustness run covered 10 further articles for the strongest configurations. Meyer’s report does not establish how the models would rank on a broader range of subjects, prompts, source formats or editorial workflows. The scores are from AI judges, not a disclosed panel of human editors, and the report’s conclusions should be read within those limits.
It is also unclear from the supplied account how model versions, judge settings and scoring disagreements were independently audited, or whether the reported per-article costs include every related expense. Meyer says Astra’s 95% interval did not overlap Luna’s in the robustness comparison, but the figures do not amount to an independent replication. The report’s observed fabrication and validity failures are serious within this test, but they do not show that every use of the models will produce the same outcomes.
Luna Remains the Production Choice
Meyer says GPT-6 Luna is now the production model and that GPT-6 Astra was judged too expensive for routine news output. He reports that the two backup models were removed from the route on October 7. The source material supplied for this article does not specify a replacement failover configuration, so the current backup arrangement is unclear.
The next useful evidence would be continued monitoring on a larger and more varied set of real articles, including checks by human editors and transparent reporting of sourcing errors, invalid outputs, cost and speed. Meyer’s account describes a completed field test and a resulting routing decision; it does not announce an independent evaluation or a wider industry standard.
Key Questions
What did the 71-configuration test find?
In Meyer’s reported test, GPT-6 Astra scored highest on article quality, while GPT-6 Luna offered a much lower reported cost and was selected for production. Two tested backup models were removed after problems with fabricated quotations or invalid output.
Why did Meyer choose Luna rather than Astra?
Meyer says Astra cost about $0.12 per article, compared with roughly $0.0012 to $0.0013 for Luna. He judged Luna’s reported quality sufficiently consistent for a high-volume workflow, despite Astra’s higher scores.
How were the articles evaluated?
Each source was turned into English and German articles using the production prompt. GPT-6.1 Sol and Claude Opus 5.5 scored qualities including source fidelity, fabrication, uncertainty, news value and structure. Strong configurations also went through a 10-article robustness run.
What happened with the backup models?
Meyer reports that GPT-4.1 nano scored poorly and produced invented quotations, while locally run gpt-oss-120b produced zero valid articles out of four in the test. He says both were removed from the news route on October 7.
Do these results show which AI model is best for all newsrooms?
No. The findings reflect one pipeline, one prompt and a limited set of sources, with article quality scored by AI judges. Other publishers would need testing on their own content and editorial requirements before drawing a similar conclusion.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
