🔍 Read the full analysis: We Put The AI Built For Law Through 15 Legal Questions on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A test published by ThorstenMeyerAI.com on Nov. 4 assessed 42 AI setups on 15 questions covering German, U.S. and EU law. Its operator reports that GLM-5.3 FlashX Legal scored 8 out of 100, while Claude Opus 5.5 scored 79; the operator also discloses that the test is run by its own company and Claude agents graded answers. The results are a limited spot check, not a definitive ranking, and the test found substantial citation problems.
ThorstenMeyerAI.com reported on November 4 that GLM-5.3 FlashX Legal scored 8 out of 100 on Lexassay, a test of 15 legal questions, and fully solved none of them. The test operator says it assessed 42 AI setups; its results also flag citation errors among stronger-scoring models, but the assessment is a limited spot check rather than a definitive measure of legal competence.
Lexassay covers German, U.S. and EU law, with questions on legal deadlines, changes to laws, court decisions and prompts based on false assumptions. Ten questions are public and five are hidden. Models answer without internet access or other tools, so the test measures what they produce from their internal knowledge rather than from legal research conducted during the task. The operator says the models received the same questions and were graded against weighted criteria.
According to the report, GLM-5.3 FlashX Legal, which its maker Z.ai describes as optimized for professional legal workflows, scored 8 overall and zero on the hidden questions. It gave no answer on two tasks after using its full 64,000-token reasoning allowance, the report says. Its base model scored 3. The test recorded 21 citations as real, 25 as misattributed and one as invented for the legal model.
Claude Opus 5.5 led the reported results with 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The article says the leading model met every criterion on only eight of the ten public tasks. Mistral Large 4 scored 10, with the report attributing much of its lost credit to citations that were wrong or did not support the answer. In total, Lexassay says it confirmed 2,644 citations as real and found 804 invented or inconsistent with the claims made. These are the publisher’s test findings, not an independent audit.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Citations Matter
The scores matter because legal AI answers can affect deadlines, claims and decisions, where a confident but incorrect answer may be more harmful than an obvious refusal. The report illustrates this with a German appeal-deadline question: a public holiday moves the deadline in Munich but not Berlin. It says one tested model gave Berlin an extra day, while another placed the deadline for appeal grounds a month late. Under the scenario described, a late appeal can be rejected as inadmissible.
Citation errors pose a related risk. A case or statute may exist but fail to support the model’s statement, or a citation may be invented. Lexassay’s scoring deducts points for those failures, and its findings suggest that a high overall score should not be treated as proof that each legal reference is reliable. For readers, the practical implication is limited but clear: verify cited authorities and do not rely on a chatbot alone for a legal deadline or a consequential step.
Products Worth Considering
EcoFlow Delta 3 Portable Power Station – 1024 Wh, 1800 W
Apple Watch SE 3 GPS Smartwatch, 40mm
Amazon Kindle Paperwhite Signature Edition, 32GB (2024), Metallic Black
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
The test’s design aims to measure model answers without live research. Lexassay uses 15 questions per setup: ten published questions and five hidden ones, with only the hidden-set score disclosed. The operator says hiding some questions makes it harder for future models to reproduce answers from online material. Prompts span three jurisdictions and include questions that test whether a model catches a false premise rather than accepting it.
Each response is compared with a gold answer containing three to six weighted criteria. The operator says citations are checked against official databases and classified as real, misattributed, invented or not findable. A misattributed or invented citation costs one-quarter of the task’s points; an answer that omits a legal-advice disclaimer or referral to a lawyer loses 5 percent. The report says it collected one answer per model and task, making small score differences potentially noisy.
The publisher also points to a separate Vals AI legal-agent benchmark, which supplies models with documents and ranks Mistral Large 4 eighth, ahead of Claude Opus 5.5 and GPT-6 Astra in that comparison. Those results measure a different setup from Lexassay’s no-tools test, so the rankings are not directly interchangeable. The contrast underscores how benchmark design changes what is being measured.
““Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.””
— Z.ai, as quoted in the report
Limits Behind the Scores
Lexassay is run by Thorsten Meyer, the publisher of the report, and the answers were graded blindly by Claude agents made by Anthropic. The article acknowledges that this creates a conflict of interest, particularly because a Claude model ranked first. Meyer says the method page discloses the relationship and that public answers, gold answers and citation checks are available for review. The source material does not provide an independent replication of the results.
The test is also narrow: it covers 15 questions, uses one answer per model and task, and evaluates models without browsing or tools. Hidden-task scores are not accompanied by the same public answers available for the open set. The results therefore do not establish how each system would perform with current legal databases, professional supervision or a different set of cases. The report also gives no basis for treating a small gap between scores as meaningful.
It remains unclear from the supplied material whether the companies whose models were tested reviewed or disputed the results, and whether Lexassay plans repeat tests to track changes in model versions. The reported citation totals apply to the published runs described by the operator; they should not be read as a universal error rate for legal AI use.
How Readers Should Use the Findings
The report’s immediate recommendation is to treat AI output as a starting point, not legal advice. Readers should check each cited statute and ruling in an authoritative source, and consult a lawyer when a matter involves money, a deadline or communication with an authority. In particular, the test’s deadline examples show why a general chatbot response may miss jurisdiction-specific holiday rules.
For the results to support firmer comparisons, useful next steps would include repeat runs, additional questions and independent review of the scoring and citations. The article does not announce a schedule for those steps. Until further evidence is available, readers should take the reported scores as a snapshot of the tested setups and prompts, not a guarantee of how a model will perform on an individual legal problem.
Key Questions
What did the law-focused model score?
Lexassay reports that GLM-5.3 FlashX Legal scored 8 out of 100 and scored zero on the five-question hidden set. The publisher says it fully solved none of the 15 tasks.
Which model ranked highest in this test?
Claude Opus 5.5 led the reported results with 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The ranking comes from Lexassay, whose operator discloses that Claude agents graded the answers.
Does the test show which AI is best for legal work?
No. The publisher calls Lexassay a spot check, based on one answer per model and task, without internet access or tools. Another benchmark cited in the report uses supplied documents and produces a different ranking.
Can I rely on an AI chatbot for a legal deadline?
The report advises readers not to take a deadline from a chatbot. Legal deadlines can depend on jurisdiction and holidays; verify the rule with an authoritative source and consult a lawyer about a matter with a deadline.
Who ran and graded the test?
Thorsten Meyer’s organization runs Lexassay. The report says Claude agents graded answers blind, and discloses that the arrangement is a conflict of interest because a Claude model ranked first.
Source: ThorstenMeyerAI.com
