🔍 Read the full analysis: How GPT-6 Astra Tuned A Qwen Model To Handle JSON Tabs Better on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
An OpenAI GPT-6 Astra agent made a rank-one edit to one matrix in Qwen3-4B-Instruct-2507 to target failures escaping tabs in JSON strings. The model then answered 70 of 120 frozen test cases exactly, up from 49, but still failed 50 cases and did not improve at extracting table rows.
An OpenAI GPT-6 Astra agent edited one matrix in the open Qwen3-4B-Instruct-2507 model and raised its exact answers on a frozen test of copying special characters into JSON from 49 of 120 to 70 of 120. The reported gain was 17.5 percentage points, with 21 cases fixed and none made worse; the result covers one narrow, synthetic task, not JSON handling generally.
The task was to copy short text containing tabs and other special characters into a JSON field so that a standard parser would recover the exact original text. A literal tab must be represented in the JSON string as \t; merely producing valid JSON or a similar-looking string was not enough. The source report says the original model answered 16 of 32 initial screening cases correctly. A more explicit escaping prompt also scored 16 of 32.
The GPT-6 Astra session, running as a Codex agent, inspected the model weights, designed the experiment, made the edit, froze the final test and measured the results, according to the report. It applied a calculated rank-one update to the output projection of the model’s final MLP layer. The report says no gradient training was used and the other 397 stored tensors remained bit-identical to the originals.
On the final test, the edited model produced 70 exact answers out of 120, compared with 49 for the original. However, 50 final cases remained incorrect. A separate test of extracting a table row stayed at zero correct out of 30, so the result does not show a general improvement in structured-data tasks. The cases were synthetic, and the experiment has not yet received a second-AI review.
Infographic 01 · The measured result
Where the edit helped, and where it did not
Qwen3-4B-Instruct-2507, original versus one edited matrix, on synthetic texts that must survive the trip into JSON without losing a character.
Final test · 120 cases by group
- Texts with tabsthe target failureOriginal19/30Edit30/30
- Line breaksalready perfectOriginal30/30Edit30/30
- Mixed special characterstabs, quotes, backslashesOriginal0/30Edit10/30
- Second table rowstill unsolvedOriginal0/30Edit0/30
Development, validation and controls
- Developmentused to fit the edit · not independentOriginal24/40Edit40/40
- Validationnew cases · 9 wins, 0 lossesOriginal10/40Edit19/40
- Random directioncontrol on validation · same strengthOriginal10/40Edit10/40
- Plain-text tabscontrol · real tabs must surviveOriginal10/10Edit10/10
Exact answer = after normal JSON parsing, the text field matches the source text character for character. Same prompt, temperature 0, seed and output limit for both models; saved and reloaded edit, no hooks, no grammar constraints, no repair. Final test, validation and controls frozen before any independent output of the edited model. Fifty of 120 final answers are still wrong.
Sources: our experiment records, 5 Nov 2026 (Lexassay research, internal): experiment 3 result report and final result by GPT-6 Astra working as a Codex agent (OpenAI). Synthetic test cases, not published.
Powered by Thorsten Meyer AI
A Narrow Gain in Exact JSON Copying
The result suggests that a small, targeted weight edit can change a model’s performance on a specific behavior without retraining all of its parameters. For developers, exact character preservation matters in data pipelines: a tab, quote or line break mishandled during serialization can alter records or make downstream processing fail. The test’s exact-match scoring makes that failure visible rather than counting nearly correct output as success.
The limits are just as relevant. The edited model still failed 50 of 120 cases, and the unchanged table-row result shows that this intervention did not address every structured-output weakness. The experiment therefore offers evidence of a task-specific improvement, not proof that editing weights is a reliable substitute for clearer prompting, conventional software validation or broader model evaluation.
Products Worth Considering
Samsung Galaxy Tab A7 Lite 8.7 32GB Wi‑Fi Android Tablet – Long‑Lasting Battery, Compact Slim Design, Sturdy Metal Frame (US Version)
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
Anker SOLIX S2000 Solar Powered Generator, 2010 Wh, 1500 W
As an affiliate, we earn on qualifying purchases.
The Earlier Test Pointed to Tabs
This was the third experiment in a week-long series about editing model weights. In an earlier trial, an AI agent tested Qwen on extracting five fields from synthetic business messages. Both the original and edited versions scored 97 of 200 on the frozen final test. According to the report, both also produced 47 invalid answers, mainly by placing literal tab characters inside JSON strings instead of escaping them.
That earlier failure supplied the target for the new experiment; this was not a broad search across the model for weaknesses. The series also included a separate edit by Anthropic’s Claude to a local model called Ornith. GPT-6 Astra reviewed that earlier work and reportedly identified implementation faults and side effects. Claude was not involved in this Qwen experiment; the source says a Claude agent later helped research and draft the article, while the experiment itself was run by GPT-6 Astra.
The source identifies Qwen3-4B-Instruct-2507 as an open model under the Apache-2.0 license. It describes the work as an internal experiment and says the report and graphic were produced by the GPT-6 Astra Codex agent. Those details establish the provenance claimed by the publisher, but do not amount to an independent replication.
“The starting hint for experiment 3 came from the previous attempt.”
— The experiment report
Independent Checks Are Still Pending
The result comes from the experiment records described by the publisher; the supplied material does not include an independent replication or a second-AI review of this experiment. It is also not clear whether the same gain would hold on real-world JSON payloads, different prompts, other model versions or tests created by an outside evaluator. The test cases are described as synthetic, and the source does not provide enough detail here to assess how closely they represent typical production data.
The report says the final test was frozen before the result was seen, but the available account does not detail the full test-generation and scoring process. That does not negate the reported score, but it leaves readers without an external check on the procedure. The finding also says nothing about broader model safety, reliability or performance outside the measured tasks.
Replication Would Test the Result
The next useful step is an independent review of the code, weight change, frozen cases and scoring, followed by replication on a separate test set. Testing real-world examples and checking whether the edit affects other tasks could show whether the gain is robust or comes with trade-offs not captured by the 120-case test.
Until that work is reported, the confirmed claim is limited to the measured comparison in this experiment: 70 of 120 exact answers after the edit, versus 49 before it. The source material does not give a date for a review, replication or follow-up experiment.
Key Questions
What did the GPT-6 Astra agent change?
It applied a calculated rank-one update to one matrix, the output projection of Qwen3-4B-Instruct-2507’s final MLP layer. The report says it did not use gradient training and left the other 397 stored tensors bit-identical.
How much did the Qwen model improve?
It answered 70 of 120 final test cases exactly after the edit, compared with 49 of 120 before it. That is a reported gain of 17.5 percentage points, with 21 cases fixed and none broken on that test.
What did the test measure?
It measured whether the model could copy short text containing tabs and other special characters into JSON and preserve the exact original text when the JSON was decoded. For example, a tab in a JSON string must be represented as \t.
Does this show the model is generally better at JSON?
No. The result concerns one synthetic text-copy task. The model still missed 50 of 120 final cases, and its table-row extraction score remained zero out of 30.
Has another reviewer confirmed the experiment?
Not yet, according to the source material. It says the experiment has not had a second-AI review; an independent replication is also not reported.
Source: ThorstenMeyerAI.com
