It caught "partner" → "pareja" in an IRS rule and missed "return" → "devolución": an offline MT checker for translators

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built#

The IRS page on the failure-to-file penalty has a rule for small partnerships: "Each partner is either an individual (excluding nonresident aliens), or the estate of a deceased partner." An open machine-translation model turned it into «Cada pareja es una persona (…) o la herencia de una pareja fallecida». A pareja is a romantic partner; the IRS's own Spanish says socio. A condition for penalty relief has become a rule about couples.

SecondRead flags it at 86% risk. On the same page it misses "the return is late" → «la devolución es tardía», the refund is late. This post covers both.

Post-editors fix machine translation against a deadline. Broken Spanish jumps out; the costly mistake is the fluent sentence that means something else. QA tools catch numbers, tags and glossary terms, not pareja, and many NDAs restrict pasting client text into cloud AI.

So I built SecondRead, a second reader that runs on the translator's Mac. It reads an XLIFF or Trados SDLXLIFF file, scores each segment for the risk of a major meaning error (wrong sense, dropped negation, wrong number, omission or addition), flags the risky ones and sorts the job so you re-read the riskiest 10% first.

Under the hood: Qwen3.5-4B at 4-bit (2.7 GB), a 36 MB LoRA adapter I fine-tuned on Tinker for $1.22, and rule checks for numbers and negation; a 39-segment page takes about 6 s on an M4 Pro, offline. On WMT English→Spanish test it ranks errors as well as Gemma 4 31B (AUROC 0.815 vs 0.807); on ten real public documents Gemma ranks better (0.940 vs 0.910), and my default flag line finds only a fifth of the errors.

Who would pay: agencies, then freelancers, who already buy QA tools such as Xbench and Verifika. WMT text is research-only, so the product is this pipeline trained on an agency's own reviewed jobs: a private 36 MB adapter on each reviewer's laptop, per seat, with no per-segment bill. What they would actually pay is still open; that's what I'm testing next.

Demo#

89 s: an FDA page on disposing of medicines, machine-translated into Spanish and checked on the Mac: 39 segments, 3 flags, and a network log with only local requests.

No-install demo: https://secondread-e49hx.ondigitalocean.app, a static App Platform build with that page's scores precomputed. The top flag is the DEA's "Take Back Day", now «Día Nacional de Retraso de las Drogas Recluidas», a "Day of Delay of Confined Drugs" (80%). It misses the instruction to flush fentanyl patches, which became «lavar», wash (16%).

Code#

The code is on GitHub at P0rt/secondread (MIT): everything from data prep to the UI, with the command behind every number below in docs/RESULTS.md and docs/REALDOCS_GOLD.md.

How I Built It#

I built it with Claude Code, starting with go/no-go checks: does Tinker train this model, does the adapter survive GGUF conversion, does Sentry receive zero text?

One token is the whole answer#

The model answers one question per segment. This is the entire user message:

EN: {source}
ES: {machine translation}
Major meaning error (wrong meaning, negation, number, omission, addition)? Answer Major or OK.

Major and OK are single tokens in Qwen's vocabulary, so one forward pass gives risk = p(Major) / (p(Major) + p(OK)), with the same prompt in training, Tinker evaluation and the app. The rule checks add highlights (which number changed, where the negation went) and a small train-measured weight on the log-odds.

Training on Tinker, on human labels#

The labels are Google's wmt-mqm-human-evaluation (Apache-2.0): every error in WMT24 English→Spanish (9,330 segments) and WMT22 English→Russian (19,292), marked by human annotators with an MQM category and severity. Major means at least one major or critical accuracy, terminology or non-translation error. I split by document.

The run: LoRA rank 16 on the MLP layers only, one epoch over 14,791 human-labelled segments (231 steps of 64), loss only on the answer.

The cookbook's suggested learning rate, 4.9e-4, flipped the model from always-OK to always-Major within three steps; 20-step probes picked 2e-4. The run took 9 min 24 s and $1.22: $1.07 for 1,452,983 training tokens, $0.15 for dev scoring. Dev AUROC rose from 0.552 to 0.739. Locally, the 4-bit model ranks like Tinker (Spearman 0.989).

Results on WMT test#

Unseen documents (25 en-es, 33 en-ru), scored by the model alone. AUROC is the chance a segment with a major error ranks above a clean one (0.5 is a coin flip); PR-AUC rewards a clean top of the list.

• Fine-tuning works: the adapter alone adds +0.039 AUROC over the base model on en-es (95% CI +0.003 to +0.070) and +0.135 on en-ru (+0.090 to +0.191).
• Level with a zero-shot model about 8× its size on ranking: v1 minus Gemma is +0.008 AUROC on en-es and +0.006 on en-ru, both CIs across 0. On en-es PR-AUC v1 leads by 0.206 (CI +0.099 to +0.286), mostly because Gemma saturates: 27% of en-es rows score 0.97 or more.

Choosing the flag line: recall first#

My first flag line, F1-optimal on dev, caught only 30% of the major errors in en-es test.

A false flag costs a post-editor seconds; a missed error reaches the client. So before computing any number I fixed a rule: per pair, the highest threshold that still finds 60% of the dev Majors. For en-es that is 0.4984 (risk ≥ 50% in the UI), flagging 10% of dev segments instead of 3%, and it became the default. Then test, once:

en-es missed its target: the whole CI is below 0.60. I didn't move the line; that would be a choice made on test.

Real documents: ten public pages, labelled before scoring#

Post-editors also get tax rules and medicine leaflets. So I took ten U.S. federal and EU pages with an official Spanish version (IRS, FDA, CDC, SSA, USCIS, EU air passenger rights and more) and translated their 720 segments twice: with OPUS-MT, an open 2020 model, locally, and with DeepSeek-V3.1, an open 671B LLM, on Tinker. Nothing was tuned on these 1,440 translations.

The labels. Three blind AI annotators (an MQM rater, a post-editor and a Spanish-only reader) labelled each translation Major or OK; two of three decide. None saw the MT system or any score. On 247 WMT24 units with professional labels, a first version of the panel reached F1 0.67 and Cohen's kappa 0.59. Labels, scorer and flag lines were fixed before any detector was scored.

The surprise is the MT systems: of the 84 major errors, 80 are OPUS-MT's (11% of its segments). DeepSeek-V3.1 made 4 (0.6%); seven of the ten pages have none.

Gemma's line is from en-ru dev; CIs come from a document bootstrap.

• Ranking: v1 beats its base model (AUROC +0.071, CI +0.036 to +0.119) but trails Gemma (−0.030, CI −0.051 to −0.009), the reverse of WMT test. On OPUS-MT's output alone the gap is within noise (−0.013, CI −0.051 to +0.017); Gemma's lead comes mostly from ranking DeepSeek's clean segments lower.
• The default line is far too quiet here: 17 of its 22 flags are real, but that is a fifth of the errors (48% on WMT test), while 52 of the 84 sit in its riskiest 10%. Again, the line stays.
• Caught: garbled names and words that become other words, such as pareja, «lista de color» for the flush list and «Desclutter»: 16 of 70 mistranslations, 1 of 2 untranslated.
• Missed: the wrong sense in fluent Spanish: fentanyl patches to be «lavar», washed; «Si hace calor», if the weather is hot, for a hot door in a fire; immigration parole as «libertad condicional»; «devolución». Also all 7 omissions, all 5 terminology errors and all 3 critical errors (Gemma flags all three).
• Quiet where it matters: v1's 5 false flags are all minor errors, only 2 on DeepSeek's 716 clean segments (Gemma 16, base 25), where a real error is one in 180. The publishers' own Spanish, scored as MT, gets 3 flags in 720 (Gemma 79). But v1 caught none of DeepSeek's 4 errors (Gemma 1).

The caveats: the labels are AI's, from three lenses that agree closely (Fleiss' kappa 0.88) and an adjudicator in the same workflow, so treat them as one annotator; no professional labelled this set, and the panel saw the official Spanish. An LLM judge like Gemma may share the panel's idea of Major more than a 4B model trained on WMT labels does.

Can synthetic errors teach it wrong senses?#

Mistranslations are 70 of the 84 real-page errors and 66–76% of WMT test's major errors. v2 showed the trap: 5,095 synthetic number, negation, omission and addition rows taught it those edits and cut en-es test PR-AUC from 0.517 to 0.446 (CI of the drop −0.113 to −0.009).

For v3, Gemma swapped one word of a clean training translation for a wrong sense, false friend or near-antonym (Major) or a synonym (OK); a blind second Gemma call checked each edit, and the real-page error words (return, flush, partner …) never entered a prompt. Under a rule fixed first and run on dev only, v1 stays: v3a and v3b both lose about 0.035 PR-AUC on en-ru dev. They learned their generator (en-es recall 0.79–0.80 on its held-out edits, v1 0.56) but ranked en-es dev's real mistranslations no higher: 46 of 73 in the riskiest 10%, as for v1. Reviewed jobs are the better training data, and they are the paid product.

Gemma 4 31B on DigitalOcean: the yardstick and the error factory#

Gemma 4 31B runs on DigitalOcean's serverless inference (the Gradient AI Platform) and sees only public text.

1. The yardstick: on WMT, 9,436 calls with the small model's prompt and 800 with GEMBA-MQM, $0.024 per 1,000 en-es segments, 0.58 s median; 2,435 more on the real pages. GEMBA-MQM ranked worse than the plain question, at 14× the cost.
2. The error factory: the v2 negations, omissions and additions (5,626 calls, $0.42) and the v3 wrong-sense edits and checks (7,101 calls, $0.81).

In all, at most $1.64 of a $5 credit. Gemma 4 is Apache 2.0, and I found nothing in DigitalOcean's terms against training on its outputs.

Sentry: tracing every review without the text#

Each review is one Sentry AI-agent trace: invoke_agent SecondRead, four execute_tool spans (parse_xliff, number_check, negation_check, lora_score), an aggregate span and a gen_ai.chat span per segment with input tokens, latency, p(Major) and cost (0: local). An FDA review is 39 chat spans and 3,159 input tokens: 6.4–7.5 s per trace below (5.9 s on the backend's own timer), about 0.6 s per model call, four in flight.

No segment text goes in, and no segment IDs either, because some CAT tools use the source sentence as the ID. The hostname (often a person's name) is replaced, and the OpenAI and httpx integrations are off, so each model call is traced once.

scripts/canary_check.py plants SR-CANARY-<uuid> in segment text, XLIFF IDs, the file name, the query string, exceptions and logs, runs real reviews and captures every envelope: 0 of 12 contain it. With the scrubbers for what integrations add on their own (exception and log messages, query strings) turned off, 4 do.

Tracing was once a suspect: after a few idle minutes, the first job took 21 s. Against bare llama-server, the backend added about 0.7 ms per segment, Sentry or not; the culprit was llama-server's RAM prompt cache, and -cram 0 holds a 51-segment job at 8.4–9.0 s. One limit: Sentry keeps about 1,000 spans per transaction, so a job over about 990 segments loses some chat spans.

What it can't do (yet)#

• Two pairs, Apple Silicon only.
• Wrong senses in fluent Spanish are the blind spot, and synthetic data didn't help.
• It ranks; it doesn't certify: the default line finds about half of the en-es test errors and a fifth on the real pages.
• Raw MT, one segment at a time: untested on edited text; errors across sentences are invisible.
• Noisy, public training data: one annotator per segment, nothing legal or medical, possibly seen by Qwen and Gemma.
• A big cloud model catches more: Gemma found 82% of the real-page errors, flagging 12%.

Why Does Open Innovation Matter?#

The model has to come to the text. Under an NDA, what matters is whether the text leaves the machine. Open weights and llama.cpp make "no outbound calls" checkable: the API listens on 127.0.0.1, nothing loads remotely, and Sentry is off unless you turn it on.

The licensing trap. The best-known quality-estimation models, Unbabel's xCOMET and CometKiwi, are CC BY-NC-SA 4.0: a paid tool needs Unbabel's permission. Qwen3.5-4B, Gemma 4, OPUS-MT and Google's annotations are Apache-2.0, so I could train my own and translate the real pages locally.

The caveat: training wasn't local, but Tinker and DigitalOcean saw only public text. Tinker billed $6.46 in all: $6.21 for Qwen3.5-4B (four training runs, probes and evaluations), $0.23 for the DeepSeek-V3.1 translations and $0.02 for checkpoint storage. DigitalOcean cost at most $1.64 at list price, and Sentry's free plan covered tracing.

Prize Categories#

• Best Use of Tinker. The shipped model is a $1.22 Tinker LoRA, run locally as a 36 MB GGUF. Over the untuned base model it lifts WMT test AUROC from 0.776 to 0.815 (en-es) and 0.666 to 0.801 (en-ru), and real-page AUROC by 0.071, level with Gemma 4 31B on WMT at $0 per segment. The v2 and v3 Tinker runs decided what ships, and Tinker sampled the DeepSeek-V3.1 translations.
• Best Use of DigitalOcean. Gemma 4 31B on serverless inference is the cloud baseline everywhere (12,671 calls) and generated the synthetic errors, checking v3's (12,727 calls), for at most $1.64; App Platform hosts the demo: https://secondread-e49hx.ondigitalocean.app.
• Best Use of Sentry Agent Tracing. One AI-agent trace per review with tokens, latency, p(Major) and cost per model call (screenshots above); a canary test proves the traces carry no text: 0 of 12 envelopes, 4 with the scrubbers off.
• Best Use of Gemma. Gemma 4 31B, served through DigitalOcean, is the yardstick on every test set, wrote the synthetic errors for v2 and v3 (and blindly checked v3's), and is the stronger detector on the real pages.

Credits#

• Data: Google's wmt-mqm-human-evaluation (Apache-2.0; Freitag et al., 2021; en-ru annotations by Unbabel) on the WMT22 and WMT24 General MT test sets (Kocmi et al., 2022 and 2024), released for research; the repo keeps ids and labels, not their text.
• Real pages: IRS, FDA, NIDDK, FEMA (Ready.gov), USCIS, SSA, CDC and MedlinePlus (U.S. federal works, public domain or free to copy) and Regulation (EC) No 261/2004 from EUR-Lex (reuse under Decision 2011/833/EU). The Spanish shown is machine translation; no endorsement is implied.
• Code: scripts/train_tinker.py is adapted from supervised/train.py in Thinking Machines' tinker-cookbook (Apache-2.0). The GGUF export uses llama.cpp's convert_lora_to_gguf.py (MIT); llama-server runs the model.
• Prompt and models: GEMBA-MQM (Kocmi & Federmann, 2023, CC BY-SA 4.0; adapted, with attribution, in prompts/gemba_mqm.txt); Qwen3.5-4B (Apache-2.0) from unsloth's GGUF; Gemma 4 31B (Apache-2.0); OPUS-MT en-es (Tiedemann & Thottingal, 2020, Apache-2.0); DeepSeek-V3.1 (MIT); Kokoro-82M (Apache-2.0) narrates the video.
• AI disclosure: the code and this post were written with AI assistance (Claude Code). Gemma 4 31B generated the synthetic training rows, labelled as such. The real-page labels come from AI annotators and a Claude adjudicator; no professional labelled them.