Gemma vs Mistral vs Qwen: Three Local LLMs Tested on German Job Ads and My Own Spoken German

Most "best local LLM" lists rank models on someone else's benchmark. These numbers come from two jobs I actually needed done, on one laptop: a MacBook Pro with an M1 Max and 64 GB of unified memory, running the models through Ollama.

1. Structured extraction. For my Master's study project I had local models read German job advertisements and return every skill, task and qualification as JSON matching a fixed annotation scheme, scored against human annotations.¹
2. Correcting spoken German. For myberuf, my German-for-work practice app, the model corrects what a learner just said. I tested the candidates on made-up sentences first, then on 58 things I'd actually said, with every mistake documented in my German journal.²

The two tests at a glance

Test 1: structured extraction from German job ads

Each model got the same prompt and one job ad at a time and had to return entities (skills, tasks, qualifications and five other types) and the relations between them as one JSON object. The scores compare that output with human annotations.¹

GPT-OSS 20B returned almost no parseable output and was dropped.¹

How to read this:

• Gemma and Mistral are close on accuracy. Fifty documents is a small sample: in the same study, five random 50-document samples of the final setup ranged from 0.386 to 0.453, and the report concludes that a sample this size can't support a one-point difference. So 0.292 against 0.277 isn't a real gap. Qwen3's 0.205 is further behind; still, re-check a gap that size on your own data.¹
• The F1 only covers documents that parsed. Mistral's 66% valid JSON means a third of its answers scored nothing usable, which flatters its F1. That's why valid JSON belongs next to accuracy in any table like this.
• Speed decided it. Gemma was the fastest and the most reliable. Its mixture-of-experts design keeps 25.2 billion parameters in memory but activates only 3.8 billion per token, a plausible reason, though the study didn't isolate architecture as the cause.¹

What happened next: Gemma became the model for all later work. With a tuned prompt (two fully annotated examples) and one retry for broken JSON, it reached entity F1 0.437 and valid JSON for 99.6% of all 498 job ads, at 31 seconds each, against 0.801 for the supervised reference system (measured on character spans, so not a like-for-like comparison).¹ Two findings from that phase matter for anyone choosing a model:

• Size didn't order quality. Gemma 4 26B beat NVIDIA's Nemotron 3 Super, which activates about 12 billion of its 120 billion parameters per token. Within the Gemma family, the larger 31B version did better (0.543 on the same 50 documents, run in the cloud).¹
• Reliable JSON and correct content are different problems. Adding a schema retry and a reviewer pass that was only allowed to fix labels lifted schema-valid output from 18% to 89% but moved entity F1 by 0.011. The code check behind the reviewer rejected 71.4% of its proposals for breaking its rules or returning invalid JSON.¹

Test 2: correcting my own spoken German

In myberuf, a learner speaks, Whisper transcribes it, and the model returns corrections as JSON. Qwen3 wasn't part of this test; it was Gemma against Mistral.²

Round 1, made-up sentences (3 October 2026):

On this evidence I kept Mistral: it flagged slightly more, the exact corrections tied, and Gemma made one meaning-changing suggestion in both runs.

Round 2, things I actually said (10 October 2026):

Real speech reversed the decision, and I switched to Gemma. Mistral caught about as many mistakes, but its fixes often broke the sentence: a dropped subject here, the article fixed while the verb stayed in the wrong place there. It also changed two sentences that were already right. Gemma's typical failure was the opposite: it missed things, mostly wrong word choices that it labelled "more natural" instead of a mistake, and verb position after a long clause. For a learner, a missed mistake costs less than a confident wrong correction.²

Setup tips that cost me time

• Switch the thinking off, and check it really is off. On Ollama's OpenAI-compatible endpoint, think: false was ignored for Gemma 4: one sentence took 42 seconds and returned nothing, while reasoning_effort: "none" gave valid JSON in 2.5 to 3.3 seconds.² Qwen3 also has a thinking mode; its model card documents how to switch it off.⁴
• Measure valid JSON separately from accuracy. A parser and a schema check catch broken output without any reference answers. They can't catch a missing entity, so you still need a scored sample.¹
• One retry for broken JSON is cheap; a reviewer model isn't. The retry was worth it; the reviewer pass made each document about 3.5 times slower for one point of F1.¹
• Never let a second model overwrite output without a check in code. That rule caught 350 bad reviews in my study. I wrote it up in how to evaluate LLM output.

Sizes and licences

All three are open weights under Apache 2.0, so the choice is about quality, speed and memory, not licensing. Search suggestions now mention newer versions (Qwen 3.5 to 3.8, Mistral Small 4); I haven't tested those, and the ranking above may not carry over.

Where my skills fit

The method behind both tests, scoring against a reference, keeping valid JSON separate from accuracy, and never trusting a model's review without a check in code, is packaged in my LLM evaluation pack: LLM output validation and LLM evaluation integrity come straight out of the study. They help whichever model you run.

FAQ

Is Gemma better than Qwen?

In my tests, yes: Gemma 4 26B scored entity F1 0.292 against 0.205 for Qwen3 14B on the same 50 German job ads, returned valid JSON more often (88% against 80%) and was faster (26.4 against 62.1 seconds per ad). Qwen3 14B is much smaller, though, and newer Qwen versions weren't part of the test.

Gemma 4 or Mistral Small: which should I run?

On my Mac, Gemma 4 26B: tied on extraction accuracy, far more valid JSON (88% against 66%), more than four times faster, and on my real spoken German it fixed more mistakes right and left correct sentences alone. Mistral Small 24B caught slightly more errors on made-up sentences. Test both on your own data before you decide.

Which local LLM is best for structured JSON output?

Of the models I tested, Gemma 4 26B A4B: the highest valid-JSON rate in the screening (88%) and the best accuracy. With one retry for broken JSON it parsed 99.6% of 498 documents. GPT-OSS 20B returned almost no parseable output in the same setup.

Can I run these models on a Mac?

I ran all three on a MacBook Pro with an M1 Max and 64 GB of unified memory, one request at a time. I didn't log peak memory during inference, so I can't give you a minimum. Ollama's downloads are 16 to 19 GB for Gemma 4 26B, 14 GB for Mistral Small 24B and 9.3 GB for Qwen3 14B.

Which is fastest?

Gemma 4 26B in both of my tests: 26.4 seconds per job ad against 62.1 for Qwen3 14B and 116.3 for Mistral Small 24B, and 3.0 against roughly 8 seconds per spoken sentence against Mistral.

Should I fine-tune a local model instead?

Only after the prompt work. In my study, a deliberately small QLoRA adapter trained on the laptop in 106.5 minutes (32.8 GB peak memory) and moved entity F1 from 0.392 to 0.404, a change within the noise.