Three local models on a 36 GB Mac, tried on real jobs against Claude

2026-11-14

I wanted to know if any of my scheduled Claude jobs could run on a local model instead. On 5 and 6 November 2026 I ran three local models on three real jobs for my project: summarising GitHub repos, writing Vietnamese news posts from an English article, and sorting articles into keep or drop. Gemma can do the sorting as a first pass if losing about 1 in 7 kept articles is acceptable. Nothing that writes Vietnamese for readers should move yet.

Setup

Apple M4 Max, 36 GB, ollama 0.21.2. All models Q4_K_M, thinking off, temperature 0.3, one run per input.
Another agent was using the machine at the same time, so the seconds below are a rough guide, not a benchmark.
Quality, invented facts and Vietnamese fluency were judged by a Claude model reading each output against its input. It is not a native Vietnamese reader. The format checks (front matter, allowed values, sections, hashtags, length) are scripts and are exact.
The newest models would not download: Qwen 3.8, Gemma 4 12B and two others failed with HTTP 412 because this ollama version is too old. They are untested.
gemma4:26b-a4b-it-q4_K_M: Type mixture of experts, 26B total, about 4B active; Disk 17 GB; Memory while running 19 GB
qwen3.6:27b-q4_K_M: Type dense 27B; Disk 17 GB; Memory while running 22 to 23 GB
qwen2.5:14b: Type dense 14B, older; Disk 9 GB; Memory while running 10 GB

Job 1: repo summaries

12 repos picked with a fixed random seed, the real summary prompt (about 3,800 tokens with the first 8,000 characters of each README). The comparison is an earlier Claude Haiku summary of the same repo, so "agrees" means it agreed with one earlier Claude run. That is not the same as being right.

Seconds per repo, mean: Gemma 4 26B-A4B 13.9; Qwen 3.6 27B 77.7; Qwen 2.5 14B 25.2
Output speed: Gemma 4 26B-A4B 64 tokens/s; Qwen 3.6 27B 12.9 tokens/s; Qwen 2.5 14B 32.5 tokens/s
Front matter fully valid: Gemma 4 26B-A4B 7 of 12; Qwen 3.6 27B 12 of 12; Qwen 2.5 14B 8 of 12
Verdict (SUBSCRIBE / WATCH / IGNORE) same as Claude: Gemma 4 26B-A4B 6 of 12; Qwen 3.6 27B 8 of 12; Qwen 2.5 14B 7 of 12
Verdicts two steps apart: Gemma 4 26B-A4B 2; Qwen 3.6 27B 0; Qwen 2.5 14B 0
Invented figures or names: Gemma 4 26B-A4B none found; Qwen 3.6 27B none found; Qwen 2.5 14B none found

Gemma is fast and kept to the README, but it took one prompt rule too literally. It named my own project as a "competitor" in 10 of 12 repos (Claude did that for 1), including an Android test tool and a prompt library, and the rule then forced the top verdict. Qwen 2.5's 7 of 12 is false comfort. It never gave the top verdict and wrote WATCH for 10 of 12.

Job 2: Vietnamese news posts

5 real Australian news articles from 4 and 5 November 2026, my full writing rules as the prompt (about 7,400 tokens in, about 1,100 to 1,350 out), output forced to a JSON schema. The comparison is the post Claude Sonnet had already written for the same article.

Seconds per post: Sonnet n/a; Gemma 4 26B-A4B 23.6; Qwen 3.6 27B 123.8; Qwen 2.5 14B 41.6
Valid JSON: Sonnet n/a; Gemma 4 26B-A4B 5 of 5; Qwen 3.6 27B 4 of 5; Qwen 2.5 14B 5 of 5
Ends with the required question: Sonnet 5 of 5; Gemma 4 26B-A4B 5 of 5; Qwen 3.6 27B 5 of 5; Qwen 2.5 14B 0 of 5
Factual errors against the source: Sonnet none seen; Gemma 4 26B-A4B 0 hard, 2 small additions; Qwen 3.6 27B 4 hard; Qwen 2.5 14B many (3 posts read)
Garbled or non-Vietnamese words: Sonnet none seen; Gemma 4 26B-A4B 3; Qwen 3.6 27B 3, one in Chinese; Qwen 2.5 14B 6 or more

The errors are the kind a skim won't catch. Qwen 3.6 gave a named MP a party the article never mentions, wrote the fraud amount in US dollars when the source said Australian dollars, put Chinese characters in the middle of a Vietnamese sentence, and renamed the drug Wegovy to "WEGOZY" in a title. Qwen 2.5 turned A$235 million into Vietnamese dong. Gemma wrote the most natural Vietnamese of the three, and still garbled a word in each of two headings, for example "ĐIỀUNG KIỆN" for "ĐIỀU KIỆN").

Job 3: keep or drop, and the trap in my first sample

On 6 November I gave each model 60 articles, title and description only, 10 per call, with the sorting rules from my project. Half were articles Claude had kept, half ones it had dropped.

Gemma 4 26B-A4B: Agrees with Claude 52 of 60; Kept articles found 28 of 30; all 60 in 33 s
Qwen 3.6 27B: Agrees with Claude 51 of 60; Kept articles found 26 of 30; all 60 in 441 s
Qwen 2.5 14B: Agrees with Claude 41 of 60; Kept articles found 14 of 30; all 60 in 177 s

87% looked good. On 6 November 2026 I ran Gemma on one real day of traffic from the same source, read-only next to Claude: 737 articles, of which Claude kept 50 (7%). Gemma agreed on 83% and found 43 of the 50. It also kept about 120 that Claude dropped. A rule that drops everything would agree 93% of the time, so raw agreement was the wrong number to look at. The half-and-half sample hid how rare a "keep" is.

Used as a pre-filter, where Gemma drops and Claude reads the rest, it would cut Claude's reads to about a fifth and lose 7 of the 50 kept articles (14%). That took 295 s for the 737.

Speed and cold start

I ran the same summary prompt twice back to back. Output speed repeated within about 15%. Loading a 17 GB model took 5 to 20 s. A job that runs once a day pays that load every time. Gemma took 25.0 s on the first run and 9.7 s on the second, which also reused the cached prompt.

Limits

12 repos, 5 posts, 60 plus 737 articles, one run each at temperature 0.3. A one-item difference (6 against 7 of 12) means nothing. The Claude answers I compared against are one earlier run, not ground truth. Thinking was off for the local models. A native Vietnamese reader has not yet compared the Gemma and Sonnet posts side by side.

What I'd do

Keep writing for readers on Claude. For high-volume sorting, Gemma 4 26B-A4B is the one local result worth acting on, as a pre-filter, if losing about one keep in seven is acceptable for your job. Before you trust any agreement score, check it on a sample with the real ratio of keeps to drops.