Turning news pages into clean Markdown for a model

The job is feeding news to a model. A scheduled reader for my project pulls news articles from RSS feeds and hands the article text to a model. It already had a small extractor that reads paragraphs and stops at 6,000 characters. On 5 November 2026 I ran crawl4ai (version 0.9.5) next to it on the same pages to see whether a dedicated crawler does the job better.

What I ran

Five articles taken that day from the feeds the reader already uses: ABC News, Guardian Australia, the Sydney Morning Herald, The Conversation and 7NEWS. Plus two Google News links from the same feed list. crawl4ai ran two ways, through a headless browser and over plain HTTP with no browser, both with its "fit Markdown" noise filter (PruningContentFilter, threshold 0.48).

git clone --depth 1 https://github.com/unclecode/crawl4ai.git repo
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python ./repo          # 10 s, venv ends up 610 MB
.venv/bin/python -m playwright install chromium           # 21 s, 203 MB browser

To compare the two, I took every sentence longer than 60 characters from my extractor's output and counted how many also appeared in crawl4ai's.

What came out

Characters per article, mine (cap removed) against crawl4ai raw and filtered, then coverage:

• ABC: 7,703 against 28,324 raw (207 links) and 9,269 filtered (4 links). Coverage 96%.
• Guardian: 7,808 against 27,629 (197 links) and 12,165 (18 links). Coverage 100%.
• SMH: 7,330 against 15,004 (71 links) and 10,966 (38 links). Coverage 98%.
• The Conversation: 3,293 against 12,288 (75 links) and 4,345 (13 links). Coverage 100%.

• 7NEWS: 6,848 against 24,745 (138 links) and 7,968 (5 links). Coverage 100%.

• Both found the article body. The filtered output kept 96 to 100% of what my extractor kept.

• crawl4ai kept text mine dropped. The ABC story lists twelve diverted flights as bullet points. My extractor reads paragraphs only, so all twelve were missing. crawl4ai kept them, plus the headline and subheadings.
• It also kept more junk: the Guardian's ten "related stories" headlines, the SMH footer and subscription links, photo captions and "Share to Facebook" text.
• Without a browser, crawl4ai ran at about the same speed as mine (0.31 to 0.74 s a page). Starting the browser added about 2 s.
• My own cap was the bigger problem. Four of the five articles ran 6,800 to 7,800 characters, so the 6,000-character cap cut the last 12 to 23% of each story.

What broke

• Browser mode hung on 7NEWS. It waited 60 seconds for the page to finish loading and gave up, twice. Plain HTTP got the same page in under a second.
• Google News links still failed. I hoped a real browser would follow them through to the publisher. Even after waiting 5 s for the network to go quiet, it landed on a Chrome error page, and crawl4ai's own detector flagged it as an anti-bot block.
• It is heavy. About 610 MB of Python packages before the 203 MB browser. If your app isn't Python, that means a second process next to it.
• It advertises on import. Every import prints a banner for their hosted version unless you set CRAWL4AI_NO_CLOUD_NOTICE=1.

What I kept

Not the library. Its filter scores each block by how much of its text is link text, and that idea moved into my own extractor: read list items as well as paragraphs, and skip any item where more than half the text is links. Tested on a scratch copy against the same five pages, it recovered the twelve flight lines, dropped two newsletter sign-up lines on the Guardian, and left the other three articles unchanged.

crawl4ai's block detector is also worth reading: a short list of patterns that recognise Cloudflare, Akamai, Imperva and similar block pages that come back with status 200. A plain extractor can't tell a blocked page from an empty one.

Pick crawl4ai if you are in Python and need pages built by JavaScript, or a whole-site crawl. For news sites that serve plain HTML, a small extractor did the same job with less noise.