A Closed Pull Request, a Fork and a PyPI Package: Fast HTML to Markdown for Python

On 15 October I published pyhtml2text on PyPI. It converts HTML to plain text or Markdown and is built on rust-html2text, Chris Emerson’s Rust crate. Converting to Markdown, it was between 3 and 21 times faster than the usual Python converters on the pages I measured. It can also resolve relative links against the page’s URL, so the links in the output still work once they’re away from the page.

pip install pyhtml2text

from pyhtml2text import html2markdown

html = """<h2>Before you start</h2>
<p>Read the <a href="setup.html">setup guide</a> and <b>back up</b> your data.</p>
<img src="/img/diagram.png" alt="How it fits together">
<table><tr><th>Plan</th><th>Price</th></tr><tr><td>Basic</td><td>Free</td></tr></table>"""

print(html2markdown(html, base_url="https://example.com/docs/intro.html"))

## Before you start

Read the [setup guide](https://example.com/docs/setup.html) and **back up** your data.

![How it fits together](https://example.com/img/diagram.png)

| Plan | Price |
| --- | --- |
| Basic | Free |

It started as a pull request to rust-html2text. The maintainer closed it because it was written with an LLM, so pyhtml2text runs on a fork instead. This post covers how that happened, what’s in the fork and the package, and how I checked that the Markdown is right.

Why I needed it

#

maru answers Accept: text/markdown with Markdown instead of HTML, for AI agents and LLM clients that would rather not wade through nav bars and footers. I covered the edge side of that in the post on Cloudflare, crawlers and the cache. Listing, job and community pages build their Markdown from the database, because their HTML is an empty Expo shell. Everything else is real HTML from Hugo (the marketing, legal and blog pages), and that gets converted.

The problem with converting it is the links. A Hugo page is full of links like /en/posts/... and ../setup/. They’re fine in a browser, which knows the page they came from. An agent reading the Markdown has no such context, so it can’t follow /login. It needs https://example.com/login.

The pure-Python html2text has a baseurl option for this, and that’s what I used at first. But it’s slow, and its Markdown has problems I’ll get to below. rust-html2text was fast and well tested, but it had no way to resolve links.

Asking upstream first

#

On 29 August I opened issue #287 with the use case and the behaviour I wanted: RFC 3986 resolution, with absolute URLs and mailto:, data: and #fragment links left alone. A few minutes later I opened pull request #288. It added a Config::base_url(...) option, and I said in the issue that it was a starting point and that I’d understand if the maintainer wanted to take a different approach.

On 11 October the maintainer closed the PR:

Thanks for the contribution, but I’m unable to accept this as it appears to be LLM generated. (I have now added a brief policy to the README.md - it wasn’t previously stated)

and replied on the issue:

Hi - this is a reasonable feature request, thanks! I have closed the PR as it appears to be LLM-written which I can’t accept, but thanks for the attempt.

A short HTML sample and expected output would definitely help get the feature implemented!

The README now has this policy:

At this time, this project is not able to accept code contributions produced with generative AI/LLMs due to the uncertain copyright status.

That was a correct read: I wrote that code with Claude Code. I don’t think the policy is unreasonable, either. The copyright status of LLM output isn’t settled, and it’s the maintainer who would carry the risk of merging it into an MIT crate that other projects depend on. It’s their project, and the reply was polite. The feature request is still open.

I still needed the feature, though, and arguing with a maintainer over their own contribution policy isn’t a good way to get it. So I kept the change on a fork and built the Python package on top of that, and both say clearly that they aren’t upstream.

What’s in the fork

#

The fork is the pyhtml2text-perf branch of my rust-html2text fork, and I’ve kept it small so that it rebases cleanly onto upstream. It adds three commits:

• Config::base_url. This is the change from the PR. Relative links are resolved per RFC 3986 §5.3, in about 200 lines inside the crate rather than through the url crate, because reference resolution is all it needs. Without a base URL the output is unchanged.
• Shared style rules. The crate cloned its stylesheet data into every document it rendered, and config::plain() rebuilt its ten decoration rules on every call. Keeping the rule lists behind an Arc (copy-on-write) and building them once took a short snippet from 7.3 µs to 5.6 µs. That’s nothing on a big page, but it adds up when you call it a few thousand times.
• Exporting the DOM node types and the URL resolver. The Markdown converter described below walks the parsed DOM itself, so it needs those types and has to resolve links the same way the text renderer does.

The Python package

#

pyhtml2text is built with PyO3 and maturin, and it has two functions:

from pyhtml2text import html2text, html2markdown

html2text(html, width=80, *, base_url=None, decorate=True, link_footnotes=True,
          table_borders=True, wrap_links=True, max_wrap_width=None, raw=False)

html2markdown(html, *, base_url=None)

html2text is rust-html2text’s plain-text renderer, which is built for terminals and the text/plain part of an email:

print(html2text('<p>Please <a href="/login">sign in</a> first.</p>',
                base_url="https://example.com/docs/page.html"))

Please [sign in][1] first.

[1]: https://example.com/login

Some things about the bindings that may not be obvious:

• Both functions take str or UTF-8 bytes, so you can pass response.content without decoding it first. The input is borrowed, not copied.
• The GIL is released while converting. It’s pure Rust from start to finish, so threads scale across cores. The module also says it doesn’t need the GIL, so free-threaded Python 3.14t doesn’t turn it back on when the module is imported.
• A panic comes back as ValueError. By default PyO3 raises a PanicException, which except Exception doesn’t catch. A web worker that thinks it has handled conversion errors would crash on that, so the bindings catch it and re-raise it as a ValueError.
• There are 30 wheels: abi3 wheels for CPython 3.10 and later, plus free-threaded 3.14t wheels, across Linux (glibc and musl), macOS and Windows on every architecture maturin’s CI can build. So nobody needs a Rust toolchain to install it.

Making it fast

#

Version 0.2.0 was 1.5 to 1.7 times faster than 0.1.0 on real pages, and twice as fast on short snippets, with byte-for-byte identical output. Three changes did most of that:

• Profile-guided optimisation. CI builds an instrumented wheel, runs it over a training set of Python documentation pages and RFCs, and uses the profile to build the real wheel. That’s about 25% faster on pages outside the training set. It’s only done where GitHub can run the instrumented build natively (Linux x86_64 and aarch64, macOS, Windows x64). The other platforms get the same code without it.
• mimalloc. html2text makes a lot of small allocations, and mimalloc handles them much better than the system allocators. It’s left out on glibc armv7, ppc64le and s390x, where the manylinux2014 cross compilers are too old to build it.
• The shared style rules from the fork.

To check the speed-ups didn’t change anything, a benchmark script hashes the output for every page in a local corpus under eleven option sets, plus edge cases like empty input, 300 levels of nesting and rowspan=0. Each change had to leave every hash the same.

Here are the median times on an Apple Silicon Mac, converting to Markdown:

And converting to plain text:

Deeply nested HTML, and a patch to html5ever

#

Nobody writes 40,000 nested lists by hand. But if you convert pages you didn’t write, sooner or later something hostile or broken will send you one. With the stock parser, 40,000 nested <ul><li> took 6 seconds to parse, and that’s a worker stuck on a single request.

The cause was in html5ever, the HTML5 parser rust-html2text uses. Most block-level start tags have to check whether there’s an open <p> element “in button scope”, so they know whether to close it. html5ever answered that by walking the whole stack of open elements, so the deeper the nesting got, the longer each tag took, and the total time grew with the square of the depth.

The package vendors html5ever 0.39.0 with one patch. The stack of open elements now keeps track of where the topmost <p> and the topmost button-scope boundary are, and updates them on every push, pop and truncate, so the check is O(1). The same document now parses in 0.14 seconds. In debug builds every answer is also checked against the old stack walk. To be sure the patch changed nothing else, I compared the DOMs from the stock and patched parsers on every html5lib tree-construction test, about 20 real pages and 300,000 random tag-soup documents, and they were all identical.

The Markdown converter has its own limits for the same reason. It walks the tree with an explicit stack rather than recursion, so no input can overflow the thread’s stack. It stops indenting blockquotes and lists past 32 levels, because every line repeats the prefixes of everything around it and the output would otherwise grow quadratically. And a table whose expanded grid would pass 100,000 cells is written as its cells alone.

html2markdown

#

Version 0.3.0 added html2markdown, and most of the work in the package went into it. rust-html2text’s renderer makes text that looks like Markdown, but it’s wrapped for a terminal and isn’t meant to be parsed. So html2markdown uses rust-html2text and the patched html5ever only to parse the page, then walks the DOM itself and writes CommonMark, plus GitHub’s pipe tables and ~~strikethrough~~:

• # headings, - bullets, _emphasis_ and **strong**.
• Inline links and images, made absolute when base_url is given. A link whose only content is an icon uses its aria-label or title as its text.
• Fenced code blocks, tagged with the language when the page names one (class="language-python").
• Pipe tables. Cells that span rows or columns become empty cells, a table without a header row gets an empty one, and tables used for page layout are written as paragraphs.
• No line wrapping. Hidden content (hidden, aria-hidden, display: none), scripts, styles and form controls are left out.

Escaping only what needs it

#

Some HTML-to-Markdown converters escape every character that could be Markdown syntax. markdownify, for example, escapes every _ and * by default. The output is correct, but it’s full of snake\_case and 5 \* 3, which is harder for people to read and a waste of tokens for a model. pyhtml2text escapes a character only where a CommonMark parser would otherwise read it as syntax:

html2markdown('<p>Use <code>snake_case</code>, 5 * 3, $5 and '
              '<a href="../setup.html">setup</a></p>',
              base_url="https://example.com/docs/intro.html")

Use `snake_case`, 5 * 3, $5 and [setup](https://example.com/setup.html)

Bold before a Korean particle

#

This one came from maru’s own pages. In Korean the grammatical particle goes straight after the noun with no space, so a bold term with brackets in it looks like this:

<p><b>타이타닉(Titanic)</b>의 감독</p>

The obvious Markdown is **타이타닉(Titanic)**의, which is what markdownify writes, and it doesn’t work. CommonMark’s flanking rules don’t let ** close after punctuation unless a space or more punctuation follows it, and 의 is a letter, so the asterisks stay as literal text. The pure-Python html2text avoids this by adding a space:

**타이타닉(Titanic)** 의 감독

That renders as bold, but it changes the Korean. The particle is now a separate word, which is a spelling mistake. pyhtml2text moves the punctuation outside the markers instead:

**타이타닉(Titanic**)의 감독

The bracket is no longer bold, but the emphasis still parses and every character of the original is kept. The same applies to Japanese and Chinese, and to any language that doesn’t put a space before what follows.

Checking the output by reading it back

#

I don’t trust a converter because its unit tests pass, because the bugs in Markdown output are in combinations nobody thought to write a test for. So the converter checks its own work in two ways.

While converting, each paragraph’s emphasis markers are checked against the flanking rules. The paragraph is then parsed again with pulldown-cmark to confirm it reads back as intended. If it doesn’t, the converter writes it again with the other emphasis style (* instead of _, or the other way round). If neither version passes, it keeps the text, links and code and drops the formatting, so no words are lost.

In the test suite, tests/markdown_check.py does a round trip with a different parser. It converts HTML to Markdown, renders that back to HTML with markdown-it, and compares every visible word, link target and image source with the original. Text eaten by syntax that wasn’t escaped, a link that didn’t parse or a stray * all show up as differences. It runs over real pages and over tens of thousands of random documents, and it can shrink a failing document down to a minimal case.

In production on maru

#

maru’s markdown negotiation switched from the pure-Python html2text to pyhtml2text the day 0.3.0 came out. On one of maru’s guide pages the conversion went from 4.2 ms to 0.36 ms, about 12 times faster. The links are absolute, the tables are real pipe tables, and the Korean bold keeps its particles attached. The abi3 wheels already cover Python 3.14 on x86_64 and arm64, so the Docker image needed nothing new to install it.

Credits, and where to report bugs

#

The plain-text rendering is rust-html2text, by Chris Emerson, under the MIT licence. pyhtml2text adds a fork of it, a patched copy of html5ever (MIT/Apache-2.0) and the Markdown converter. The package isn’t affiliated with or endorsed by either project, so please report problems to pyhtml2text’s issue tracker, not upstream. Any bug in this code is mine to fix, and the upstream maintainers shouldn’t get reports about code they declined to take.

If you’re converting HTML in Python, especially for a model to read, give it a try:

pip install pyhtml2text

The source is on GitHub, and the changelog lists everything in 0.1.0 to 0.3.0.