A Closed Pull Request, a Fork and a PyPI Package: Fast HTML to Markdown for Python

On 15 October I published pyhtml2text on PyPI. It converts HTML to plain text or Markdown and is built on rust-html2text, Chris Emerson’s Rust crate. Converting to Markdown, it was between 3 and 21 times faster than the usual Python converters on the pages I measured. It can also resolve relative links against the page’s URL, so the links in the output still work once they’re away from the page.

pip install pyhtml2text

from pyhtml2text import html2markdown

html = """<h2>Before you start</h2>
<p>Read the <a href="setup.html">setup guide</a> and <b>back up</b> your data.</p>
<img src="/img/diagram.png" alt="How it fits together">
<table><tr><th>Plan</th><th>Price</th></tr><tr><td>Basic</td><td>Free</td></tr></table>"""

print(html2markdown(html, base_url="https://example.com/docs/intro.html"))

## Before you start

Read the [setup guide](https://example.com/docs/setup.html) and **back up** your data.

![How it fits together](https://example.com/img/diagram.png)

| Plan | Price |
| --- | --- |
| Basic | Free |

It started as a pull request to rust-html2text. The maintainer closed it because it was written with an LLM, so pyhtml2text runs on a fork instead. This post covers how that happened, what’s in the fork and the package, and how I checked that the Markdown is right.

Why I needed it

#

A marketplace and community app I build answers Accept: text/markdown with Markdown instead of HTML, for AI agents and LLM clients that would rather not wade through nav bars and footers. I covered the edge side of that in the post on crawlers and the cache. Pages that come from app data build their Markdown directly, because their HTML is an empty JavaScript shell. Everything else is static HTML (the marketing, legal and blog pages), and that gets converted.

The problem with converting it is the links. A static site is full of links like /en/posts/... and ../setup/. They’re fine in a browser, which knows the page they came from. An agent reading the Markdown has no such context, so it can’t follow /login. It needs https://example.com/login.

The pure-Python html2text has a baseurl option for this, and that’s what I used at first. But it’s slow, and its Markdown has problems I’ll get to below. rust-html2text was fast and well tested, but it had no way to resolve links.

Asking upstream first

#

On 29 August I opened issue #287 with the use case and the behaviour I wanted: RFC 3986 resolution, with absolute URLs and mailto:, data: and #fragment links left alone. A few minutes later I opened pull request #288. It added a Config::base_url(...) option, and I said in the issue that it was a starting point and that I’d understand if the maintainer wanted to take a different approach.

On 11 October the maintainer closed the PR:

Thanks for the contribution, but I’m unable to accept this as it appears to be LLM generated. (I have now added a brief policy to the README.md - it wasn’t previously stated)

and replied on the issue:

Hi - this is a reasonable feature request, thanks! I have closed the PR as it appears to be LLM-written which I can’t accept, but thanks for the attempt.

A short HTML sample and expected output would definitely help get the feature implemented!

The README now has this policy:

At this time, this project is not able to accept code contributions produced with generative AI/LLMs due to the uncertain copyright status.

That was a correct read: I wrote that code with an AI coding assistant. I don’t think the policy is unreasonable, either. The copyright status of LLM output isn’t settled, and it’s the maintainer who would carry the risk of merging it into an MIT crate that other projects depend on. It’s their project, and the reply was polite. The feature request is still open.

I still needed the feature, though, and arguing with a maintainer over their own contribution policy isn’t a good way to get it. So I kept the change on a fork and built the Python package on top of that, and both say clearly that they aren’t upstream.

What’s in the fork

#

The fork is the pyhtml2text-perf branch of my rust-html2text fork. I’ve kept it to three small commits so that it rebases cleanly onto upstream. The main one is the Config::base_url option from the PR: relative links are resolved per RFC 3986 §5.3, and without a base URL the output is unchanged. Another stops the crate rebuilding its style rules on every call, which took a short snippet from 7.3 µs to 5.6 µs. That’s nothing on a big page, but it adds up when you call it a few thousand times. The third just exposes what the Markdown converter below needs from the crate.

The Python package

#

pyhtml2text is built with PyO3 and maturin, and it has two functions:

from pyhtml2text import html2text, html2markdown

html2text(html, width=80, *, base_url=None, decorate=True, link_footnotes=True,
          table_borders=True, wrap_links=True, max_wrap_width=None, raw=False)

html2markdown(html, *, base_url=None)

html2text is rust-html2text’s plain-text renderer, which is built for terminals and the text/plain part of an email:

print(html2text('<p>Please <a href="/login">sign in</a> first.</p>',
                base_url="https://example.com/docs/page.html"))

Please [sign in][1] first.

[1]: https://example.com/login

Some things about the bindings that may not be obvious:

• Both functions take str or UTF-8 bytes, so you can pass response.content without decoding it first. The input is borrowed, not copied.
• The GIL is released while converting. It’s pure Rust from start to finish, so threads scale across cores. The module also says it doesn’t need the GIL, so free-threaded Python 3.14t doesn’t turn it back on when the module is imported.
• A panic comes back as ValueError. By default PyO3 raises a PanicException, which except Exception doesn’t catch. A web worker that thinks it has handled conversion errors would crash on that, so the bindings catch it and re-raise it as a ValueError.
• There are 30 wheels: abi3 wheels for CPython 3.10 and later, plus free-threaded 3.14t wheels, across Linux (glibc and musl), macOS and Windows on every architecture maturin’s CI can build. So nobody needs a Rust toolchain to install it.

Making it fast

#

Version 0.2.0 was 1.5 to 1.7 times faster than 0.1.0 on real pages, and twice as fast on short snippets, with byte-for-byte identical output. Most of that came from profile-guided optimisation: CI builds an instrumented wheel, runs it over a training set of documentation pages and RFCs, and uses the profile to build the real wheel. That’s about 25% faster on pages outside the training set, on the platforms where CI can run the instrumented build natively. The rest came from switching to the mimalloc allocator, which copes far better with lots of small allocations, and from the style-rule change in the fork.

To check the speed-ups didn’t change anything, I hashed the output for a corpus of real pages under every set of options, plus edge cases like empty input and very deep nesting. Each change had to leave every hash the same.

Here are the median times on an Apple Silicon Mac, converting to Markdown:

And converting to plain text:

Deeply nested HTML, and a patch to html5ever

#

Nobody writes 40,000 nested lists by hand. But if you convert pages you didn’t write, sooner or later something hostile or broken will send you one. With the stock parser, 40,000 nested <ul><li> took 6 seconds to parse, and that’s a worker stuck on a single request.

The cause was in html5ever, the HTML5 parser rust-html2text uses. Most block-level start tags have to check whether there’s an open <p> element “in button scope”, and html5ever answered that by walking the whole stack of open elements. The deeper the nesting, the longer each tag took, so the total time grew with the square of the depth.

The package ships html5ever 0.39.0 with one patch that keeps the answer up to date as elements are opened and closed, so the check no longer depends on depth. The same document now parses in 0.14 seconds. To be sure the patch changed nothing else, I compared the DOMs from the stock and patched parsers on every html5lib tree-construction test, about 20 real pages and 300,000 random tag-soup documents, and they were all identical.

The Markdown converter has its own limits for the same reason. No input can overflow the thread’s stack, blockquotes and lists stop indenting past 32 levels so the output can’t grow quadratically, and a table whose expanded grid would pass 100,000 cells is written as its cells alone.

html2markdown

#

Version 0.3.0 added html2markdown, and most of the work in the package went into it. rust-html2text’s renderer makes text that looks like Markdown, but it’s wrapped for a terminal and isn’t meant to be parsed. So html2markdown only borrows the parser and writes its own CommonMark, plus GitHub-style pipe tables and ~~strikethrough~~:

• # headings, - bullets, _emphasis_ and **strong**.
• Inline links and images, made absolute when base_url is given. A link whose only content is an icon uses its aria-label or title as its text.
• Fenced code blocks, tagged with the language when the page names one (class="language-python").
• Pipe tables. Cells that span rows or columns become empty cells, a table without a header row gets an empty one, and tables used for page layout are written as paragraphs.
• No line wrapping. Hidden content (hidden, aria-hidden, display: none), scripts, styles and form controls are left out.

Escaping only what needs it

#

Some HTML-to-Markdown converters escape every character that could be Markdown syntax. markdownify, for example, escapes every _ and * by default. The output is correct, but it’s full of snake\_case and 5 \* 3, which is harder for people to read and a waste of tokens for a model. pyhtml2text escapes a character only where a CommonMark parser would otherwise read it as syntax:

html2markdown('<p>Use <code>snake_case</code>, 5 * 3, $5 and '
              '<a href="../setup.html">setup</a></p>',
              base_url="https://example.com/docs/intro.html")

Use `snake_case`, 5 * 3, $5 and [setup](https://example.com/setup.html)

Bold before a Korean particle

#

This one came from my app’s own pages. In Korean the grammatical particle goes straight after the noun with no space, so a bold term with brackets in it looks like this:

<p><b>타이타닉(Titanic)</b>의 감독</p>

The obvious Markdown is **타이타닉(Titanic)**의, which is what markdownify writes, and it doesn’t work. CommonMark’s flanking rules don’t let ** close after punctuation unless a space or more punctuation follows it, and 의 is a letter, so the asterisks stay as literal text. The pure-Python html2text avoids this by adding a space:

**타이타닉(Titanic)** 의 감독

That renders as bold, but it changes the Korean. The particle is now a separate word, which is a spelling mistake. pyhtml2text moves the punctuation outside the markers instead:

**타이타닉(Titanic**)의 감독

The bracket is no longer bold, but the emphasis still parses and every character of the original is kept. The same applies to Japanese and Chinese, and to any language that doesn’t put a space before what follows.

Checking the output by reading it back

#

I don’t trust a converter because its unit tests pass, because the bugs in Markdown output are in combinations nobody thought to write a test for. So the output gets checked by reading it back.

While converting, each paragraph is parsed again as CommonMark to confirm the emphasis reads as intended. If it doesn’t, the paragraph is written again with the other emphasis style (* instead of _, or the other way round), and if neither works the text, links and code are kept without the formatting, so no words are lost. The test suite does a round trip through a different Markdown parser and compares every visible word, link target and image source with the original HTML. Text eaten by syntax that wasn’t escaped, a link that didn’t parse or a stray * all show up as differences. It runs over real pages and tens of thousands of random documents, and shrinks any failure down to a minimal case.

In production

#

My app’s Markdown negotiation switched from the pure-Python html2text to pyhtml2text the day 0.3.0 came out. On one of its guide pages the conversion went from 4.2 ms to 0.36 ms, about 12 times faster. The links are absolute, the tables are real pipe tables, and the Korean bold keeps its particles attached. The abi3 wheels already cover Python 3.14 on x86_64 and arm64, so the container image needed nothing new to install it.

Credits, and where to report bugs

#

The plain-text rendering is rust-html2text, by Chris Emerson, under the MIT licence. pyhtml2text adds a fork of it, a patched copy of html5ever (MIT/Apache-2.0) and the Markdown converter. The package isn’t affiliated with or endorsed by either project, so please report problems to pyhtml2text’s issue tracker, not upstream. Any bug in this code is mine to fix, and the upstream maintainers shouldn’t get reports about code they declined to take.

If you’re converting HTML in Python, especially for a model to read, give it a try:

pip install pyhtml2text

The source is on GitHub, and the changelog lists everything in 0.1.0 to 0.3.0.