mdream - Moses Yebei

The Problem

HTML to Markdown conversion is often a slog. Traditional converters are slow and produce output that’s bloated and not optimized for Large Language Models (LLMs). If you're working with text-heavy applications or need to prepare data for LLMs, you need something faster and leaner. Enter mdream, designed to tackle these issues head-on.

What This Does

mdream is a lightning-fast HTML to Markdown converter built in JavaScript. It’s not just another tool in the toolbox; it’s a finely-tuned machine. The core of the project lives in packages/mdream/src/index.ts, where the actual conversion magic happens. It generates GitHub Flavored Markdown with support for frontmatter and nested HTML, making it versatile for various use cases.

The bench/ directory contains benchmark tests that show just how quick it is—converting 1.8MB of HTML in around 60ms. You can check out bench/README.md for the methodology behind those impressive numbers. Plus, the packages/mdream/src/plugins.ts file allows for easy extensibility with a plugin system, meaning you can adapt it to fit your unique needs.

Real-World Use

Imagine you’re pulling data from a web scraper and need to convert a hefty amount of HTML into Markdown for your documentation site. You could use the CLI tool available in packages/mdream/bin/mdream.mjs to pipe the HTML directly from stdin, like so:

cat your-file.html | node packages/mdream/bin/mdream.mjs > output.md

This command will process the HTML and spit out a Markdown file, ready for use. You could also integrate it into a CI pipeline, leveraging the GitHub Actions in .github/workflows/ for automated documentation updates.

The Bottom Line

mdream is a solid choice if you need a fast, lightweight HTML to Markdown converter tailored for LLMs. It’s not overkill for small projects but shines in larger applications where speed and efficiency matter. If you’re tired of the sluggishness and bloat of existing converters, give this one a shot. Just be prepared for the slight learning curve if you want to dive deep into its plugin system.

What the analyser found

Deployment readiness

2/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

0 files

TypeScript84
Vue31
JSON21
Markdown12
JavaScript9
YAML7

Architecture

Top-level areas of the codebase, sized by module count. Arrows show how many imports cross from one area into another.

graph TD g0["packages 128 modules"] g1["docs 26 modules"] g2["examples 13 modules"] g3["bench 10 modules"] g4["(root) 2 modules"] g3 -->|2| g0 g2 -->|1| g0

Module dependencies

The 10 most depended-upon modules of 180, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_packages_mdream_src_index["index"] n1_packages_mdream_src_types["types"] n2_packages_mdream_src_const["const"] n3__mdream_src_pluggable_plugin["plugin"] n4_packages_vite_src_plugin["plugin"] n5_packages_crawl_src_types["types"] n6_packages_mdream_src_parse["parse"] n7_packages_llms_db_src_types["types"] n8_dream_src_plugins_extraction["extraction"] n9_packages_llms_db_src_schema["schema"] n7_packages_llms_db_src_types --> n9_packages_llms_db_src_schema n0_packages_mdream_src_index --> n1_packages_mdream_src_types n0_packages_mdream_src_index --> n2_packages_mdream_src_const n0_packages_mdream_src_index --> n6_packages_mdream_src_parse n0_packages_mdream_src_index --> n3__mdream_src_pluggable_plugin n6_packages_mdream_src_parse --> n1_packages_mdream_src_types n6_packages_mdream_src_parse --> n2_packages_mdream_src_const n3__mdream_src_pluggable_plugin --> n1_packages_mdream_src_types n8_dream_src_plugins_extraction --> n1_packages_mdream_src_types n8_dream_src_plugins_extraction --> n3__mdream_src_pluggable_plugin n1_packages_mdream_src_types --> n8_dream_src_plugins_extraction classDef cyc stroke:#ef4444,stroke-width:2px; class n1_packages_mdream_src_types,n3__mdream_src_pluggable_plugin,n8_dream_src_plugins_extraction cyc;