mdream - Moses Yebei

The Problem Many HTML‑to‑Markdown libraries are generic, heavyweight, and generate output that wastes tokens when fed to LLMs. Teams that need fast, low‑overhead conversion for prompt construction end up writing ad‑hoc parsers or paying the performance penalty of larger converters.

What This Does mdream provides a zero‑dependency, pure‑JS converter tuned for LLM token efficiency. The core lives in packages/mdream/src/index.ts (the hub module, imported by 39 other files) and delegates to a small set of helpers:

packages/mdream/src/parse.ts – tokenises and walks the DOM, producing Markdown.
packages/mdream/src/markdown-processor.ts – applies minimal GitHub‑flavoured rules.
packages/mdream/src/pluggable/plugin.ts and the plugins/ folder – optional extensions (e.g., plugins/extraction.ts).

A thin CLI (packages/mdream/src/cli.ts) reads from stdin or a file and streams the result, exposed via the executable bin/mdream.mjs. The same core is re‑exported by the top‑level package.json for consumption as a library.

How It Is Wired

bin/mdream.mjs
   → packages/mdream/src/cli.ts
        → packages/mdream/src/index.ts   (hub)
            → packages/mdream/src/parse.ts
                → packages/mdream/src/libs/query-selector.ts
            → packages/mdream/src/markdown-processor.ts
            → plugins/* (optional)
                → packages/mdream/src/pluggable/plugin.ts

Execution starts at bin/mdream.mjs, which parses CLI args and calls run() in cli.ts.
run() imports index.ts; this module orchestrates conversion by calling parse() (in parse.ts) and then feeding the AST to markdownProcessor() (in markdown-processor.ts).
If any plugin is registered (via plugins.ts), the plugin processor (plugin-processor.ts) injects additional node handlers before the final Markdown string is emitted.
The only side‑effects are I/O (reading HTML, writing Markdown) and optional network calls made by plugins (e.g., plugins/extraction.ts).

The import graph shows packages/mdream/src/index.ts as the most‑connected hub (Ca = 39, Ce = 6). Four modules form a cycle (types.ts, pluggable/plugin.ts, plugins/extraction.ts, plugins.ts), increasing cognitive load for changes. The largest single file is parse.ts (≈694 LOC), a natural candidate for refactoring into smaller, testable units.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/mdream
cd mdream

# Install dependencies (pnpm is the lockfile manager)
pnpm install

# Run the CLI on a local HTML file
node bin/mdream.mjs < path/to/page.html > page.md

# Or use the library in code
npm install mdream   # publishes the core package

For containerised deployment, build the published Docker image:

docker build -t mdream:latest .
docker run --rm -i mdream:latest < page.html > page.md

Real‑World Use A content‑generation pipeline can fetch a rendered article, pipe it through mdream to obtain a compact Markdown prompt, and feed the result directly to an LLM. Example (Node):

import { convert } from 'mdream';
import fetch from 'node-fetch';

async function promptFromUrl(url) {
  const html = await (await fetch(url)).text();
  const md = await convert(html);   // core library call
  return md;
}

Code Health & Issues

High – Import cycles (packages/mdream/src/types.ts, pluggable/plugin.ts, plugins/extraction.ts). Break the cycle by extracting shared types.
High – Hub module (packages/mdream/src/index.ts). Stabilise; avoid adding volatile logic here.
High – Oversized files (parse.ts, test/unit/templates/nav.test.ts, tags.ts). Split into focused modules.
High – Duplicated code in bench/bundle/*.ts. Consolidate helpers.
High – Deep nesting in test/unit/templates/nav.test.ts. Refactor with early returns.
Medium – High branching density in packages/crawl/src/glob-utils.ts, llms-txt.ts, libs/query-selector.ts. Consider strategy tables.

CI / DevOps

Pin GitHub Action versions to commit SHAs.
Add a test step to the existing workflows so CI actually runs the 107 test files.
Declare least‑privilege permissions: for GITHUB_TOKEN.
Enable Dependabot (.github/dependabot.yml).
Pin the Docker base image by digest.
Add a dependency‑vulnerability scan (e.g., dependency-review-action).
Set persist-credentials: false on checkout steps.
Define timeout-minutes on all workflow jobs.

The Bottom Line mdream delivers a lightweight, LLM‑aware HTML‑to‑Markdown converter with a clear CLI and library API, but the core module is a high‑impact hub surrounded by import cycles and several large files that raise maintenance risk. Addressing the measured code‑health issues—especially the cycles and oversized modules—will make the package more robust for production use. Teams needing fast, token‑efficient conversion can adopt it now, provided they budget time for the suggested refactors and CI hardening.

What the analyser found

Deployment readiness

7/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

338 files

TypeScript172
Vue41
JSON30
Markdown24
HTML15
YAML9

ReactVueExpressDocker

Architecture

Top-level areas of the codebase, sized by module count. Arrows show how many imports cross from one area into another.

graph TD g0["packages 128 modules"] g1["docs 26 modules"] g2["examples 13 modules"] g3["bench 10 modules"] g4["(root) 2 modules"] g3 -->|2| g0 g2 -->|1| g0

Module dependencies

The 10 most depended-upon modules of 180, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_packages_mdream_src_index["index"] n1_packages_mdream_src_types["types"] n2_packages_mdream_src_const["const"] n3__mdream_src_pluggable_plugin["plugin"] n4_packages_vite_src_plugin["plugin"] n5_packages_crawl_src_types["types"] n6_packages_mdream_src_parse["parse"] n7_packages_llms_db_src_types["types"] n8_dream_src_plugins_extraction["extraction"] n9_packages_llms_db_src_schema["schema"] n7_packages_llms_db_src_types --> n9_packages_llms_db_src_schema n0_packages_mdream_src_index --> n1_packages_mdream_src_types n0_packages_mdream_src_index --> n2_packages_mdream_src_const n0_packages_mdream_src_index --> n6_packages_mdream_src_parse n0_packages_mdream_src_index --> n3__mdream_src_pluggable_plugin n6_packages_mdream_src_parse --> n1_packages_mdream_src_types n6_packages_mdream_src_parse --> n2_packages_mdream_src_const n3__mdream_src_pluggable_plugin --> n1_packages_mdream_src_types n8_dream_src_plugins_extraction --> n1_packages_mdream_src_types n8_dream_src_plugins_extraction --> n3__mdream_src_pluggable_plugin n1_packages_mdream_src_types --> n8_dream_src_plugins_extraction classDef cyc stroke:#ef4444,stroke-width:2px; class n1_packages_mdream_src_types,n3__mdream_src_pluggable_plugin,n8_dream_src_plugins_extraction cyc;

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

high2

medium5

low1

Pin third-party GitHub Actions to a commit SHAhigh5 occurrences

.github/workflows

docker/setup-buildx-action@v3, docker/login-action@v3, docker/metadata-action@v5, docker/build-push-action@v5

A tag can be moved, so the action running with your token and secrets is whatever its owner last pushed; this is how tj-actions/changed-files leaked secrets from thousands of repos.

Fix: Replace each @vN with the 40-character commit SHA, keep # vN as a comment, and let Dependabot bump the SHAs.

Make CI invoke the test suite it hashigh107 occurrences

.github/workflows

107 test files, no test command in the 4 workflows read

A green check that never executed an assertion is worse than no check, because reviewers trust it.

Fix: Add a test step to the existing workflow rather than a new workflow.

Declare least-privilege permissions for GITHUB_TOKENmedium

.github/workflows/test-action.yml

1 workflow(s) declare no permissions

With no declaration the token inherits the repository default, so any injected step can push commits or mint releases from inside your own CI.

Fix: Add permissions: contents: read at the top of the workflow and widen per job only where needed.

Enable Dependabot or Renovatemedium

8 manifest(s), no update bot configured

Without a bot a published advisory sits unpatched until someone audits by hand, which across 1,322 repositories means never.

Fix: Commit .github/dependabot.yml covering the repo ecosystems plus github-actions.

Pin the container base image by digestmedium

Dockerfile

apify/actor-node-playwright-chrome:20

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Gate pull requests on a dependency vulnerability scanmedium

.github/workflows

no dependency scan in CI

This is the one gate that would catch a known-vulnerable package before it reaches a build, and no repository in the sample had it.

Fix: Add dependency-review-action on pull_request, or osv-scanner on push and a schedule.

Set persist-credentials: false on checkoutmedium

.github/workflows/release-patch.yml

checkout keeps the token, then dependencies are installed

The token stays in .git/config for every later step, so a malicious postinstall script reads a pushable credential without one ever being passed to it.

Fix: Add with: persist-credentials: false, and pass an explicit token only to the step that pushes.

Set timeout-minutes on the workflow jobslow4 occurrences

.github/workflows/release-docker.yml

4 workflow(s) declare no job timeout

A wedged step runs to the six-hour platform default, which on a two-hourly schedule means three runs overlap behind it.

Fix: Add timeout-minutes with a realistic bound to each job.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.