Markdown for LLMs and RAG

Recipes for clean page text for language models, JavaScript-heavy pages, chunking by heading and following links.

Clean text, fewer tokens#

{ "url": "https://example.com/article", "include_images": false, "block_cookie_banners": true }

include_images: false drops image Markdown, which models without vision don't need.

Pages that load their content with JavaScript#

{ "url": "https://app.example.com/changelog", "wait_until": "networkidle", "wait_for_selector": "main article" }

Chunk by heading#

import re

page = r.json()
sections = re.split(r"\n(?=## )", page["content"])
chunks = [
    {"text": f"# {page['title']}\n\n{s.strip()}", "source": page["metadata"].get("canonical_url") or page["url"]}
    for s in sections if s.strip()
]

const page = await res.json();
const chunks = page.content
  .split(/\n(?=## )/)
  .filter((s) => s.trim())
  .map((s) => ({ text: `# ${page.title}\n\n${s.trim()}`, source: page.metadata.canonical_url || page.url }));

Follow links one level deep#

Send "include_links": true to get every absolute link on the page in links, then extract the ones you need. Only fetch pages you are allowed to process.

Skip empty pages#

Check metadata.word_count before indexing. A very low count usually means a login wall, a bot challenge or an app shell that never rendered.

Read the longer guide on the blog: Clean Markdown from any web page for your LLM or RAG pipeline.

Stuck? Write to [email protected] with the request you sent and the error code you got back.