When you feed a web page to a language model, the useful part is usually a small share of the HTML. The rest is navigation, footers, scripts, inline styles and consent pop-ups. All of it costs tokens and adds noise to retrieval.
POST /v1/extract loads the page in a real browser, keeps the main content and returns it as Markdown, together with the metadata you need to cite the source.
The request#
curl https://rendersbolt.com/v1/extract \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://en.wikipedia.org/wiki/Headless_browser"}'The answer is JSON:
{
"url": "https://en.wikipedia.org/wiki/Headless_browser",
"title": "Headless browser",
"output": "markdown",
"content": "A **headless browser** is a web browser without a graphical user interface...",
"metadata": {
"description": "...",
"language": "en",
"canonical_url": "https://en.wikipedia.org/wiki/Headless_browser",
"image": "...",
"word_count": 812
}
}content is Markdown with headings, lists, links, tables and images. word_count is a quick way to skip pages that came back nearly empty.
How the main content is found#
The page is rendered first, so text that JavaScript adds is there. Then a readability pass, the same approach Firefox Reader View uses, picks the article and drops navigation, sidebars, footers and ads. If a page has no clear article, for example a pricing page or a dashboard, you get the body without nav, header, footer, aside and forms instead.
To keep the whole page, send "only_main_content": false.
Options that matter for LLM input#
| Option | Default | Use it when |
|---|---|---|
output |
markdown |
text gives plain text without formatting, html gives the cleaned HTML |
include_images |
true |
Set false for text-only models to save tokens |
include_links |
false |
true adds a links array with every absolute URL on the page, useful for crawling one level deeper |
wait_until |
load |
networkidle for pages that fetch their content after load |
wait_for_selector |
Wait for a specific element, such as article or .post-body |
|
block_cookie_banners |
false |
Keeps consent text out of your content |
For single-page apps, combine "wait_until": "networkidle" with "wait_for_selector": "main" so the text is on the page before extraction starts.
Chunking for retrieval#
Because the output keeps the document's headings, you can split on them instead of cutting every N characters:
import os, re, requests
r = requests.post(
"https://rendersbolt.com/v1/extract",
headers={"Authorization": f"Bearer {os.environ['API_KEY']}"},
json={"url": "https://en.wikipedia.org/wiki/Headless_browser", "include_images": False},
)
r.raise_for_status()
page = r.json()
# one chunk per section, each with the page title and source for citations
sections = re.split(r"\n(?=## )", page["content"])
chunks = [
{"text": f"# {page['title']}\n\n{s.strip()}", "source": page["metadata"]["canonical_url"] or page["url"]}
for s in sections if s.strip()
]Sections that are still too long can be split again on ### headings or on paragraphs.
Cost and caching#
Each successful extract is one render, the same as a screenshot or a PDF. If your pipeline may ask for the same page several times, for example while re-indexing, send "cache": true with a cache_ttl in seconds, up to 7 days. Identical requests then return the stored result.
What it can't read#
- Pages behind a login, and paywalled text that isn't in the page for anonymous visitors.
- Sites that show an interactive bot challenge before the content.
- Private network addresses, localhost and cloud metadata endpoints. These are always refused, including through redirects.
Only fetch pages you are allowed to process, and check each site's terms for automated access.
Every option is listed in the extract reference.