Stop Letting AI Agents Read Truncated Web Pages — Use agent-fetch

Your AI agent just tried to read a critical research paper. It got back three paragraphs and a paywall notice. Sound familiar?

Here's the dirty secret plaguing every developer building with LLMs: standard web fetch is fundamentally broken for AI workflows. When your agent calls curl, wget, or some built-in HTTP helper, servers don't see a browser. They see a bot. And they respond accordingly — with truncated content, CAPTCHA challenges, or completely different pages than what humans see.

The result? Your RAG pipeline ingests garbage. Your NotebookLM sources are useless. Your carefully crafted prompts get fed summaries instead of substance. You've built an intelligent system on a foundation of sand.

But what if your AI agent could browse like a real human? What if it could bypass fingerprinting, impersonate Chrome's TLS signature, and extract complete articles with every heading, paragraph, and link intact?

Enter agent-fetch — the open-source weapon that's making cloud extraction APIs obsolete and giving developers back control over their data pipelines.

What is agent-fetch?

agent-fetch is a full-content web fetcher designed specifically for AI agents and content workflows. Created by Teng Lin and published under the MIT license, this Node.js tool solves a problem that seems simple but has tormented developers for years: getting the actual content of web pages, not what servers serve to bots.

The repository lives at github.com/teng-lin/agent-fetch and has been gaining serious traction among developers building RAG systems, AI research tools, and automated content pipelines. Unlike cloud-based extraction services that demand API keys and ship your data to third parties, agent-fetch runs entirely locally. No subscriptions. No rate limits. No privacy concerns.

Why is this trending now? Three forces have converged:

• AI agents are proliferating — Claude Code, Codex, Cursor, and Copilot all need to read URLs, and their built-in fetchers are embarrassingly limited
• Anti-bot detection has escalated — Cloudflare, DataDome, and custom WAFs now fingerprint TLS handshakes, not just User-Agent strings
• RAG quality demands have risen — Garbage in, garbage out. Truncated content destroys embedding quality and retrieval accuracy

agent-fetch addresses all three by combining browser impersonation (via the httpcloak library) with seven parallel extraction strategies. When servers inspect your client's network fingerprint, they see Chrome 143, not a script. When the HTML arrives, multiple algorithms race to extract the cleanest, most complete content.

Key Features That Change Everything

Browser Impersonation & TLS Fingerprinting

This isn't your grandma's User-Agent spoofing. agent-fetch replicates Chrome's complete TLS fingerprint — cipher suites, extensions, signature algorithms, even the exact byte order. Modern anti-bot systems inspect these cryptographic handshakes. agent-fetch passes inspection.

Multiple presets available: chrome-143 (default), ios-safari-18, and more. Switch fingerprints for different targeting scenarios.

Seven Parallel Extraction Strategies

No single extraction method works everywhere. agent-fetch runs all of these simultaneously and intelligently selects the winner:

Winner selection logic: Strategies extracting 500+ characters become candidates. If text-density or RSC finds 2x more content than Readability, it wins. Otherwise, longest result prevails. Metadata is composed from the best source per field across all strategies.

Multi-Output Formats

Get exactly what you need: markdown (structured with headings, links, lists), plain text, raw HTML, or full JSON with metadata. The default output includes title, author, site, publication date, language, and fetch latency.

Advanced Crawling & Session Management

Crawl entire sites with depth control, concurrency limits, URL pattern matching, and rate limiting. Maintain authenticated sessions via inline cookies or Netscape cookie files exported from your browser.

Zero Dependencies on External APIs

Runs entirely on your infrastructure. No API keys to manage, no quota anxiety, no vendor lock-in, no data leaving your network.

Real-World Use Cases Where agent-fetch Dominates

1. RAG Pipelines That Actually Retrieve

You're building a documentation Q&A system. Your vector database is only as good as the chunks you feed it. Standard fetchers return HTML soup or truncated summaries. agent-fetch delivers clean markdown with preserved structure — headings become semantic anchors, links maintain context, lists stay intact. Your embeddings improve. Your retrieval improves. Your answers improve.

2. NotebookLM Sources That Work

Google's NotebookLM refuses many URLs outright. When it succeeds, you often get a useless stub. With agent-fetch, extract the complete article as clean text, paste it directly as a source, and unlock NotebookLM's full analytical power on content that was previously inaccessible.

3. LLM Conversations with Full Context

Your agent needs to analyze a 5,000-word technical analysis. Built-in fetch returns 300 words and "...". With agent-fetch, the entire article enters your context window with proper formatting. The LLM can cite specific sections, follow argument structures, and perform genuine analysis rather than hallucinating around gaps.

4. Competitive Intelligence & Research Automation

Monitor competitor blogs, pricing pages, and documentation changes. Crawl with authenticated sessions to access gated content. Extract structured data from JSON-LD for automated comparison. The --delay flag keeps you respectful; the --concurrency flag keeps you efficient.

5. Content Migration & Archival

Migrate a WordPress site? The REST API strategy pulls clean content directly. Archive a research publication? Browser impersonation gets past access controls that block scripts. Need specific sections only? CSS selectors target exactly what matters.

Step-by-Step Installation & Setup Guide

Prerequisites

agent-fetch requires Node.js 20, 22, or 25. Verify your version:

node --version

Installation Options

Option 1: Global installation via npm

# Install globally for system-wide access
npm install -g @teng-lin/agent-fetch

# Or install locally in your project
npm install @teng-lin/agent-fetch

Option 2: Run without installation (recommended for quick tests)

npx agent-fetch https://example.com/page

Option 3: AI Agent Integration via Agent Skill

For Claude Code, Codex, Cursor, or Copilot users, install the skill for automatic URL handling:

npx skills add teng-lin/agent-fetch

This teaches your agent when and how to invoke agent-fetch — zero configuration required. When the agent encounters a URL, it automatically routes through agent-fetch instead of its broken built-in fetcher.

First Extraction Test

Verify installation with a live extraction:

npx agent-fetch https://example.com/article

Expected output format:

Title: Page Title
Author: Author Name
Site: example.com
Published: 2025-01-26T12:00:00Z
Language: en
Fetched in 523ms
---
# Heading

Full content with **formatting**, [links](https://example.com), and structure preserved...

Environment Configuration

No API keys needed. No .env files required. However, for authenticated content, export cookies from your browser using the Get cookies.txt Locally Chrome extension, then:

npx agent-fetch https://members-only.example.com/article --cookie-file ~/.cookies.txt

REAL Code Examples from the Repository

Example 1: Basic Programmatic Usage

The core httpFetch function returns a structured result with multiple content formats. Here's the canonical pattern from the README:

import { httpFetch } from '@teng-lin/agent-fetch';

// Fetch and extract article content
const result = await httpFetch('https://example.com/article');

if (result.success) {
  // Full article as markdown with preserved structure
  console.log(result.markdown);
  // Output: "# Article Title\n\nParagraph with **bold** and [links](url)..."

  // Extracted metadata
  console.log(result.title);      // "Article Title"
  console.log(result.byline);     // "By John Smith"
  console.log(result.textContent); // Plain text without formatting
  console.log(result.latencyMs);   // 523 — performance metric
}

What's happening here? The function returns a discriminated union: result.success tells you whether extraction succeeded. The markdown property contains the structured content you'll typically feed to LLMs. The textContent variant strips formatting for simpler processing. Metadata fields enable automatic source attribution in your outputs.

Example 2: Advanced Options Configuration

For sites with aggressive bot detection or slow response times, customize the fetch behavior:

import { httpFetch } from '@teng-lin/agent-fetch';

// Configure timeout and TLS fingerprint preset
const result2 = await httpFetch('https://slow-site.com/article', {
  timeout: 30000,      // 30 seconds instead of default 20s
  preset: 'chrome-143', // Chrome TLS fingerprint (default)
});

// Alternative: impersonate iOS Safari for mobile-targeted content
const mobileResult = await httpFetch('https://mobile-only.example.com', {
  preset: 'ios-safari-18',
});

Critical insight: The preset parameter isn't cosmetic. It changes the actual TLS handshake at the cryptographic level. Different sites serve different content to mobile vs. desktop, or block specific browser fingerprints entirely. Having multiple presets lets you rotate identities when one gets blocked.

Example 3: CLI with Content Selection and Cleanup

Target specific page regions and remove noise:

# Extract only article content, strip navigation and sidebars
npx agent-fetch https://example.com/article \
  --select "article" \
  --remove "nav, .sidebar"

The --select parameter uses standard CSS selectors to identify your content container. The --remove parameter accepts comma-separated selectors for elements to strip. This is essential for pages where the extraction heuristics include unwanted chrome — navigation menus, cookie banners, related article widgets.

Example 4: Full-Featured Site Crawl

For systematic content collection, the crawl command offers production-grade control:

# Deep crawl with conservative rate limiting
npx agent-fetch crawl https://example.com \
  --depth 5 \           # Follow links up to 5 levels deep
  --limit 50 \          # Maximum 50 pages total
  --concurrency 3 \     # Only 3 simultaneous requests
  --delay 1000 \        # 1 second between requests (be polite!)
  --include "*/blog/*" \ # Only blog posts
  --exclude "**/archive/**" \ # Skip archive pages
  --cookie-file ~/.cookies.txt \ # Authenticated session
  --select "article" \   # Target article containers
  --json                # Structured output for processing

Production considerations: The --delay flag prevents overwhelming target servers. The --concurrency limit manages your outbound connection pool. JSON output enables piping to jq or feeding directly into downstream processing pipelines. This pattern replaces fragile custom scrapers that break on every site redesign.

Example 5: Cookie-Based Authentication

Access content behind login walls without building OAuth flows:

# Inline cookies for simple cases
npx agent-fetch https://example.com/article \
  --cookie "sessionId=abc123; theme=dark"

# Netscape cookie file for complex sessions (exported from browser)
npx agent-fetch https://example.com/article \
  --cookie-file ~/.cookies.txt

The Netscape format is the same cookie export format used by curl and wget, making migration trivial. This enables your AI agents to access the same content you browse manually — subscription articles, internal dashboards, authenticated APIs.

Advanced Usage & Best Practices

Strategy Selection Overrides

While automatic strategy selection works for 90% of cases, force specific approaches when you know the target stack:

# WordPress site — use REST API directly
npx agent-fetch https://wp-site.com/article --strategy wp-api

# Next.js App Router — prioritize RSC parsing
npx agent-fetch https://nextjs-site.com/article --strategy rsc

Performance Optimization

For high-volume pipelines, run agent-fetch as a persistent service rather than spawning CLI processes:

import { httpFetch } from '@teng-lin/agent-fetch';

// Reuse connection pools, DNS cache, and TLS sessions
const batchResults = await Promise.all(
  urls.map(url => httpFetch(url, { timeout: 15000 }))
);

Error Handling Patterns

const result = await httpFetch(url);

if (!result.success) {
  // result.error contains structured failure information
  console.error(`Extraction failed: ${result.error.code}`);
  // Common codes: TIMEOUT, BLOCKED, PARSE_ERROR, NETWORK_ERROR
  return;
}

// Validate content quality before processing
if (result.markdown.length < 1000) {
  console.warn('Suspiciously short extraction — possible paywall or bot detection');
}

Rate Limiting & Ethics

Always respect robots.txt. Use --delay on crawls. The --concurrency flag isn't just for performance — it's for being a good web citizen. The repository includes a clear responsible use disclaimer; heed it.

Comparison with Alternatives

When to choose what:

• Built-in fetch: Only for trivial cases where you control the target server
• Cloud APIs: When you need zero infrastructure and have budget for metered usage
• agent-fetch: When you need reliability, privacy, cost control, and production-grade extraction

FAQ

Does agent-fetch work with JavaScript-rendered sites?

Yes. The browser impersonation passes anti-bot checks, and extraction strategies handle both server-rendered HTML and client-side frameworks (Next.js, React Server Components). For heavy SPAs not using these frameworks, the text-density and CSS selector strategies still extract visible content.

Can I use agent-fetch in a serverless environment?

Yes, but with caveats. The TLS fingerprinting requires Node.js 20+ and sufficient cold-start time. For Vercel Edge or Cloudflare Workers, you may need the Node.js runtime rather than Edge. Test latency budgets carefully.

How does this differ from Puppeteer or Playwright?

Puppeteer/Playwright launch full Chromium instances — heavy, slow, resource-intensive. agent-fetch uses lightweight HTTP requests with fingerprint spoofing. It's 10-50x faster and uses fraction of the memory. For pure content extraction, it's the superior architecture.

What about legal and ethical concerns?

agent-fetch is a tool, not a license. You remain responsible for complying with Terms of Service, robots.txt, copyright law, and data protection regulations. The repository's responsible use section is explicit: this fetches publicly accessible content, nothing more.

Can I contribute new extraction strategies?

The MIT license and active CI pipeline suggest community contributions are welcome. The modular strategy architecture makes adding new parsers straightforward. Check the GitHub issues for requested strategies.

How do I debug extractions that return poor results?

Use --json for full metadata, --raw to inspect the actual HTML received, and --preset to try different browser fingerprints. The --select and --remove flags refine targeting for problematic layouts.

Is there a Python version?

Currently Node.js only. The architecture depends on specific TLS manipulation libraries available in the Node ecosystem. Python ports would need equivalent httpcloak integration.

Conclusion

The AI revolution is being built on data, and most of that data lives on the web. But we've been feeding our intelligent systems with truncated, bot-gated, structurally destroyed content — then wondering why their outputs disappoint.

agent-fetch fixes this at the root. Browser impersonation gets you past the guards. Seven parallel extraction strategies get you the complete content. Local execution keeps you in control. And the price — free, open source, zero API keys — makes it accessible to every developer.

I've watched too many RAG pipelines fail because of garbage inputs. I've seen too many AI agents stumble on simple research tasks because they couldn't read a web page properly. agent-fetch isn't just a nice-to-have utility; it's infrastructure for the next generation of AI applications.

Your next step: Head to github.com/teng-lin/agent-fetch. Star the repo. Run npx agent-fetch on a URL that's been giving you trouble. See what complete, structured, properly extracted content looks like. Then build something that actually works.

The web was made for humans. Now your AI agents can browse like humans too.