Documentation
Home GitHub

Core Concepts

Core concepts of go-browser — stealth anti-detection, cookie session injection, multi-snapshot merge, and multi-format output.

Stealth Anti-Detection

go-browser embeds a stealth.js script (core/embed/stealth.js) that is injected into every page via EvalOnNewDocument before navigation. This script patches browser runtime properties to reduce detection by anti-bot services such as Cloudflare.

Key techniques:

For pages behind login walls, go-browser can extract cookies from the local Chrome profile and inject them into a temporary browser instance.

The process (core/cookie.go):

  1. Profile discovery — Locates the Chrome profile directory (~/Library/Application Support/Google/Chrome/<Profile> on macOS, ~/.config/google-chrome/<Profile> on Linux).
  2. Cookie database copy — Copies the SQLite Cookies file (plus WAL and SHM) to a temporary directory to avoid locking the live profile.
  3. Key derivation — Retrieves the Chrome Safe Storage password from the OS keychain, then derives an AES key using PBKDF2-SHA1 with 1003 iterations and the salt saltysalt.
  4. Decryption — Decrypts v10-prefixed encrypted cookie values using AES-CBC with a space-filled initialization vector.
  5. Injection — Launches a temporary Chrome instance with the copied profile directory and sets the decrypted cookies via browser.SetCookies.

This feature is enabled by setting Option.SameSession: true and is only supported on macOS and Linux.

Multi-Snapshot Merge

To capture dynamically loaded content (infinite scroll, lazy images, paginated feeds), go-browser simulates scrolling behavior and merges multiple page snapshots.

The scroll loop (core/fetch.go):

  1. Captures an initial HTML snapshot after DOM stabilization.
  2. For each scroll iteration (default 3, configurable via Option.ScrollCount):
    • Waits a random delay of 1–3 seconds.
    • Smoothly scrolls to the bottom using requestAnimationFrame.
    • Waits for DOM stability.
    • Captures another HTML snapshot.
  3. Merges all snapshots by appending body children from subsequent snapshots to the base document body (core/markdown.go Merge function).
  4. Runs readability extraction on each snapshot and deduplicates paragraphs by hash.

Multi-Format Output

go-browser supports three output formats controlled by Option.Type:

Type Constant Description
Markdown TypeMarkdown (0) Default. Extracts article content via go-readability, converts to Markdown, deduplicates paragraphs.
HTML TypeHTML (1) Merges raw HTML snapshots, inlines <time> elements, returns the combined HTML.
JSON TypeJSON (2) Converts HTML to a structured []*Node tree with typed nodes (heading, paragraph, link, image, list, code_block, etc.), serialized as JSON.

Output Format Details

Markdown — Each snapshot is parsed by go-readability to extract the article. Content parts are joined, converted to Markdown via HTMLToMarkdown, and deduplicated via DedupMarkdownParagraphs. The result includes Title, Author (byline), Excerpt, and PublishedAt when available.

HTML — Snapshots are merged by appending body children. <time> elements are inlined to plain text with their datetime attribute. The result is a single combined HTML document.

JSON — The readability-extracted content is converted to a []*Node tree via HTMLToNode. Each node has a type (text, heading, paragraph, link, image, list, code_block, etc.) and optional fields (text, level, href, src, alt, children). The tree is deduplicated and flattened before serialization.

Interactive Tab Operations

Beyond single-page fetch, go-browser provides stateful tab management for multi-step workflows:

Tabs share a single interactive browser instance. When all tabs are closed, the browser is shut down automatically.

中文