v0.3.3
Home GitHub

Architecture

Last updated


go-browser is four modules: routing picks a browser, the browser loads the page, the page becomes snapshots, the snapshots become output; the only loop back is one headed retry when blocked.

System overview


graph TB
    subgraph Entry
        A[Fetch]
    end
    subgraph Routing
        B{Option.Headless}
        C{Visible or requiresSession, and hasDisplay}
        D[headless first]
        E{isBlocked + hasDisplay}
    end
    subgraph Launcher
        F{SameSession}
        G[launchWithSnapshot: temp profile + cookies]
        H[ensureBrowser: cached by headless + UA]
    end
    subgraph Pipeline
        I[load: Navigate + Settle]
        J[handleConsent]
        K[Scroll + Snapshots]
    end
    subgraph Output
        L[Merge / Readability]
        M[HTMLToMarkdown]
        N[HTMLToNode]
    end
    A --> B
    B -->|true| F
    B -->|false| C
    C -->|yes: headed| F
    C -->|no| D --> F
    F -->|yes| G
    F -->|no| H
    G -->|ErrProfileNotFound| H
    G --> I
    H --> I --> J --> K --> L
    L --> M
    M -->|TypeJSON| N
    D -.->|result 403 / 429 / 503| E
    E -->|yes: one headed retry| F
  1. graph TB
  2.     subgraph Entry
  3.         A[Fetch]
  4.     end
  5.     subgraph Routing
  6.         B{Option.Headless}
  7.         C{Visible or requiresSession, and hasDisplay}
  8.         D[headless first]
  9.         E{isBlocked + hasDisplay}
  10.     end
  11.     subgraph Launcher
  12.         F{SameSession}
  13.         G[launchWithSnapshot: temp profile + cookies]
  14.         H[ensureBrowser: cached by headless + UA]
  15.     end
  16.     subgraph Pipeline
  17.         I[load: Navigate + Settle]
  18.         J[handleConsent]
  19.         K[Scroll + Snapshots]
  20.     end
  21.     subgraph Output
  22.         L[Merge / Readability]
  23.         M[HTMLToMarkdown]
  24.         N[HTMLToNode]
  25.     end
  26.     A --> B
  27.     B -->|true| F
  28.     B -->|false| C
  29.     C -->|yes: headed| F
  30.     C -->|no| D --> F
  31.     F -->|yes| G
  32.     F -->|no| H
  33.     G -->|ErrProfileNotFound| H
  34.     G --> I
  35.     H --> I --> J --> K --> L
  36.     L --> M
  37.     M -->|TypeJSON| N
  38.     D -.->|result 403 / 429 / 503| E
  39.     E -->|yes: one headed retry| F

Layers


LayerFileResponsibilityExported
Entry and routingcore/fetch.goMode selection, blocked detection, retry decision, load pipelineFetch, Option, Viewport, Result, Error, TypeMarkdown, TypeHTML, TypeJSON
Browser lifecyclecore/launcher.goInstance cache, idle eviction, concurrency gate, temp-profile launchClose, SetMaxConcurrency, Browser, DefaultUserAgent, ErrProfileNotFound
Cookie decryptioncore/cookie.go (macOS / Linux), core/cookieOther.go (stub that returns an error; unreachable in practice because SameSession falls back to a regular browser on other platforms before reaching it)Keychain password, PBKDF2, AES-CBC decryption, CDP cookie assemblynone (internal)
Content processingcore/markdown.goSnapshot merge, time inlining, Markdown and tree output, deduplicationMerge, HTMLToMarkdown, HTMLToNode, DedupTree, DedupMarkdownParagraphs, InlineTimeElements, Node

Chrome is driven through go-rod over the Chrome DevTools Protocol. go-rod and chromedp are CDP drivers that you script step by step; go-browser wraps navigation, settling, consent handling, scrolling and conversion into one Fetch call.
For content conversion, JohannesKaufmann/html-to-markdown converts HTML you already hold; go-browser renders the page in Chrome first, extracts the main content with go-readability, and converts it with its own HTMLToMarkdown built on golang.org/x/net/html. playwright-go is a browser automation library for Chromium, Firefox and WebKit, and geziyor is a crawling and scraping framework with JS rendering; go-browser drives Chrome only and returns one page per Fetch call. Anakin is a self-hosted web scraping API service that returns Markdown or JSON; go-browser is a library imported directly into your Go program.
The three injected scripts are embedded from core/embed/ with go:embed: stealth.js for anti-detection, listener.js for load settling, consent.js for consent banners.

Cross-cutting rules


RuleImplementation
Context flows throughThe ctx given to Fetch reaches settling, consent evaluation and scroll waits; timeout wraps it in a child context
Concurrency gated by a semaphoreacquireSem must grant a slot before a page is created, and load releases it on return
Resource release is pairedlaunchWithSnapshot returns a cleanup that closes the browser and removes the temp directory; cached instances are reclaimed by the evictor or Close
Degradation does not abortA missing Chrome profile returns ErrProfileNotFound, and fetchWith continues on a regular browser
Failures stay distinguishableSite blocks return *Error{Status}, extraction failures return a plain error, and routing treats them differently

Further reading


The module-level expansion — per-module diagrams, sequences and state machines — lives in doc/architecture.md in the repository. For the detail of any single flow, see Core Concepts.
中文