firecrawl-adapter/README.md

Firecrawl Browserless Adapter (POC)

Thin adapter that mimics a subset of the Firecrawl API using:

• Direct HTTP fetch for SSR/static pages.
• Browserless (headless Chromium) as fallback / SPA renderer.
• Readability + Turndown to extract clean Markdown.

Tech stack:

• NestJS, TypeScript, hexagonal architecture.
• Vitest for tests.
• Docker + docker-compose only; no host dependencies required.

Endpoints:

• GET /v1/health -> { success: true, status: "ok", service: "firecrawl-browserless-adapter" }
• POST /v1/scrape
  • Body: { url: string, render?: boolean }
  • Response: { url, title, markdown, html?, metadata: { source } }

Usage (local):

• docker compose up --build
• curl http://localhost:3000/v1/health
• curl -X POST http://localhost:3000/v1/scrape -H "Content-Type: application/json" -d '{"url": "https://example.com"}'

SPA / render mode:

• Set render=true to force Browserless for SPAs.
• Without it, the adapter tries direct fetch first and falls back if content looks incomplete.

Notes:

• This is a POC; not production-hardened.
• No auth, no rate limiting, basic error handling only.

Deployment (Home-Server / Production)

This service is designed to be a drop-in replacement for Firecrawl-style scraping endpoints.

Prerequisites

• Docker + Docker Compose installed on the target server.
• Port 3002 free (default external port).
• Internet access from the container (to scrape targets and pull Browserless image).

Steps

1. Copy this repository to your server:

   • Example path: /opt/ai-tools

2. Ensure docker-compose.yaml uses a non-conflicting external port:

   • Default mapping: 3002:3000
   • If 3002 is in use, change it (e.g., 4000:3000).

3. Start the stack:

   • docker compose up --build -d

4. Verify:

   • Health check: GET http://localhost:3002/v1/health
   • Scrape test: POST http://localhost:3002/v1/scrape with JSON body: { "url": "https://example.com" }

Environment / Configuration

Key environment variables (set in docker-compose.yaml or via .env):

• PORT=3000

  • Internal port the NestJS app listens on. Do not change unless you also update Dockerfile/CMD.

• BROWSERLESS_URL=http://browserless:3000

  • Must point to the Browserless service (default uses internal Compose DNS).
  • When running outside Compose, adjust this to the real Browserless endpoint.

• BROWSERLESS_TOKEN=your-token-here (optional)

  • If your Browserless instance requires a token:
  • Set TOKEN in browserless service environment.
  • Set BROWSERLESS_TOKEN with the same value in firecrawl-adapter.

Using as Drop-in Replacement

Once running, this adapter can be used like a Firecrawl-style scrape endpoint:

• Endpoint: POST http://:3002/v1/scrape
• Request body (JSON):
  • url: string (required) – page to scrape
  • render: boolean (optional) – force Browserless/JS rendering when true
• Response includes:
  • url, title, markdown, html, metadata.source

To integrate into other tools or agents:

• Replace any Firecrawl scrape URL with this endpoint.
• Ensure the same JSON schema is used for requests.