# Firecrawl Browserless Adapter (POC)

Thin adapter that mimics a subset of the Firecrawl API using:
- Direct HTTP fetch for SSR/static pages.
- Browserless (headless Chromium) as fallback / SPA renderer.
- Readability + Turndown to extract clean Markdown.

Tech stack:
- NestJS, TypeScript, hexagonal architecture.
- Vitest for tests.
- Docker + docker-compose only; no host dependencies required.

Endpoints:
- GET /v1/health -> { success: true, status: "ok", service: "firecrawl-browserless-adapter" }
- POST /v1/scrape
  - Body: { url: string, render?: boolean }
  - Response: { url, title, markdown, html?, metadata: { source } }

Usage (local):
- docker compose up --build
- curl http://localhost:3000/v1/health
- curl -X POST http://localhost:3000/v1/scrape     -H "Content-Type: application/json"     -d '{"url": "https://example.com"}'

SPA / render mode:
- Set render=true to force Browserless for SPAs.
- Without it, the adapter tries direct fetch first and falls back if content looks incomplete.

Notes:
- This is a POC; not production-hardened.
- No auth, no rate limiting, basic error handling only.

## Deployment (Home-Server / Production)

This service is designed to be a drop-in replacement for Firecrawl-style scraping endpoints.

### Prerequisites

- Docker + Docker Compose installed on the target server.
- Port 3002 free (default external port).
- Internet access from the container (to scrape targets and pull Browserless image).

### Steps

1. Copy this repository to your server:

   - Example path: /opt/ai-tools

2. Ensure docker-compose.yaml uses a non-conflicting external port:

   - Default mapping: 3002:3000
   - If 3002 is in use, change it (e.g., 4000:3000).

3. Start the stack:

   - docker compose up --build -d

4. Verify:

   - Health check: GET http://localhost:3002/v1/health
   - Scrape test: POST http://localhost:3002/v1/scrape with JSON body:
     { "url": "https://example.com" }

### Environment / Configuration

Key environment variables (set in docker-compose.yaml or via .env):

- PORT=3000
  - Internal port the NestJS app listens on. Do not change unless you also update Dockerfile/CMD.

- BROWSERLESS_URL=http://browserless:3000
  - Must point to the Browserless service (default uses internal Compose DNS).
  - When running outside Compose, adjust this to the real Browserless endpoint.

- BROWSERLESS_TOKEN=your-token-here (optional)
  - If your Browserless instance requires a token:
    - Set TOKEN in browserless service environment.
    - Set BROWSERLESS_TOKEN with the same value in firecrawl-adapter.

### Using as Drop-in Replacement

Once running, this adapter can be used like a Firecrawl-style scrape endpoint:

- Endpoint: POST http://<server>:3002/v1/scrape
- Request body (JSON):
  - url: string (required) – page to scrape
  - render: boolean (optional) – force Browserless/JS rendering when true
- Response includes:
  - url, title, markdown, html, metadata.source

To integrate into other tools or agents:
- Replace any Firecrawl scrape URL with this endpoint.
- Ensure the same JSON schema is used for requests.