AiContent Scraper

AiContent Services — Integration Guide

Canonical reference for apps and AI agents that need to consume our services. Every service below uses the same standardized section template:

  1. Overview — what it does, base URL
  2. Auth — how to authenticate
  3. Pricing & quotas — credits, limits, billing
  4. Endpoints — request/response shapes
  5. Errors — HTTP status meanings + retry guidance
  6. TypeScript client — drop-in sample
  7. Python client — drop-in sample
  8. Admin SQL — managing clients / inspecting usage

For agents: when integrating one of our services into another project, you only need the section for that service. Copy the env vars, drop in the client sample, done.

Services index

Service Base URL Status
AiContent Scraper API: https://scraperbox.aicontent.engineering · Marketing: https://scraper.aicontent.engineering Live (test billing)

AiContent Scraper

Overview

Stealth headless-browser web scraping. Executes JavaScript, evades bot detection (Cloudflare, Datadome, PerimeterX), returns the full rendered HTML plus extracted text.

  • API base URL: https://scraperbox.aicontent.engineering
  • What it returns: raw HTML (post-JS), extracted text, HTTP status, title, timing
  • Stealth: every request runs through a fingerprint-correct, anti-detection browser stack — bot signatures, navigator quirks, and TLS fingerprints look like a real visitor.

Auth

Send your client key in the X-API-Key header. One key per consuming app. Keys are issued in test mode now; we mirror to live mode when ready (no contract change).

Future keys use the sc_ prefix. Existing keys with other prefixes keep working.

curl -X POST https://scraperbox.aicontent.engineering/scrape \
  -H "X-API-Key: sc_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Pricing & quotas

Line Cost Notes
Base subscription $99 / month Per client. Required. Covers platform access + support.
Stealth JS scrape (default) 5 credits $0.125 per scrape · $125 / 1k
Auto-escalation to residential +20 credits $0.50 extra per escalated scrape. Only fires when the default path is blocked, thin, or challenged. Billed only for the residential attempt.

Credits price at $0.025 each. Usage is reported as Stripe meter events and rolls up monthly into the same invoice as the base subscription.

Local quota. Each client also has a monthly_credit_limit (default 100,000). When the month's usage would exceed it, the service returns 429 {"error":"monthly credit limit reached","limit":N,"used":N}. Reset is the 1st of each calendar month (UTC). The limit is enforced regardless of Stripe billing status.

Endpoints

POST /scrape

Scrape a single URL. (Path is /scrape for backward compatibility; will alias to /scrape on the new subdomain.)

Request:

{ "url": "https://example.com", "wait_ms": 1500 }
Field Type Required Notes
url string yes Full URL with scheme
wait_ms int no Extra wait after networkidle. Default 1500. Auto-polls up to 6s for >200 chars text already.

Response (200):

{
  "url": "https://example.com",
  "status": 200,
  "title": "Example Domain",
  "html": "<!DOCTYPE html>...",
  "text": "Example Domain\n\nThis domain is for...",
  "render_ms": 9404,
  "stealth": true,
  "credits_charged": 5
}

Response (502):

{ "url": "...", "error": "navigation timeout", "render_ms": 30000, "stealth": true, "credits_charged": 5 }

Failures still charge — we spent the compute.

POST /scrape-batch

Up to 50 URLs per request. Workers process serially with bounded concurrency.

Request:

{ "urls": ["https://a.com", "https://b.com"] }

Response:

{
  "results": [
    { "url": "https://a.com", "status": 200, "html": "...", "text": "...", "render_ms": 4200, "stealth": true, "credits_charged": 5 },
    { "url": "https://b.com", "error": "Navigation timeout", "render_ms": 30000, "stealth": true, "credits_charged": 5 }
  ]
}

Whole batch fails (429) up-front if it would exceed your monthly limit.

GET /health

No auth. Liveness probe.

{ "ok": true }

Errors

HTTP When
400 Missing url, malformed JSON, batch >50 URLs
401 Missing or invalid X-API-Key
429 Monthly credit limit reached
502 Scrape failed (timeout, browser crash). Body has error field.

Retry guidance: retry transient 502s once with backoff. Don't retry 401 (key issue) or 429 (need limit increase).

TypeScript client

const SCRAPER_URL = 'https://scraperbox.aicontent.engineering'
const SCRAPER_KEY = process.env.SCRAPER_API_KEY!   // sc_... (or legacy ro_...)

export async function scrapePage(url: string, waitMs?: number) {
  const res = await fetch(`${SCRAPER_URL}/scrape`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', 'X-API-Key': SCRAPER_KEY },
    body: JSON.stringify({ url, ...(waitMs ? { wait_ms: waitMs } : {}) }),
    signal: AbortSignal.timeout(90_000),
  })
  if (res.status === 401) throw new Error('scraper: invalid key')
  if (res.status === 429) throw new Error('scraper: quota exceeded')
  const data = await res.json()
  if (data.error) throw new Error(`scraper: ${data.error}`)
  return data as {
    html: string
    text: string
    status: number
    title: string
    render_ms: number
    credits_charged: number
  }
}

Python client

import os, requests

SCRAPER_URL = 'https://scraperbox.aicontent.engineering'
KEY = os.environ['SCRAPER_API_KEY']

def scrape_page(url: str, wait_ms: int | None = None) -> dict:
    body = {'url': url}
    if wait_ms:
        body['wait_ms'] = wait_ms
    r = requests.post(
        f'{SCRAPER_URL}/scrape',
        json=body,
        headers={'X-API-Key': KEY},
        timeout=90,
    )
    r.raise_for_status()
    data = r.json()
    if data.get('error'):
        raise RuntimeError(f"scraper: {data['error']}")
    return data

Admin SQL

Until the /admin/scraper UI ships, manage clients with psql against the aicontent Supabase pooler.

Internal table names use scraper_* (renamed from the original render_* on 2026-05-28).

-- Issue a new client. Generate the raw key first:
--   KEY="sc_$(openssl rand -base64 24 | tr '+/=' '_-')"
--   echo $KEY                              # hand to the client
--   PREFIX=$(echo $KEY | cut -c1-8)
--   HASH=$(printf '%s' "$KEY" | shasum -a 256 | awk '{print $1}')
INSERT INTO scraper_api_clients (name, contact_email, key_prefix, key_hash, monthly_credit_limit)
VALUES ('ClientName', 'contact@client.com', '<PREFIX>', '<HASH>', 100000);

-- Wire metered billing (after creating Stripe customer + subscription):
UPDATE scraper_api_clients
SET stripe_customer_id = 'cus_...',
    stripe_subscription_id = 'sub_...',
    stripe_subscription_item_id = 'si_...',
    stripe_billing_enabled = TRUE
WHERE name = 'ClientName';

-- Bump or remove a credit limit
UPDATE scraper_api_clients SET monthly_credit_limit = 500000 WHERE name = 'ClientName';
UPDATE scraper_api_clients SET monthly_credit_limit = 0 WHERE name = 'ClientName';  -- 0 = unlimited

-- Disable a key (auth fails immediately)
UPDATE scraper_api_clients SET status = 'disabled' WHERE name = 'ClientName';

-- Current month usage per client
SELECT c.name,
       scraper_client_credits_used_this_month(c.id) AS used,
       c.monthly_credit_limit AS limit,
       c.stripe_billing_enabled
FROM scraper_api_clients c
WHERE c.status = 'active';

-- Which scrapes were NOT reported to Stripe? (success+billing_enabled+not reported)
SELECT u.occurred_at, u.url, u.credits_charged
FROM scraper_usage u
JOIN scraper_api_clients c ON c.id = u.client_id
WHERE c.stripe_billing_enabled = TRUE
  AND u.success = TRUE
  AND u.stripe_event_reported = FALSE
ORDER BY u.occurred_at DESC
LIMIT 50;

Stripe references (test mode)

Object ID
Product prod_UbL9M1ahNphKFu (AiContent Scraper)
Meter mtr_test_61UlWQqaFHOMIlMTd41PFJueV2quz1ua (event_name render_credit, display "AiContent Scraper Credits")
Price price_1TcAfBPFJueV2quzmlJ6pr1I (metered, monthly, $0.025/credit)

Preview a customer's upcoming invoice:

curl -u "$STRIPE_SECRET_KEY:" -X POST \
  https://api.stripe.com/v1/invoices/create_preview \
  -d subscription=sub_...