AiContent Services - Integration Guide

Canonical reference for apps and AI agents that need to consume our services. Every service below uses the same standardized section template:

Overview - what it does, base URL
Auth - how to authenticate
Pricing & quotas - credits, limits, billing
Endpoints - request/response shapes
Errors - HTTP status meanings + retry guidance
TypeScript client - drop-in sample
Python client - drop-in sample
Admin SQL - managing clients / inspecting usage

For agents: when integrating one of our services into another project, you only need the section for that service. Copy the env vars, drop in the client sample, done.

Services index

Service	Base URL	Status
AiContent Scraper	API: `https://scraperapi.aicontent.engineering` · Marketing: `https://scraper.aicontent.engineering`	Live

AiContent Scraper

Overview

Stealth headless-browser web scraping. Executes JavaScript, evades bot detection (Cloudflare, Datadome, PerimeterX), returns the full rendered HTML plus extracted text.

API base URL: https://scraperapi.aicontent.engineering
Docs (this page): https://scraper.aicontent.engineering/docs
What it returns: raw HTML (post-JS), extracted text, HTTP status, title, timing
Stealth: every request runs through a fingerprint-correct, anti-detection browser stack - bot signatures, navigator quirks, and TLS fingerprints look like a real visitor.

For agents: the API host is scraperapi.aicontent.engineering. The marketing/docs host (scraper.aicontent.engineering) is a different server and does not serve /scrape. Hitting the wrong host returns a 404 or HTML page, not a useful error. If you ever see HTTP 401 with {"error":"unauthorized"} you are hitting the right host but missing or sending an invalid X-API-Key header.

Auth

Send your client key in the X-API-Key header. One key per consuming app.

Keys use the sc_ prefix. Existing keys with other prefixes keep working.

# Submit a job
curl -X POST https://scraperapi.aicontent.engineering/jobs \
  -H "X-API-Key: sc_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com"]}'
# -> {"jobs":[{"id": 123, "url": "https://example.com", "status": "queued", ...}]}

# Poll
curl https://scraperapi.aicontent.engineering/jobs/123 -H "X-API-Key: sc_YOUR_KEY_HERE"
# -> {"id": 123, "status": "done", "tier_used": "datacenter", "screenshot_url": "...", ...}

Pricing & quotas

Line	Cost	Notes
Base subscription	$99 / month	Per client. Required. Covers platform access + support.
Stealth datacenter scrape	5 credits	$0.008 each. Failed datacenter attempts are FREE - you're only charged if we returned usable content.
Residential proxy	25 credits total	$0.040 each. Fires automatically when datacenter is blocked or the domain is known-hard. Charged whether the attempt succeeded or not (we pay for residential bandwidth either way).
Web Unblocker	50 credits total	$0.080 each. Hosted fall-through for the hardest 1-2% of targets. Charged on attempt.
/fetch (binaries)	1 / 10 / 50 credits	Per tier (datacenter/residential/web_unblocker). 80% cheaper than /scrape since there's no JS render.

Credits price at $0.0016 each. Usage is reported as Stripe meter events and rolls up monthly into the same invoice as the base subscription.

Billing principles:

A scrape of an easy site that succeeds on datacenter = 5 credits ($0.008).
A scrape that fails on datacenter and succeeds on residential = 25 credits ($0.040). Datacenter is free; you pay for the tier that worked.
A scrape that fails on every tier = 75 credits ($0.120). No charge for the free datacenter attempt; 25 for the residential attempt + 50 for the Web Unblocker attempt (we paid Rayobyte for both).

Local quota. Each client also has a monthly_credit_limit (default 100,000). When the month's usage would exceed it, the service returns 429 {"error":"monthly credit limit reached","limit":N,"used":N}. Reset is the 1st of each calendar month (UTC). The limit is enforced regardless of Stripe billing status.

Endpoints - async jobs (primary)

The canonical flow: POST jobs, get ids back, poll for results. Submission returns in well under a second regardless of how slow the targets are. The scraper runs the escalation ladder server-side; your client just polls.

`POST /jobs`

Submit one or more URLs for async processing.

Request body:

Field	Type	Required	Default	Notes
`urls`	array of strings (or single string)	yes	-	Up to 50 URLs. Each must have a scheme.
`op`	string	no	`"scrape"`	`"scrape"` = full browser render, returns html+text+screenshot. `"fetch"` = raw HTTP bytes (PDFs, images, JSON, anything), cheaper.
`wait_ms`	int	no	1500	scrape only - max wait after page DOM is ready. Adaptive polling for >200 chars body text up to `max(wait_ms, 6000)`. Use to wait LONGER on slow SPAs.
`max_bytes`	int	no	52,428,800 (50MB)	fetch only - body size cap. Maximum 50MB.

Response (200 - returns immediately, all jobs queued):

{
  "jobs": [
    {"id": 123, "url": "https://example.com", "domain": "example.com", "op": "scrape", "status": "queued"},
    {"id": 124, "url": "https://another.com/file.pdf", "domain": "another.com", "op": "fetch", "status": "queued"}
  ]
}

Pre-flight credit check uses the minimum tier cost. The 429 fires only when even the cheapest possible execution would exceed your monthly limit.

`GET /jobs/{id}`

Poll a single job's status. Returns the full result when status="done".

Response shape (status varies by lifecycle):

{
  "id": 123,
  "url": "https://example.com",
  "domain": "example.com",
  "op": "scrape",
  "status": "done",
  "tier_used": "datacenter",
  "status_code": 200,
  "blocked": false,
  "block_reason": null,
  "render_ms": 8643,
  "bytes_html": 12435,
  "credits": 5,
  "queued_at": "2026-05-30T00:52:08.50304+00:00",
  "expires_at": "2026-06-06T00:52:08.50304+00:00",
  "screenshot_url": "https://.../63.jpg",
  "title": "Example Domain",
  "html": "<!doctype html>...full rendered page...",
  "text": "Example Domain  This domain is for use in..."
}

For op="scrape" when done you get the full rendered page back directly - html (post-JS DOM), text (extracted body text from <main>/<article>/<body>), and title. No need to re-scrape to get content. (Note: html/text can be large; the list endpoint below omits them - poll the specific id when you want the body.)

For op="fetch" when done, the response also has:

{
  "content_type": "image/jpeg",
  "bytes": 8001,
  "body_url": "https://qiesrihozwimcxcwsugy.supabase.co/storage/v1/object/sign/scraper-fetched/...?token=..."
}

body_url is a Supabase Storage signed URL valid for 1 hour. Fetch it with a plain GET (no auth needed on that URL) to get the raw bytes. Re-poll /jobs/{id} later to mint a fresh URL if needed.

Lifecycle:

`status`	Meaning
`queued`	Submitted, awaiting a worker.
`running`	A worker picked it up; render or fetch in flight.
`done`	Terminal - success. Result fields populated.
`failed`	Terminal - all escalation tiers failed. `error` and `block_reason` describe what happened.
`cancelled`	You called `DELETE /jobs/{id}` before it ran.
`stale`	A worker started it but never finished within 5 minutes (presumed crashed). Treat as `failed` for retry purposes.

Polling cadence: scrapes typically complete in 5-30s. A 2s poll interval is reasonable.

`GET /jobs?ids=1,2,3,...`

Bulk poll up to 100 ids in one shot. Returns the same per-job shape as /jobs/{id}, filtered to jobs owned by your key.

curl "https://scraperapi.aicontent.engineering/jobs?ids=123,124,125" \
  -H "X-API-Key: sc_..."
# -> { "jobs": [...] }

`GET /jobs` (no `ids`) - list ALL your jobs

Omit ids and you get every job owned by your key, newest first. This is how you recover ids after a crashed or interrupted batch run - you don't have to have saved the ids at submit time. List rows are lightweight: they include status, title, byte counts, and the screenshot URL, but not html/text/body_url (poll the specific id for content).

Filters (all optional):

Param	Meaning
`status`	`queued`, `running`, `done`, `failed`, `cancelled`, `stale` - comma-separated for multiple.
`op`	`scrape` or `fetch`.
`limit`	Page size, default 50, max 200.
`before_id`	Cursor: return jobs with `id < before_id`. Use the `next_before_id` from the previous page.

# Everything still in flight
curl "https://scraperapi.aicontent.engineering/jobs?status=queued,running" -H "X-API-Key: sc_..."

# Page through completed scrapes, 200 at a time
curl "https://scraperapi.aicontent.engineering/jobs?status=done&op=scrape&limit=200" -H "X-API-Key: sc_..."
# -> { "jobs": [...], "count": 200, "next_before_id": 4021 }
curl "https://scraperapi.aicontent.engineering/jobs?status=done&op=scrape&limit=200&before_id=4021" -H "X-API-Key: sc_..."

next_before_id is null on the last page.

Retention - content is auto-deleted after 7 days

Completed jobs (and their stored html/text/screenshots/fetch-bodies) are kept for 7 days, then a background sweep deletes the row and its storage objects. Each job carries an expires_at timestamp so you know the window. Pull what you need within 7 days of completion - after that the job is gone and you'd re-scrape (and re-pay). There's no extra charge for storage during the window.

`DELETE /jobs/{id}`

Cancel a job. Only works while status="queued" - a running job can't be cancelled mid-render. Returns 409 if the job is already running or terminal.

`GET /health`, `GET /openapi.json`

No auth. Liveness probe + machine-readable spec.

Errors

HTTP	When
400	Missing `url`, malformed JSON, batch >50 URLs, max_bytes out of range.
401	Missing or invalid `X-API-Key`.
421	You hit `scraper.aicontent.engineering` (marketing host) instead of `scraperapi.aicontent.engineering` (API host). Body has `api_base`.
429	Monthly credit limit reached.
502	All escalation tiers tried and failed. Body has `attempts[]` array detailing each attempt.

Retry guidance: retry transient 502s once with backoff. Don't retry 401 (key issue) or 429 (need limit increase).

TypeScript client

const SCRAPER_URL = 'https://scraperapi.aicontent.engineering'
const SCRAPER_KEY = process.env.SCRAPER_API_KEY!   // sc_...

type Job = {
  id: number
  url: string
  domain: string
  op: 'scrape' | 'fetch'
  status: 'queued' | 'running' | 'done' | 'failed' | 'cancelled' | 'stale'
  tier_used?: 'datacenter' | 'residential' | 'web_unblocker'
  status_code?: number
  blocked?: boolean
  block_reason?: string | null
  render_ms?: number
  bytes_html?: number
  bytes?: number
  content_type?: string
  body_url?: string         // fetch op only; 1-hour signed URL
  screenshot_url?: string   // scrape op only
  credits?: number
  error?: string
}

const POLL_INTERVAL_MS = 2000
const POLL_TIMEOUT_MS = 5 * 60 * 1000

async function api<T>(path: string, init: RequestInit = {}): Promise<T> {
  const res = await fetch(`${SCRAPER_URL}${path}`, {
    ...init,
    headers: { 'Content-Type': 'application/json', 'X-API-Key': SCRAPER_KEY, ...(init.headers || {}) },
  })
  if (res.status === 401) throw new Error('scraper: invalid key')
  if (res.status === 429) throw new Error('scraper: quota exceeded')
  const data = await res.json()
  if (!res.ok) throw new Error(data?.error || `scraper: HTTP ${res.status}`)
  return data as T
}

// Submit one or more URLs. Returns the job ids immediately (status=queued).
export async function submitJobs(urls: string[], op: 'scrape' | 'fetch' = 'scrape'): Promise<Job[]> {
  const { jobs } = await api<{ jobs: Job[] }>('/jobs', {
    method: 'POST',
    body: JSON.stringify({ urls, op }),
  })
  return jobs
}

// Poll a single job until it reaches a terminal state.
export async function waitForJob(id: number): Promise<Job> {
  const start = Date.now()
  while (true) {
    const job = await api<Job>(`/jobs/${id}`)
    if (['done', 'failed', 'cancelled', 'stale'].includes(job.status)) return job
    if (Date.now() - start > POLL_TIMEOUT_MS) throw new Error(`scraper: job ${id} timed out`)
    await new Promise(r => setTimeout(r, POLL_INTERVAL_MS))
  }
}

const chunk = <T>(a: T[], n: number): T[][] =>
  Array.from({ length: Math.ceil(a.length / n) }, (_, i) => a.slice(i * n, i * n + n))

// List YOUR jobs (newest first). Recover ids after a crash, or sweep for unfinished work.
// Omit `status` for everything. Pages via before_id automatically.
export async function listMyJobs(
  opts: { status?: string; op?: 'scrape' | 'fetch'; max?: number } = {}
): Promise<Job[]> {
  const out: Job[] = []
  let before: number | undefined
  while (out.length < (opts.max ?? Infinity)) {
    const q = new URLSearchParams({ limit: '200' })
    if (opts.status) q.set('status', opts.status)
    if (opts.op) q.set('op', opts.op)
    if (before) q.set('before_id', String(before))
    const r = await api<{ jobs: Job[]; next_before_id: number | null }>(`/jobs?${q}`)
    out.push(...r.jobs)
    if (r.next_before_id == null) break
    before = r.next_before_id
  }
  return out
}

// Convenience: submit + wait. CHUNKS correctly - submit in 50s, poll in 100s - so it
// scales to thousands of URLs. This is the right way to run a large batch.
export async function scrapeAll(urls: string[], op: 'scrape' | 'fetch' = 'scrape'): Promise<Job[]> {
  // 1. Submit in chunks of 50 (the POST limit), collect all ids.
  const queued: Job[] = []
  for (const c of chunk(urls, 50)) queued.push(...await submitJobs(c, op))

  // 2. Poll in chunks of 100 (the bulk-poll limit) until every job is terminal.
  const pending = new Map(queued.map(j => [j.id, j]))
  const done: Job[] = []
  const start = Date.now()
  while (pending.size > 0) {
    if (Date.now() - start > POLL_TIMEOUT_MS) throw new Error('scraper: batch timed out')
    await new Promise(r => setTimeout(r, POLL_INTERVAL_MS))
    for (const c of chunk([...pending.keys()], 100)) {
      const { jobs } = await api<{ jobs: Job[] }>(`/jobs?ids=${c.join(',')}`)
      for (const j of jobs) {
        if (['done', 'failed', 'cancelled', 'stale'].includes(j.status)) {
          done.push(j); pending.delete(j.id)
        }
      }
    }
  }
  return done
}

// Example: scrape one URL the simple way
const [job] = await submitJobs(['https://example.com'])
const result = await waitForJob(job.id)
if (result.status === 'done') {
  console.log(result.title, result.text?.slice(0, 200))  // content comes back inline
}

// Example: run a 660-URL batch reliably (chunked submit + chunked poll, retry failures)
const results = await scrapeAll(allUrls)              // handles chunking internally
const failed = results.filter(j => j.status !== 'done').map(j => j.url)
if (failed.length) await scrapeAll(failed)            // one retry pass

Python client

import os, time, requests
from typing import Iterable, Literal

SCRAPER_URL = 'https://scraperapi.aicontent.engineering'
KEY = os.environ['SCRAPER_API_KEY']
HEADERS = {'X-API-Key': KEY, 'Content-Type': 'application/json'}

POLL_INTERVAL_S = 2
POLL_TIMEOUT_S = 5 * 60


def submit_jobs(urls: Iterable[str], op: Literal['scrape', 'fetch'] = 'scrape') -> list[dict]:
    r = requests.post(f'{SCRAPER_URL}/jobs', headers=HEADERS, json={'urls': list(urls), 'op': op}, timeout=15)
    r.raise_for_status()
    return r.json()['jobs']


def wait_for_job(job_id: int) -> dict:
    deadline = time.time() + POLL_TIMEOUT_S
    while True:
        r = requests.get(f'{SCRAPER_URL}/jobs/{job_id}', headers={'X-API-Key': KEY}, timeout=15)
        r.raise_for_status()
        job = r.json()
        if job['status'] in ('done', 'failed', 'cancelled', 'stale'):
            return job
        if time.time() > deadline:
            raise RuntimeError(f'scraper: job {job_id} timed out')
        time.sleep(POLL_INTERVAL_S)


def _chunks(seq, n):
    for i in range(0, len(seq), n):
        yield seq[i:i + n]


def list_my_jobs(status: str = '', op: str = '', max_rows: int | None = None) -> list[dict]:
    """List YOUR jobs, newest first. Recover ids after a crash, or find unfinished work.
    Pass status='queued,running' for in-flight only. Pages via before_id automatically."""
    out, before = [], None
    while max_rows is None or len(out) < max_rows:
        params = {'limit': 200}
        if status: params['status'] = status
        if op: params['op'] = op
        if before: params['before_id'] = before
        r = requests.get(f'{SCRAPER_URL}/jobs', headers={'X-API-Key': KEY}, params=params, timeout=15)
        r.raise_for_status()
        body = r.json()
        out.extend(body['jobs'])
        before = body.get('next_before_id')
        if before is None:
            break
    return out


def scrape_all(urls: list[str], op: Literal['scrape', 'fetch'] = 'scrape') -> list[dict]:
    """Submit + wait, CHUNKED correctly (50 per POST, 100 per poll) so it scales to
    thousands of URLs without hitting any per-request limit. The right way to run a batch."""
    # 1. Submit in chunks of 50, collect ids.
    pending = {}
    for c in _chunks(urls, 50):
        for j in submit_jobs(c, op):
            pending[j['id']] = j
    # 2. Poll in chunks of 100 until everything is terminal.
    done, deadline = [], time.time() + POLL_TIMEOUT_S
    while pending:
        if time.time() > deadline:
            raise RuntimeError('scraper: batch timed out')
        time.sleep(POLL_INTERVAL_S)
        for c in _chunks(list(pending), 100):
            ids = ','.join(str(i) for i in c)
            r = requests.get(f'{SCRAPER_URL}/jobs', headers={'X-API-Key': KEY}, params={'ids': ids}, timeout=15)
            r.raise_for_status()
            for j in r.json()['jobs']:
                if j['status'] in ('done', 'failed', 'cancelled', 'stale'):
                    done.append(j)
                    pending.pop(j['id'], None)
    return done


# Example: run a large batch reliably, then retry only the failures.
results = scrape_all(all_urls)                      # chunks internally
for j in (r for r in results if r['status'] == 'done' and r['op'] == 'scrape'):
    save(j['url'], j['html'], j['text'])            # content comes back inline
failed = [r['url'] for r in results if r['status'] != 'done']
if failed:
    scrape_all(failed)                              # one retry pass

# Example: recover after a crash - find everything still running and wait on it
in_flight = list_my_jobs(status='queued,running')
print(f'{len(in_flight)} jobs still in flight; their ids:', [j['id'] for j in in_flight])

# Example: fetch a PDF, then download the bytes via the signed URL
[fetch_job] = submit_jobs(['https://example.com/file.pdf'], op='fetch')
done = wait_for_job(fetch_job['id'])
if done['status'] == 'done':
    pdf_bytes = requests.get(done['body_url'], timeout=60).content
    open('out.pdf', 'wb').write(pdf_bytes)

Legacy synchronous endpoints

Kept for back-compatibility with integrations that pre-date the async API. Do not use these for new integrations - long-running calls can hit Cloudflare's 100-second tunnel timeout on slow targets. Use POST /jobs instead.

Method	Path	Behavior
POST	`/scrape`	Synchronous scrape. Held connection. Returns the same shape as `done` job result, plus an `attempts[]` array.
POST	`/scrape-batch`	Synchronous batch (≤50 URLs). All-or-nothing latency: returns when every URL has been attempted.
POST	`/fetch`	Synchronous fetch - body returned inline with original `Content-Type` (no `body_url` since the connection itself streams the bytes).

These pre-date the async API and use the same auth, same escalation ladder, same billing.

Admin SQL

Until the /admin/scraper UI ships, manage clients with psql against the aicontent Supabase pooler.

Internal table names use scraper_* (renamed from the original render_* on 2026-05-28).

-- Issue a new client. Generate the raw key first:
--   KEY="sc_$(openssl rand -base64 24 | tr '+/=' '_-')"
--   echo $KEY                              # hand to the client
--   PREFIX=$(echo $KEY | cut -c1-8)
--   HASH=$(printf '%s' "$KEY" | shasum -a 256 | awk '{print $1}')
INSERT INTO scraper_api_clients (name, contact_email, key_prefix, key_hash, monthly_credit_limit)
VALUES ('ClientName', 'contact@client.com', '<PREFIX>', '<HASH>', 100000);

-- Wire metered billing (after creating Stripe customer + subscription):
UPDATE scraper_api_clients
SET stripe_customer_id = 'cus_...',
    stripe_subscription_id = 'sub_...',
    stripe_subscription_item_id = 'si_...',
    stripe_billing_enabled = TRUE
WHERE name = 'ClientName';

-- Bump or remove a credit limit
UPDATE scraper_api_clients SET monthly_credit_limit = 500000 WHERE name = 'ClientName';
UPDATE scraper_api_clients SET monthly_credit_limit = 0 WHERE name = 'ClientName';  -- 0 = unlimited

-- Disable a key (auth fails immediately)
UPDATE scraper_api_clients SET status = 'disabled' WHERE name = 'ClientName';

-- Current month usage per client
SELECT c.name,
       scraper_client_credits_used_this_month(c.id) AS used,
       c.monthly_credit_limit AS limit,
       c.stripe_billing_enabled
FROM scraper_api_clients c
WHERE c.status = 'active';

-- Which scrapes were NOT reported to Stripe? (success+billing_enabled+not reported)
SELECT u.occurred_at, u.url, u.credits_charged
FROM scraper_usage u
JOIN scraper_api_clients c ON c.id = u.client_id
WHERE c.stripe_billing_enabled = TRUE
  AND u.success = TRUE
  AND u.stripe_event_reported = FALSE
ORDER BY u.occurred_at DESC
LIMIT 50;

Stripe references

Object	ID	Notes
Product (metered usage)	`prod_UbL9M1ahNphKFu`	AiContent Scraper credits
Product (base sub)	`prod_UbNSKT4EQCJegl`	$99/mo platform fee
Meter	`mtr_test_61UlWQqaFHOMIlMTd41PFJueV2quz1ua`	event_name `render_credit`, display "AiContent Scraper Credits"
Price (base sub)	`price_1TcAhCPFJueV2quzYCKlIguD`	$99/month, licensed, recurring
Price (metered credits)	`price_1TcAyHPFJueV2quzE0yGPYjK`	$0.0016/credit, metered, recurring

Preview a customer's upcoming invoice:

curl -u "$STRIPE_SECRET_KEY:" -X POST \
  https://api.stripe.com/v1/invoices/create_preview \
  -d subscription=sub_...