AiContent Scraper

Connect to the Scraper API

Everything an AI agent or application needs to start scraping. Submit a batch of URLs, poll for results, read the rendered page back. No SDK required.

Base URL
https://scraperapi.aicontent.engineering
Authentication
Send your API key in the X-API-Key header on every request. One key per application. If you do not have a key yet, request one from your provider.
1

Submit a batch of URLs

POST up to 50 URLs in one call. You get job ids back immediately, in well under a second, no matter how slow the target sites are. The render happens server-side while you do other work.

curl -X POST https://scraperapi.aicontent.engineering/jobs \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com"], "op": "scrape"}'

# -> { "jobs": [ { "id": 123, "url": "https://example.com", "status": "queued" } ] }

op: "scrape" runs a full stealth browser render (HTML + text + screenshot). "fetch" returns raw bytes for PDFs, images, or JSON and is cheaper.

2

Poll for results

Keep the ids and poll until each job reaches a terminal status. Poll up to 100 ids in one call. A 2-second interval is reasonable.

curl "https://scraperapi.aicontent.engineering/jobs?ids=123,124,125" -H "X-API-Key: YOUR_KEY"

# status walks: queued -> running -> done | failed | cancelled | stale

Lost your ids (a crash, a restart)? List everything your key owns, newest first:

curl "https://scraperapi.aicontent.engineering/jobs?status=queued,running" -H "X-API-Key: YOUR_KEY"
# -> { "jobs": [...], "count": N, "next_before_id": 4021 }   (page with ?before_id=)
3

Read the content

When a scrape job is done, the rendered page comes back inline: html (post-JavaScript DOM), text (extracted body text), and title. No second request, no re-scrape.

curl "https://scraperapi.aicontent.engineering/jobs/123" -H "X-API-Key: YOUR_KEY"

# -> {
#   "id": 123, "status": "done", "status_code": 200,
#   "title": "Example Domain",
#   "html": "<!doctype html>...full rendered page...",
#   "text": "Example Domain  This domain is for use in...",
#   "tier_used": "datacenter", "credits": 5,
#   "expires_at": "2026-06-23T..."   // results are kept 14 days, then deleted
# }

For op:"fetch", the raw bytes come via a 1-hour signed body_url in the response instead of inline html.

Drop-in client

Submit and wait, chunked correctly so it scales from one URL to thousands. Copy, set your key, run.

TypeScript

// Minimal client. Set SCRAPER_API_KEY in your environment.
const API = 'https://scraperapi.aicontent.engineering'
const KEY = process.env.SCRAPER_API_KEY!

async function call<T>(path: string, init?: RequestInit): Promise<T> {
  const res = await fetch(API + path, {
    ...init,
    headers: { 'X-API-Key': KEY, 'Content-Type': 'application/json', ...(init?.headers || {}) },
  })
  const data = await res.json().catch(() => ({}))
  if (!res.ok) throw new Error(data?.error || `scraper: HTTP ${res.status}`)
  return data as T
}

const chunk = <T,>(a: T[], n: number): T[][] =>
  Array.from({ length: Math.ceil(a.length / n) }, (_, i) => a.slice(i * n, i * n + n))

// Submit + wait, chunked correctly (50 per submit, 100 per poll) so it scales to
// thousands of URLs without hitting any per-request limit.
async function scrapeAll(urls: string[], op: 'scrape' | 'fetch' = 'scrape') {
  const queued: any[] = []
  for (const c of chunk(urls, 50)) {
    const { jobs } = await call<{ jobs: any[] }>('/jobs', {
      method: 'POST', body: JSON.stringify({ urls: c, op }),
    })
    queued.push(...jobs)
  }
  const pending = new Map(queued.map(j => [j.id, j]))
  const done: any[] = []
  while (pending.size) {
    await new Promise(r => setTimeout(r, 2000))
    for (const c of chunk([...pending.keys()], 100)) {
      const { jobs } = await call<{ jobs: any[] }>(`/jobs?ids=${c.join(',')}`)
      for (const j of jobs) {
        if (['done', 'failed', 'cancelled', 'stale'].includes(j.status)) {
          done.push(j); pending.delete(j.id)
        }
      }
    }
  }
  return done
}

const results = await scrapeAll(['https://example.com'])
for (const j of results) {
  if (j.status === 'done') console.log(j.title, j.text?.slice(0, 200))
}

Python

# Minimal client. Set SCRAPER_API_KEY in your environment.
import os, time, requests

API = 'https://scraperapi.aicontent.engineering'
KEY = os.environ['SCRAPER_API_KEY']
H = {'X-API-Key': KEY, 'Content-Type': 'application/json'}

def _chunks(seq, n):
    for i in range(0, len(seq), n):
        yield seq[i:i + n]

# Submit + wait, chunked correctly (50 per submit, 100 per poll).
def scrape_all(urls, op='scrape'):
    pending = {}
    for c in _chunks(urls, 50):
        r = requests.post(f'{API}/jobs', headers=H, json={'urls': c, 'op': op}, timeout=15)
        r.raise_for_status()
        for j in r.json()['jobs']:
            pending[j['id']] = j
    done = []
    while pending:
        time.sleep(2)
        for c in _chunks(list(pending), 100):
            ids = ','.join(str(i) for i in c)
            r = requests.get(f'{API}/jobs', headers={'X-API-Key': KEY}, params={'ids': ids}, timeout=15)
            r.raise_for_status()
            for j in r.json()['jobs']:
                if j['status'] in ('done', 'failed', 'cancelled', 'stale'):
                    done.append(j); pending.pop(j['id'], None)
    return done

for j in scrape_all(['https://example.com']):
    if j['status'] == 'done':
        print(j['title'], (j.get('text') or '')[:200])

Good to know

  • Always async. Submit, then poll. The service runs an escalation ladder behind the scenes and the cheapest tier that works is billed; you never hold a connection open during the render.
  • Chunk large batches. 50 URLs per submit, 100 ids per poll. The drop-in client above handles this for you.
  • Retry failures, not the whole batch. A failed or stale job is one URL to resubmit.
  • Results expire after 14 days. Pull what you need within the window; after that the job and its content are deleted.
  • The machine-readable spec lives at /openapi.json.