This is the sequel to the weekend prototype. That build answered "can we derive address data ourselves instead of buying it?" in 48 hours with Node and an Electron GUI. This one answers "can it run in production for months, across six countries, without me babysitting it?" — different ballpark entirely.
The problem
The sales team at a Dutch home-services marketplace needed a continuous supply of qualified partner prospects: painters, plumbers, roofers, installers across NL, BE, FR, ES, IT and DE. Not a one-off list — a supply. Lists go stale, and a weekend scraper that works on Monday is broken by Thursday when a source redesigns its markup.
So the interesting engineering problem was never extraction. It was staying alive without me.
What got built
A production platform that crawls 304 configured sources — business directories, trade-association registers, competitor platforms — then normalizes, scores, deduplicates and LLM-enriches the results into CRM-ready prospects for HubSpot and the marketplace itself.
Pipeline: Python 3.12 + Celery across crawl / etl / enrich / analytics queues → spiders (Playwright with Camoufox anti-detect, curl_cffi TLS-fingerprint impersonation, residential proxies with geo-targeting) → raw JSONL to object storage → decoupled ETL → Supabase Postgres. A FastAPI control API sits in front, with a mobile-first React control UI on Vercel.
Infrastructure: a pure control-plane coordinator node plus autoscaled worker pools booted from pre-baked snapshots, scaled by queue depth, brokered over managed Valkey/TLS. Deploys promote a version bump rather than recycling the fleet.
LLM use, cost-engineered: OpenRouter for selector recovery and enrichment — company websites into firmographics and decision-maker contacts. The model was picked by a 7-model bake-off scored on decision-makers per dollar, not on benchmarks. A self-hosted Firecrawl instance (~$48/mo) then replaced per-call LLM enrichment for address backfill entirely, at ~620 addresses/hour.
The self-healing loop
This is the part worth stealing.
When a spider breaks on a site redesign, the crawler doesn't just alert — it files a repair ticket, which push-triggers a scheduled AI coding agent. The agent fixes the selectors against the live site, verifies with the test suite plus a real re-crawl, and opens a PR. A human reviews a diff instead of debugging a scraper.
A second watchdog layer sits above that, rebooting a wedged control plane through the cloud provider's API. Two layers, because the first one can't fix itself.
Making that safe took 17 project skills encoding the ops runbooks — spider-repair, worker-pools, database-health, coordinator-deploy — so an autonomous routine has a procedure to follow rather than improvising against production. CLAUDE.md became institutional memory: every production outage distilled into a rule, and every rule backed by a guard test so it can't silently drift back.
AI at content scale
One two-day burst ran ~1,175 headless AI coding sessions to bulk-rewrite original Dutch company introduction texts, replacing competitor-sourced descriptions ahead of the CRM upload. That got codified afterwards into a reusable skill with checkpoint/resume, 429 backoff and cost controls — so the next bulk job is a command, not a project.
Outcomes
| Companies captured | 217k+ — NL 102.7k, ES 31.6k, BE 30k, FR 28.2k, IT 24.9k, plus DE |
| Sources configured | 304 |
| Flagship CRM deliverable | ~153,641-row NL company upload, with a 500-row adversarially web-verified sample, a marketing report and a source-provenance report for management |
| Self-heal loop | verified end-to-end in production |
| Commits | 808 in ~10 weeks, largely agent-written under short plain-language direction |
What broke, and what it taught
Health is not presence. A worker counted "healthy" by its cloud tag ran disconnected for ~24 hours. Liveness needs two independent layers: zombie recycling inside the autoscaler, plus an external watchdog that doesn't trust the same signal.
Know your broker's defaults. Celery's 1-hour visibility timeout silently redelivered long crawls forever, burning roughly half of all crawl capacity on work that was already done. Fixed, then pinned with a lockstep guard test so the value can't drift back.
Silent data loss is more dangerous than loud errors. A stale category config dropped 14.2k ready providers from an export with no error at all. Separately, a dispatcher that selected rows on the same column it filled re-scraped identical unfillable rows forever. Both became enum-parametrized guard tests and attempt-markers — the class of bug, not just the instance.
Fix the query, not the compute. Database CPU pinning at 100% traced to per-record ILIKE scans. Bulk prefetch solved what a bigger instance would only have hidden.
WAFs fingerprint TLS, not user agents. curl_cffi impersonation cleared Imperva, DataDome and Cloudflare without a browser at all — with a deliberate escalation ladder so expensive residential proxies are the last resort, not the default.
Multi-agent git needs rules. Concurrent AI sessions sharing worktrees once swept staged work into the wrong commit. That produced a handoff doc and hard worktree discipline.
Durable ops config belongs in a database row, not a hand-edited .env that the next deploy quietly reverts.
How it was run
Human-as-director, agents-as-engineers. I steered with short prompts — "why are we stale on 25h last-scraped", "I'm on holiday next week and can't monitor you" — while the coding agent did the engineering.
The leverage was never in prompting better. It came from writing the operating knowledge down: skills as runbooks, CLAUDE.md as outage memory, a guard test for every incident. That's what makes it safe to let an autonomous routine touch production.