Serverless data pipeline on AWS
Pricing research needed a steady feed of used-parts listings from the market. I designed, built, and ran an event-driven pipeline on AWS. In testing it reached the target of 5 million listings a day, on the smallest database instance.
- My role
- I designed, built, and ran it alone, after taking over the AWS infrastructure from a senior engineer who left.
- Result
- 5M listings a day in testing. In daily production, the source's rate limits held it to about 5,000.
- Stack
- AWS Lambda (Docker images), SQS, EventBridge, RDS PostgreSQL, RDS Proxy, CloudWatch, SNS, ECR, Python
- Scope
- Marketplace listings for the 7 vehicle brands the company handles, collected every day
The problem
The company's price-prediction work needed broad, fresh market data. The target was about 5 million listings a day across the 7 vehicle brands it handles.
The pipeline had to be reliable, stay within budget, and cope with a source that limits how fast it can be read.
Architecture
- Schedule EventBridge starts each crawl cycle. A small Lambda resets the day's targets.
- Queue SQS holds the work, replacing an earlier design that used the database as a queue. Failed messages go to a dead-letter queue.
- Workers Lambda functions, packaged as Docker images, fetch and parse pages in parallel.
- Storage Listings land in RDS PostgreSQL, with RDS Proxy for connection pooling. Writes are incremental and atomic.
- Alerts CloudWatch metrics and alarms, with notifications through SNS.
- Retry and recoverySafe target claiming, retries, stuck-job recovery, and DLQ redrive
- Monitoring LambdaRead-only job that reports database health to CloudWatch
- Health checksAWS CLI scripts that run on any OS
Built to slow down, not fall over
A data source that limits request rates will push back. The pipeline was designed for that:
- Circuit breaker. When pages start getting blocked, workers pause instead of retrying harder. A shared cooldown state lives in PostgreSQL.
- Adaptive rate limiting. Request timing has jitter, the pipeline backs off automatically when the source pushes back, and workers ramp up gradually after a cooldown.
- Safe concurrency. Workers claim targets safely, retry, and recover stuck jobs, so many of them can run in parallel without clashing.
Observability and operations
- Alarms for what matters. CloudWatch alerts me when the orchestrator stops, workers fail, Lambda throttles them, the queue backs up or its messages grow old, pages get blocked, database writes fail, or the daily count of listings drops.
- Health checks. AWS CLI scripts check the pipeline from any machine, and a small read-only Lambda posts database health to CloudWatch.
- Written procedures. They say how to throttle or stop the Lambdas and purge the queues, so a runaway job cannot run up costs.
Hardening after senior review
A senior engineer reviewed the pipeline, and I hardened it from that review:
- Every database query has a timeout, and every write is atomic.
- Workers fetch only URLs on an allow-list and reject responses that are too large.
- A page that comes back suspiciously empty fails fast, and each failure is recorded with its context.
-
I removed an expensive
COUNT(*)from the hot path and added tools for index maintenance. - Retention policies for ECR images and CloudWatch logs keep storage costs down.
Results
- 5M/daythe target, reached in testing on the smallest RDS instance
- ~5,000/daythe rate the source's limits allowed in production
- Dailycollection, storage, and alerts in production
- 7 brandscovered for the company's pricing research
What I learned: the pipeline could handle 5 million listings a day, but the source could not. When it tightened its rate limits, the sustainable rate fell to about 5,000 listings a day, and that became normal operation. At this scale the data source, not the infrastructure, sets the limit, so backing off has to be part of the design from day one.
This page leaves out employer code, internal data, and private URLs, and it does not name the data source.