Webhook Delivery System
Challenge
Webhook delivery requires guaranteed async processing, retry logic on failure, full audit trails, and operational visibility, none of which you get from a simple HTTP call inside an API handler. Built on a producer/consumer architecture with Redis as the message queue.
Approach
- Two-tier AWS architecture: application layer (FastAPI, RQ worker, Redis, Nginx) on Instance 1, data and observability layer (PostgreSQL, Prometheus, Grafana) on Instance 2, communicating exclusively over a private VPC network. No internal traffic crosses the public internet.
- Decoupled delivery pipeline: API accepts the event, persists to PostgreSQL, and pushes a job to Redis, returning immediately. The RQ worker processes delivery asynchronously, retrying on failure. Every attempt is recorded. The API never blocks on delivery.
- Runtime secrets fetched from AWS Parameter Store via IAM role at deploy time. No secrets on disk, no secrets in the repository. GitHub Secrets used only for CI/CD pipeline operations.
- Four-job GitHub Actions pipeline: test, build, deploy-staging (auto), deploy-production (manual approval gate). Single Docker image, different entrypoint commands for API and worker: one build artifact, two runtime roles.
- Prometheus scrapes four targets (API, node-exporter, redis-exporter, self). Two active alert rules: target-down fires immediately, high error rate fires when 4xx/5xx ratio exceeds 5% over a 5-minute window.
Infrastructure
Outcome
Demonstrates a private two-tier AWS design, asynchronous delivery with retries and audit records, and alerting on failures and error rates.