# System Scalability Checklist

Use this checklist when designing for horizontal scalability and high availability.

## Load Estimation

- [ ] **Current load:** [N] requests/second, [N] concurrent users
- [ ] **Peak load:** [N] requests/second (expected during [event/time])
- [ ] **Growth projection:** [X]% yearly growth
- [ ] **Target capacity:** Support [N]x current load

## Scalability Dimensions

### Horizontal Scaling (Preferred)

- [ ] **Stateless services:** All application servers are stateless
- [ ] **Session storage:** Use Redis/Memcached for distributed sessions
- [ ] **File storage:** Use object storage (S3/GCS) instead of local filesystem
- [ ] **Auto-scaling:** Configure based on CPU/memory/RPS metrics
- [ ] **Load balancer:** Layer 7 (application-aware) load balancing

### Vertical Scaling (Limited)

- [ ] **Database instance:** Right-sized for current + 6 months growth
- [ ] **Cache instance:** Sized for working set + 20% headroom
- [ ] **Max capacity:** Identified vertical scaling limit

## Database Scalability

### Read Scaling

- [ ] **Read replicas:** [N] replicas for read-heavy workloads
- [ ] **Read/write splitting:** Route reads to replicas, writes to primary
- [ ] **Connection pooling:** PgBouncer/ProxySQL to limit connections
- [ ] **Query optimization:** Queries < [10ms] with proper indexing

### Write Scaling

- [ ] **Sharding strategy:** [Hash-based / Range-based / Geographic]
  - Shard key: [user_id / tenant_id / region]
  - Number of shards: [N] (plan for [10x] growth)
- [ ] **Write-ahead logging:** Asynchronous replication for replicas
- [ ] **Bulk operations:** Batch inserts/updates to reduce round trips

### Caching Strategy

- [ ] **Cache layers:**
  - L1: In-memory application cache ([Caffeine / Guava])
  - L2: Distributed cache ([Redis / Memcached])
  - L3: CDN for static assets ([CloudFront / Fastly])
- [ ] **Cache hit ratio:** Target > [90]%
- [ ] **TTL strategy:** Balance freshness vs load ([5min] for hot data, [1h] for warm)
- [ ] **Cache eviction:** LRU/LFU policy configured
- [ ] **Cache warming:** Pre-populate on deployment
- [ ] **Cache invalidation:** Event-driven invalidation for updates

## API Gateway

- [ ] **Rate limiting:**
  - Per-user: [N] req/min
  - Per-IP: [N] req/min
  - Burst allowance: [N] requests
- [ ] **Request throttling:** Queue requests during spikes
- [ ] **Response compression:** Gzip/Brotli enabled
- [ ] **API versioning:** Support [N] concurrent versions

## Asynchronous Processing

- [ ] **Message queue:** [Kafka / RabbitMQ / AWS SQS]
  - Throughput: [N] messages/second
  - Retention: [X] days
- [ ] **Worker pools:** [N] workers per queue
- [ ] **Backpressure:** Reject requests when queue length > [N]
- [ ] **Dead letter queue:** For failed message handling

## Content Delivery

- [ ] **CDN:** CloudFront/Fastly for static assets
- [ ] **Edge caching:** Cache-Control headers configured
- [ ] **Image optimization:** WebP format, lazy loading
- [ ] **Asset bundling:** Minified and bundled CSS/JS

## Data Storage Patterns

### Hot/Warm/Cold Data

- [ ] **Hot data:** Last [7] days in primary DB (fast access)
- [ ] **Warm data:** Last [30] days in read replicas
- [ ] **Cold data:** Older than [30] days archived to S3/Glacier
- [ ] **Archival strategy:** Automated data lifecycle policies

### Data Partitioning

- [ ] **Time-based partitioning:** Partition by month/year for time-series data
- [ ] **Hash partitioning:** Distribute by hash(user_id) for even distribution
- [ ] **List partitioning:** Partition by region/tenant for isolation

## Connection Management

- [ ] **Database connection pool:**
  - Min connections: [N]
  - Max connections: [N]
  - Connection timeout: [Xms]
- [ ] **HTTP keep-alive:** Reuse connections to upstream services
- [ ] **Circuit breaker:** Prevent cascade failures

## Observability for Scalability

### Key Metrics

- [ ] **Golden signals:**
  - Latency: p50, p95, p99 response times
  - Traffic: Requests per second
  - Errors: Error rate (4xx, 5xx)
  - Saturation: CPU, memory, disk, network usage
- [ ] **Capacity metrics:**
  - Database connections used/available
  - Queue depth and processing rate
  - Cache hit/miss ratio
  - Thread pool utilization

### Alerts

- [ ] CPU > [70]% for [5] minutes → Scale out
- [ ] Memory > [80]% for [5] minutes → Investigate memory leak
- [ ] Database connections > [80]% → Add replicas
- [ ] Queue depth > [1000] → Add workers
- [ ] Error rate > [1]% → Page on-call

## Load Testing

- [ ] **Baseline test:** Measure current performance under typical load
- [ ] **Stress test:** Identify breaking point (max capacity before failure)
- [ ] **Spike test:** Test behavior under sudden 10x traffic spike
- [ ] **Soak test:** Run at [2x] typical load for [24] hours
- [ ] **Tools:** [k6 / JMeter / Gatling / Locust]

### Load Test Scenarios

- [ ] Scenario 1: [N] users browsing product catalog
- [ ] Scenario 2: [N] users checking out simultaneously
- [ ] Scenario 3: [N] API clients polling for updates
- [ ] Target performance: p95 < [200ms], error rate < [0.1]%

## Cost Optimization

- [ ] **Right-sizing:** Use smallest instance that meets SLA
- [ ] **Reserved instances:** [70]% reserved, [30]% on-demand/spot
- [ ] **Auto-scaling policies:**
  - Scale up: When CPU > [70]% for [2] minutes
  - Scale down: When CPU < [30]% for [10] minutes
  - Min instances: [N], Max instances: [N]
- [ ] **Database optimization:** Remove unused indexes, optimize slow queries
- [ ] **CDN optimization:** Increase cache TTL where possible

## Geographic Distribution

- [ ] **Multi-region deployment:**
  - Primary region: [us-east-1]
  - Secondary region: [eu-west-1]
  - Failover: Automatic DNS failover
- [ ] **Data locality:** Store user data in nearest region (GDPR compliance)
- [ ] **Latency optimization:** < [100ms] within region, < [300ms] cross-region

## Disaster Recovery

- [ ] **RTO (Recovery Time Objective):** [X] hours
- [ ] **RPO (Recovery Point Objective):** [X] minutes
- [ ] **Backup frequency:** Database backup every [X] hours
- [ ] **Failover testing:** Quarterly DR drills
- [ ] **Data replication:** Asynchronous cross-region replication

## Security at Scale

- [ ] **DDoS protection:** CloudFlare/AWS Shield enabled
- [ ] **Rate limiting:** Per-IP and per-user limits
- [ ] **WAF rules:** Block common attack patterns
- [ ] **Certificate management:** Auto-renewal with Let's Encrypt/ACM

## Deployment Strategy

- [ ] **Blue-green deployment:** Zero-downtime releases
- [ ] **Canary releases:** [10]% traffic to new version initially
- [ ] **Feature flags:** Toggle features without deployment
- [ ] **Rollback plan:** Automated rollback if error rate > [X]%

## Checklist Before Production

- [ ] Load tested at [3x] expected peak traffic
- [ ] Auto-scaling policies validated
- [ ] Database read replicas configured
- [ ] Caching strategy implemented and tested
- [ ] Monitoring and alerts configured
- [ ] Disaster recovery plan documented and tested
- [ ] Cost monitoring and budgets set
- [ ] Security review completed
- [ ] Runbooks created for common incidents
- [ ] On-call rotation established