📝 Problem Description
Design an A/B testing platform that allows running experiments on product features. Assign users to variants consistently, track metrics, and determine statistical significance. Handle millions of users and thousands of concurrent experiments.
👤 Use Cases
1.
PM wants to creates experiment so that experiment configured and launched
2.
User wants to visits product so that assigned to experiment variant
3.
System wants to user takes action so that event tracked for analysis
4.
Analyst wants to views results so that sees statistical significance
✅ Functional Requirements
- •Create experiments with variants (A, B, C...)
- •Target specific user segments
- •Consistent user-to-variant assignment
- •Track conversion events
- •Calculate statistical significance
- •Gradual rollout percentage
⚡ Non-Functional Requirements
- •Assignment latency < 10ms
- •Support 10K concurrent experiments
- •Handle 1M assignment requests/sec
- •No bias in variant distribution
⚠️ Constraints & Assumptions
- •Assignment must be deterministic (same user → same variant)
- •Experiments shouldn't interfere with each other
- •Historical data must be preserved
📊 Capacity Estimation
👥 Users
100M users
💾 Storage
10TB (events data)
⚡ QPS
Assignments: 1M/sec, Events: 500K/sec
📐 Assumptions
- • 100M monthly active users
- • 10K active experiments concurrently
- • 2-5 variants per experiment average
- • 10 tracked events per user per session
- • ~10 sessions per user per month
- • Experiment duration: 2-4 weeks average
💡 Key Concepts
CRITICAL
Deterministic Assignment
Hash(user_id + experiment_id) % 100 for consistent bucketing.
CRITICAL
Statistical Significance
Chi-squared or t-test to determine if results are real.
HIGH
Experiment Isolation
Use experiment layers to prevent interference.
MEDIUM
Sample Size Calculation
Determine required users for statistical power.
💡 Interview Tips
- 💡Start with the assignment mechanism (hashing)
- 💡Discuss statistical significance and sample size
- 💡Emphasize the importance of consistent assignment
- 💡Be prepared to discuss experiment interactions
- 💡Know the difference between A/B tests and feature flags
- 💡Understand the metrics collection pipeline