LLD-013 · DESIGN PRACTICE
Rate Limiter
Exact time accounting, atomic admission, safe cleanup and versioned policy changes.
Local Java 21 reference verified: 547 checks. Learner drafts are not executed. Author self-review; independent review pending.
Existing introductory lesson · Version 1
Understand the problem
Build a local admission controller that allows a short burst while enforcing a refill budget over time. It must atomically refill and consume tokens for concurrent callers, preserve fractional credit, bound tracked-key memory and change policy without granting a fresh budget to existing callers.
Before you start
- Concurrency and correctness
- Hash maps and numeric bounds
- Monotonic time vs wall clock
Scope and explicit decisions
- One Limiter instance in one JVM. Every key shares the configured capacity/rate policy; budgets are independent per key. It is not a global distributed quota.
- capacity, tokensPerSecond and request cost are positive longs. Cost above capacity is invalid because the request can never be admitted; it does not receive a misleading finite retry time.
- The trusted time adapter supplies nonnegative elapsed nanoseconds from one origin, monotonic across all calls. Tests pass time directly. A runtime adapter must sample time inside its serialized admission boundary to avoid stale sampled timestamps arriving out of order.
- One token equals 1,000,000,000 fixed-point units. Integer tokensPerSecond times elapsed nanoseconds yields exact refill units. BigInteger protects intermediate products from long overflow.
- The first request for a new key starts full. Existing-key rejection still stores accrued fractional credit and last observed time. wholeTokens floors the remaining fraction only in the response.
- retryAfterNanos is a ceiling estimate under unchanged policy and no competing consumers; it is not a reservation or promise of future admission. Values beyond long range saturate to Long.MAX_VALUE.
- At maxKeys, a new key is rejected with an explicit capacity error. Existing keys continue to work. Cleanup removes only fully refilled buckets, whose recreation is equivalent to retained state.
- Increasing policy capacity never tops up existing buckets. At cutover refill under the old rate, clamp to the new capacity, then use the new rate. New keys after cutover start at the new full capacity.
Worked examples
Input: capacity=10, rate=2/sec. At t=0 request cost 10, then cost 1.
Outcome: First allowed with zero tokens. Second rejected with retryAfterNanos=500000000.
The first request spends the burst; refill restores one token every half-second.
Input: Empty bucket, old rate=2/sec; change to rate=4/sec, capacity=20 at t=0.5sec.
Outcome: Existing key has one token at cutover, not two or twenty. At t=0.75sec it earns one more under the new rate.
Historical time is evaluated using the old policy and capacity increase is not a budget grant.
Your design attempt
Each language has a separate draft across all three levels. Saving is device-only and does not sync to your account. Anyone using this browser profile can load it. Loading is always explicit.
Java 21 reference code is provided for local study. Use Run program in the editor to execute your code and test driver. Program runs show output; they do not grade your design.
Run program · Java 21
No unsaved changes · 231/20000
Self-review
Checks apply to this visit. Self-review is not a passed submission or verified mastery.
Try before revealing
Hint 1
Separate the maximum burst from the rate at which credit returns.
Hint 2
Write the balance equation with units. What survives a rejected request at half a token?
Hint 3
Put refill, check, spend and save under one owner, including cleanup and policy changes.
Requirements and invariants
R1 — A request is admitted only if the refilled budget covers its entire cost. allow protects I1.
R2 — Concurrent requests share one atomic refill-and-consume decision. allow protects I2.
R3 — Key count stays at or below maxKeys without resetting depleted budgets. evictFull protects I3.
R4 — Policy change is version-guarded and preserves accumulated credit. replacePolicy protects I4.
I1: For every bucket, 0 <= fixed-point units <= capacity × 1,000,000,000.
I2: Each accepted cost is deducted once from the same synchronized state used by all contenders.
I3: The map contains at most maxKeys; eviction grants no extra tokens compared with keeping a full bucket.
I4: At policy cutover, old policy earns all pre-cutover credit; existing budgets are clamped, never replenished merely by increasing capacity.
Derive the design
Fixed window: correct for its contract, insufficient for smooth refill
A fixed-window limiter counts requests in each one-second interval and resets at the next boundary. It is compact and predictable, but ten requests at 999,999,999 ns followed by ten at 1,000,000,000 ns all pass for capacity ten. That is not a coding defect: the window contract permits it. If the requirement is a burst budget with gradual recovery, derive token bucket rather than pretending fixed windows satisfy it.
Alternative: A sliding log records timestamps and gives an exact rolling-window count but uses memory proportional to accepted requests in the window and cleanup work. A sliding counter approximates that count. A leaky-bucket queue smooths output by delaying work; this token bucket rejects immediately rather than queuing.
Turn requirements into quantities
Ask whether users require an exact rolling-window count, a sustained rate with bursts, or delayed smooth output. Those are different contracts. Here capacity describes burst, rate describes recovery, and cost can represent weighted work.
Preserve fractions before optimizing arithmetic
With two tokens per second, a quarter second is half a token. Store tokens in billionths so nanosecond elapsed time times an integer rate has an exact representation. Floor only the response display, never the internal balance.
Choose the atomic boundary before the data structure
The operation is refill-and-consume, not get and put. The synchronized method protects both the bucket and its policy/time context. Use expectedVersion to prevent a stale administrator from overwriting a newer policy.
Make resource exhaustion an explicit API result
Rate-limited is not the same as identity-store full. This example throws a distinct capacity error for a new key. Production should map it to a deliberate overload response and measure saturation; do not silently grant untracked requests.
Only reclaim equivalent state
A full retained bucket and a freshly created bucket start with the same credit. A depleted retained bucket and a new bucket do not. That equivalence is the proof for evictFull, and the shared monitor prevents a concurrent consumption between the check and deletion.
Responsibilities and ownership
Limiter
Owns: Bucket map, policy version, observed time, key limit
Methods: allow, evictFull, replacePolicy
Single monitor is the authoritative local admission owner.
Policy
Owns: Version, capacity and integer refill rate
Methods: Validated immutable record
No mutable configuration read midway through a request.
Bucket
Owns: Exact fixed-point balance and last update time
Methods: Private immutable record
Never exposed for direct caller mutation.
Decision
Owns: Allowed result, floored remaining tokens, ceiling retry estimate
Methods: Immutable accessors
An estimate is not an allocation of future capacity.
FixedWindow
Owns: One-key window index and used count
Methods: allow(now)
Correct comparison baseline with a different burst guarantee.
Cleanup and policy replacement use the same owner. Network calls are outside this local contract.
- Trusted key / time adapter
↓ Validate identity, cost and elapsed time
Limiter monitor↓ Refill → compare → consume → save
Bucket map + policy↓ Return allowed / remaining / retry estimate
Immutable Decision
These are alternative outcomes from the comparison, not sequential steps. Both store fractional state; the table below provides the exact execution order.
Compare against cost × UNIT
Return rejection with ceiling retry estimate
Data model, schema and relationships
A policy defines capacity/refill meaning; a bucket owns token balance and elapsed-time state for one scoped key. A decision is an observation, not another independent mutable balance.
Implementation boundary: Token bucket state lives in a local map with policy-version changes. Persistence design would need one authoritative atomic update across processes; a cache replica is not sufficient.
These are proposed logical schemas with field contracts, not executable database migrations. PK identifies a row, FK references an existing parent, UK is a uniqueness rule. A composite PK identifies the combination. Nullable references are marked with ?. A foreign key protects identity; cross-row limits, timing and state transitions also require the transaction rules below.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Entity definitions and field contracts
limit_policy
Immutable policy interpretation.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | text · PK | Policy/version identity |
capacity | bigint · VALUE | Maximum burst tokens |
refill | bigint · VALUE | Declared rate units |
Code mapping: Policy record.
- Positive bounded rate/capacity and explicit policy migration semantics.
limit_subject
Quota scope.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | text · PK | Tenant/user/route compound identity |
scope | text · VALUE | Declared isolation boundary |
Code mapping: Bucket key input.
- Normalize key and bound distinct-key admission/memory.
bucket
Current refill/consumption owner.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | uuid · PK | Bucket identity |
policy_id | text · FK | Pinned policy · limit_policy.id |
subject_id | text · FK | Quota subject · limit_subject.id |
tokens | bigint · VALUE | Available fixed-point tokens |
last_tick | bigint · VALUE | Trusted elapsed time |
Code mapping: Mutable local bucket.
- Unique(subject_id,policy_id) under explicit migration; 0<=tokens<=capacity.
limit_decision
Optional audit sample.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | uuid · PK | Decision identity |
bucket_id | uuid · FK | Authority used · bucket.id |
allowed | boolean · VALUE | Decision result |
Code mapping: Optional observability extension.
- Audit sampling must not become an unbounded synchronous hot path.
Relationship cardinalities and their reasons
| Parent → child | Multiplicity | FK / why |
|---|---|---|
| limit_policy → bucket configures | 1 parent per child; 0..* children per parent | bucket.policy_id → limit_policy.idBucket values must be interpreted under their policy version. |
| limit_subject → bucket owns quota | 1 parent per child; 0..* children per parent | bucket.subject_id → limit_subject.idA subject may use distinct policies; migration cannot create accidental extra burst. |
| bucket → limit_decision produces decisions | 1 parent per child; 0..* children per parent | limit_decision.bucket_id → bucket.idAudits observe decisions and must not mutate token truth. |
Transaction boundaries
- Refill/clamp, compare requested cost and deduct under one bucket owner.
- Policy change must transform/reset/clamp state under a documented rule, atomically with active policy identity.
Indexes and uniqueness
- Unique bucket scoped key for direct lookup; expire idle buckets only under a safe lifecycle contract.
- Avoid writing every decision to a globally contended index; sample or aggregate asynchronous metrics.
Schema traps and failure cases
- Per-process buckets permit aggregate traffic above a supposed global quota.
- Wall-clock rollback or overflow in refill arithmetic must not mint tokens.
Trace the records through one operation
- Capacity 10 admits a burst of 10 under policy v1.
- Elapsed refill restores only up to capacity.
- Concurrent last-token callers serialize; one succeeds and the other is denied.
Method contracts
allow(key, cost, elapsedNanos) → Decision
Reject invalid input or time reversal before mutation. Refill, compare, consume and save while synchronized. Key-capacity failure is distinct from ordinary rate rejection.
evictFull(elapsedNanos) → removedCount
Scan K retained keys and remove only those whose refilled balance equals capacity at this instant. An exhausted bucket cannot be dropped to reset its budget.
replacePolicy(next, expectedVersion, elapsedNanos)
Reject stale expectedVersion or non-increasing next.version. Materialize old-policy refill into a temporary map, clamp balances to new capacity, then atomically replace configuration and state.
keyCount() → int
Synchronized observation of retained buckets; cannot exceed configured maxKeys.
What, when, why and how
Each concept is attached to the requirement it protects, its implementation, an alternative and a test.
Token bucket
- what
- A bounded budget grows with elapsed time and is spent by accepted requests.
- when
- The product permits a burst but needs a sustained refill rate.
- why
- Capacity and rate control different behavior: ten immediately, then two per second.
- how
- Refill units=min(capacity×UNIT,oldUnits+elapsedNanos×tokensPerSecond). Consume cost×UNIT only if sufficient. BigInteger keeps arithmetic exact for long-sized parameters.
Code: refill, allow · Invariants: I1
Alternatives, cost and failure example
Alternative: Fixed windows are simpler when boundary bursts are acceptable; sliding logs suit exact rolling-window rules.
Cost: BigInteger allocation costs more than bounded fixed-width arithmetic. Operand sizes here are bounded by long inputs/products, but performance must still be measured.
Counterexample: Round each refill to whole tokens: repeated calls at quarter-second intervals discard credit and may starve a caller.
Test: At rate two, reject at 250ms with half-token retained; accept at 500ms.
Monotonic time and injectable clock
- what
- Measure elapsed duration using a source that cannot move backwards, supplied as a dependency.
- when
- Refill or expiry depends on time and tests must avoid sleeping.
- why
- Wall-clock corrections can create negative refill or an unintended fresh burst.
- how
- This API accepts trusted elapsed nanoseconds and rejects values older than the last observed service time. A real adapter uses a monotonic origin and samples within the serialized boundary.
Code: validateTime, refill · Invariants: I1
Alternatives, cost and failure example
Alternative: Inject a LongSupplier sampled inside allow instead of accepting now; this reduces caller misuse. A shared store must define its own time authority.
Cost: Explicit time makes tests deterministic but adds a contract obligation. Raw System.nanoTime values may be negative; normalize against an origin and handle long-running wrap according to a declared horizon.
Counterexample: Two threads sample 100 and 101; the 101 call arrives first. The older 100 call is rejected instead of silently earning negative credit.
Test: Call with a smaller time and assert error; use exactly 499999999 and 500000000 ns boundary inputs.
Linearizable compound admission
- what
- Each concurrent call appears to make one indivisible budget decision.
- when
- Several workers call allow for the same key.
- why
- A safe map cannot stop two callers from spending the same last token after separate reads.
- how
- The monitor covers lookup, refill, comparison, deduction and write. It also serializes cleanup and policy replacement.
Code: allow, replacePolicy · Invariants: I2, I4
Alternatives, cost and failure example
Alternative: Per-key locks can improve parallelism, but shared policy changes and key deletion require coordinated generations/lock ownership.
Cost: Different keys serialize in this reference. This is deliberately simple; sharding is justified by measured contention, not aesthetics.
Counterexample: Read one token in two threads, then both write zero: two requests passed using one token.
Test: Release twenty concurrent unit requests at t=0 with capacity ten; exactly ten must succeed.
Bounded state and semantics-preserving eviction
- what
- Limit retained identities without resetting live budget.
- when
- A public endpoint can see many distinct keys.
- why
- Unbounded maps exhaust memory; evicting a depleted key grants a new burst on recreation.
- how
- Reject new keys when full. evictFull refills for comparison and removes only full buckets. A recreated full bucket has the same budget at eviction time.
Code: evictFull, keyCount · Invariants: I3
Alternatives, cost and failure example
Alternative: TTL cleanup is safe only when TTL guarantees full refill under the relevant policy and no concurrent replacement changes that conclusion.
Cost: Cleanup is O(K) under the monitor and can pause admission. Schedule bounded sweeps in a larger implementation; preserve atomic equivalence.
Counterexample: Evict a zero-token key merely because it is least recently used; the next request creates it with capacity ten.
Test: At t=0.5 depleted keys remain; at t=5.5 they are full and may be safely removed.
Versioned policy cutover
- what
- Change configuration with a guard and a declared credit-transfer rule.
- when
- Operations change limits while clients are active.
- why
- Applying the new refill rate retroactively or topping up capacity gifts unintended tokens.
- how
- Refill each bucket with OLD rate to the cutover time. Clamp to NEW capacity. Require expectedVersion and next version greater than current, then publish all state atomically.
Code: replacePolicy · Invariants: I4
Alternatives, cost and failure example
Alternative: Resetting all budgets is simpler only if the product explicitly accepts a fresh burst; it must not be hidden behind a generic config update.
Cost: O(K) cutover and temporary O(K) map. A lazy migration needs per-bucket policy versions and more complex accounting.
Counterexample: Changing rate two→four at 0.5s incorrectly credits two tokens instead of the one earned under the old rate.
Test: Test old-rate earning, no top-up on capacity increase, clamp on decrease and stale-version rejection.
Read and run the Java reference
Reveal complete Solution.java
Java 21. Save as Solution.java; compile with javac --release 21. The repository also contains Test.java and the lld:java verification command. This reference implements the local contract above.
refill: exact units and clamp
elapsed is a BigInteger subtraction after monotonic validation. Multiplying by integer tokensPerSecond produces fixed-point units. Clamp to ceiling so a long idle period cannot bank unlimited burst credit.
allow: one transaction over budget
Validate cost and key before reading time. New keys check capacity. Compute need, choose allowed, deduct only on success and save fractional units on either branch. retryAfter uses ceiling division and saturates rather than overflowing.
evictFull: equivalence, not arbitrary LRU
Recompute potential balance at now for each entry. A full result may be removed because recreation is equivalent. Depleted entries stay even if they are old; no unbounded new-key admission is implied.
replacePolicy: build before publish
Validate versions, refill using the still-current old policy into a temporary map and clamp each balance. Publish map, policy and observedTime together under the monitor. Failures before replacement leave the original map authoritative.
Complexity and limits
- allow performs expected O(1) map access plus exact arithmetic on bounded-size numbers derived from long inputs. General BigInteger complexity depends on bit length; this is not a universal constant-time cryptographic primitive.
- Space is O(K) with K<=maxKeys. Input key strings also consume memory; a production key resolver must bound key length and normalize authenticated tenant/user identity.
- evictFull and replacePolicy are O(K); cutover additionally uses O(K) temporary state. Both hold the monitor, so large K can produce admission latency spikes.
- FixedWindow is O(1) time/space for one key; SlidingLog uses O(A) memory for A retained accepted timestamps, with amortized expiry and worst-case cleanup bursts.
- A single JVM monitor guarantees local serialization, not fairness, lock-wait bounds, durability or global distributed quota.
Walk through the actual reference
These walkthrough fixtures are emitted only after the Java harness verifies their operations and expected state. Concurrency tests additionally exercise competing callers.
Burst, rejection and fractional refill
Step 1 of 4: At 0 ns consume 10
Allowed; 0 tokens remain
Invariant: Budget never negative
Read the complete trace table
| Action | State after action | Invariant |
|---|---|---|
| At 0 ns consume 10 | Allowed; 0 tokens remain | Budget never negative |
| At 0 ns consume 1 | Rejected; retry after 500000000 ns | No token spent on rejection |
| At 250000000 ns consume 1 | Rejected; half a token accumulated | Fractional credit retained exactly |
| At 500000000 ns consume 1 | Allowed; 0 tokens remain | Refill and consumption form one atomic decision |
Mistakes and counterexamples
Reset the balance every second and call it token bucket
Failing case: Ten requests just before and ten just after a window boundary all pass.
Repair: Name this fixed-window behavior; use elapsed refill if burst recovery must be gradual.
Drop fractional refill on each request
Failing case: At rate two/sec, checking every 250ms always rounds 0.5 to zero.
Repair: Retain exact units; round only wholeTokens in the response.
Evict any idle/LRU key under memory pressure
Failing case: An exhausted key is removed and returns immediately to a full bucket.
Repair: Only evict equivalent full state; otherwise reject new keys or use another bounded ownership scheme.
Multiply long capacity by a billion without checks
Failing case: A large valid long capacity overflows negative during fixed-point conversion.
Repair: Use checked bounded arithmetic or BigInteger; test Long.MAX_VALUE parameters.
Treat retryAfter as guaranteed admission
Failing case: Another caller spends the token first, or a policy cutover changes recovery.
Repair: It is an estimate; retry must perform another atomic decision.
Engineer → Senior → Staff
Implement correct fractional refill and explicit input behavior.
Executed: token bucket and fixed-window comparison, weighted costs, exact boundaries and overflow-safe arithmetic.
Change exercise
With capacity one and rate three/sec, calculate the first nanosecond at which a request can pass.
Worked answer guidance
Ceiling(1e9/3)=333333334ns. The preceding nanosecond is insufficient. Keep fractional credit so repeated checks do not destroy progress.
Interview follow-ups
Why does capacity differ from rate?
Capacity permits a short burst; rate controls refill. A capacity of ten and rate two admits ten immediately, but not ten every second.
Would you always use BigInteger?
No. If capacity, elapsed range and rate have tight enforced limits, checked long arithmetic is faster. Explain those bounds and overflow policy before replacing exact arithmetic.
Provenance and scope of evidence
Original teaching material with contracts defined by the chapter and Java 21 reference/tests. No claim of a vendor-specific distributed limiter implementation.
Reference verification is not independent editorial review, production deployment verification or a claim that distributed adapters are implemented.