Centralized Logging
Collect structured application logs into searchable storage with bounded agent buffers, explicit loss policy and affordable retention.
Collect structured application logs into searchable storage with bounded agent buffers, explicit loss policy and affordable retention. This worked design targets SDE-2; the scale figures below are exercise assumptions, not claims about a named company's implementation.
Clarify the prompt
| Ask the interviewer | Decision to record |
|---|---|
| Are logs best-effort diagnostics or regulated audit records? | Separate durability contracts. |
| What are ingestion-to-search latency, retention and query requirements? | |
| Which fields may contain personal data or secrets? | Redact before broad distribution. |
Requirements and scope
The core deliverable is: Collect structured application logs into searchable storage with bounded agent buffers, explicit loss policy and affordable retention. The first design implements the APIs below and the state constraints in the data model. Agree the latency, freshness, retention and failure policy before adding infrastructure. Every successful acknowledgment must have an explicit meaning: accepted intent, committed state or completed external work. These are separate milestones.
Capacity and bottleneck reasoning
Assume 100,000 events/s at 1 KB: 100 MB/s, 8.64 TB/day raw and 259 TB for 30 days. Three replicas multiply raw storage before compression and indexes. Hot searchable retention should be shorter than cheap archive retention.
Keep average and peak demand separate. Add replication, indexes, retries and safety headroom to raw arithmetic before selecting capacity. Load-test the busiest ownership key and recovery traffic as well as uniform happy-path throughput.
APIs and data model
- POST /ingest/batches {batchId,events} -> durable acceptance status
- POST /logs/search {tenant,timeRange,filters,cursor} -> bounded page
- GET /ingestion/status -> watermark and rejected counts
log(event_id,tenant,service,event_time,ingest_time,severity,trace_id,body,schema_version)
agent_checkpoint(file_identity,offset); partition(tenant,time_bucket)
archive_manifest(bucket,objects,checksums); retention_policy(tenant,class)Authorize resource ownership at the API boundary; do not trust IDs or state supplied by a client. Bind retry identity to the caller and payload for mutating operations. Reject an incompatible replay instead of returning an unrelated stored result.
Component relationships
Read the diagram
Follow the labeled relationship panels; they are separate directed paths, not one global timeline.
Step-by-step design
1. Establish the authority
Agents batch and compress structured records, persist bounded local spool files and advance checkpoints only after the documented acknowledgment. Rotation tracking uses file identity, not just a reused path.
2. Make retries and concurrent work safe
Collectors authenticate tenants, validate schemas and apply size limits/redaction. Append to a durable partitioned stream before acknowledging durable acceptance. Indexers batch into time/tenant shards while archives retain replayable immutable batches.
3. Scale and recover the critical path
Query gateways impose time bounds, quotas and cancellation. Limit high-cardinality indexing, maintain tiered retention and expose searchable-watermark lag. Audit records use a separate path with stronger rejection/availability policy instead of silently sampling them.
Worked request and concurrency example
A node emits event E and its batch reaches the durable log, but the acknowledgment disappears. Its agent resends the batch. Downstream deduplicates by event ID where exact counts matter; diagnostic search may explicitly tolerate duplicates. If indexing is down for ten minutes at 100 MB/s, at least 60 GB of logical backlog accumulates. Indexer catch-up must exceed incoming rate or search freshness never recovers.
Failure and recovery walkthrough
If the spool fills during a long collector outage, the product must choose backpressure, dropping selected diagnostics or stopping the application. Record drop counters locally; never promise unlimited lossless buffering on finite disks.
Read the diagram
This numbered trace follows one scenario; explain the durable evidence at each transition.
Decisions and alternatives
| Decision | Trade-off |
|---|---|
| Alternative 1 | Index every field for flexible search at high storage cost, or index a bounded schema and keep raw bodies in object storage for slower forensic queries. |
| Alternative 2 | Partition by time for retention, then tenant/service for isolation. Excess tiny shards waste resources; very large shards make recovery and rebalancing slow. |
Operational contract
Monitor accepted versus indexed counts, oldest stream lag, spool utilization, redaction failures and query cost. Test disk-full agents, duplicate batches, index loss and an archive rebuild with access controls preserved.
Choose alert thresholds from the agreed user-facing objective. Assign an owner for stuck work and a replay/repair procedure that preserves the original business identity. Bound queue depth, concurrency and recovery throughput so an outage does not produce an uncontrolled catch-up burst.
Foundation, senior and staff depth
- Foundation: explain the user action, model, first complete request and the correctness constraint without relying on a list of technologies.
- Senior: defend this lesson's partition/coordination boundary, capacity estimate and failure recovery; explain the cost of the rejected alternative.
- Staff: identify failure domains and operational ownership, stage a rollout with measured guardrails, and state when rollback stops being safe.
Practice before revealing the answer
Spend 45 minutes: 5 clarifying scope, 7 on capacity/APIs/state, 12 on the first architecture, 15 on the concurrency and failure examples, and 6 on trade-offs and operations. Draw your own design before comparing it with this chapter.
Question: Will adding more collectors solve an indexing backlog?
Show answer and explanation
Answer: No. Measure indexer service rate and search watermark lag. More collectors can accelerate ingestion into an already growing backlog; throttle or expand the constrained indexing/storage stage.
Verification and rollout
Reproduce the worked example using two concurrent actors or a repeated request where applicable. Inject the stated failure between the durable transition and its acknowledgment; verify the resulting state and user-visible outcome. For read/ML paths, test missing or stale dependencies and verify the documented fallback still enforces permissions. Use a small canary, compare correctness and user-facing latency with the baseline, and roll back when those guardrails regress. Do not use an average throughput number as evidence that a tail-latency target is met.
Primary technical reference
Official documentation for the mechanism used in this exercise. This lesson is an original interview design, not a description of undisclosed company internals.
Related study
Before this design, review these mechanisms and identify which owns the critical state transition:
Use the Interview Playbook to adapt the depth and timing, then return to the question breakdown overview.
Shared foundation: SLOs, Observability and Capacity
SLOs, Observability and Capacity: Define measurable user outcomes, calculate error budgets and diagnose finite-capacity failures with safe telemetry. Work its failure history before extending this design.
Will adding more collectors solve an indexing backlog?
Your design draft
Clarify assumptions, explain your approach, and test the difficult cases. Save your draft, then compare it with the study notes.
Self-review checklist
Self-guided practice. Automated AI feedback and code execution are not connected.