Core Concepts Overview
Use core concepts as tools for explaining a workload, not as vocabulary to recite.
What, why, when and how
What is it?
The core concepts are the foundations for reasoning about request paths, contracts, authoritative state, access paths and distributed trade-offs.
Why is it needed?
They let you explain why a user operation works and which boundary fails before choosing a product.
When should I use it?
Use this map when a mock reveals an unclear request, query, freshness or ownership decision.
When should I choose something else?
Do not treat the nine topics as a component checklist. A small baseline may not need a distributed cache or sharding.
How do I use it?
Follow one accepted write and one bounded read, map each decision to a foundation, then test the dominant failure and workload limit.
Can you show a concrete example?
A feed projection can still contain p7 after visibility revision 2 makes it private. Under a strict access policy, the response uses an authority that can enforce the current policy before returning content.
Read the diagram
Commit post p7 with event e7: accepted, not in every feed; Feed candidates may lag; read a bounded ordered page; Current visibility controls what content may be returned
Follow the detailed diagrams below to trace the additional state changes and failure cases behind these answers.
Outcome: explain how the nine foundations cooperate in one feed request, and choose the next chapter from a concrete weakness in your design.
All identifiers and rates on this page are exercise assumptions. We will design a small tenant-scoped feed: an author publishes a post, followers read their recent candidates, and a private or deleted post must not be disclosed by a stale projection. Begin with one relational owner and bounded queries. The goal is to understand why a mechanism enters the design.
Establish the contract before the components
Agree that publishing succeeds when the authoritative post and its publication record commit. Feed inclusion may lag under a declared freshness target. A candidate ID in a feed is not permission to return its content. Decide whether a visibility change must affect reads begun after that change completes; if so, specify an authorization source and read consistency that can enforce that policy. A replica or permission cache that may be stale cannot silently satisfy it.
| User action | API promise | State that enforces it |
|---|---|---|
| Publish a post | Return a stable post ID after acceptance commits | Post record, operation key, outbox event |
| Browse recent feed | Return a bounded ordered page of eligible posts | Candidate access path plus policy check |
| Make a post private | Commit a visibility revision under the agreed ordering policy | Authoritative post/policy version |
Ask how repeated publication, equal timestamps and revoked access behave. These questions select APIs, keys and consistency before the diagram grows.
Follow the write and the read as separate paths
The API authenticates the caller and checks tenant and author scope. One transaction stores post p7 at version 1, the client operation result and event e7. Returning p7 now proves acceptance; it does not prove every follower inbox already includes it. An outbox publisher can retry e7 after a lost acknowledgement. The projector therefore upserts the applicable post revision instead of blindly adding a duplicate candidate.
For a feed read, use an access path keyed by tenant and follower, ordered by a stable sequence or by time and a tie-breaking ID. Retrieve a bounded number of candidate IDs, hydrate current data and apply authorization before returning content or snippets. If filtering removes many candidates, set a scan/work limit; do not turn a request for twenty posts into an unbounded search for twenty eligible ones. The cursor represents progress through the candidate order, not a permanent permission grant.
Read the diagram
Commit p7/v1 and outbox e7; return p7; Project p7/v1 as a follower candidate; Commit visibility v2 = private; A stale candidate does not grant disclosure; Version-aware repair updates the projection
Map each foundation to a question you can answer
| Foundation | Decision in this example | Evidence to produce | Follow-up that tests it |
|---|---|---|---|
| Networking Essentials | Request route, connection reuse and deadline | A latency budget for the API and its dependencies | Which retry still fits the remaining deadline? |
| API Design | Publish identity, acknowledgement and feed cursor | One write and one bounded read contract | What does the same operation key return after a timeout? |
| Data Modeling | Post truth versus inbox projection | Keys, ownership and relationships | Which record is authoritative after a visibility change? |
| Database Indexing | Find one follower's recent candidates | Filter/order matched to a composite access path | How do equal timestamps affect pagination? |
| Caching | Reuse eligible work with declared freshness | Cache key, expiry, invalidation and miss policy | Can a late fill restore an older candidate list? |
| Sharding | Distribute an owner whose measured capacity is exceeded | Routing key, hot-key analysis and migration | Does one celebrity concentrate fanout or reads? |
| Consistent Hashing | One possible placement rule | Key movement and membership-version trace | What additional protocol transfers data safely? |
| CAP Theorem | Operation behavior while owners cannot communicate | A completed-write/later-read partition history | Which operation becomes unavailable under the required consistency? |
| Numbers to Know | Test fanout and candidate-read costs | Units and the threshold that changes a choice | What happens when incoming work exceeds completion rate? |
Add scale only after the arithmetic exposes it
Assume 100 new posts/s and 200 eligible followers per ordinary post. Eager fanout produces roughly 20,000 candidate writes/s before retries and metadata overhead. A single author with one million followers creates a burst far beyond that average. These assumptions justify comparing eager, lazy and hybrid candidate construction; they do not establish that any storage system can sustain the workload.
A bounded worker queue can smooth a temporary burst, but sustained input above processing capacity increases lag. Partitioning by follower distributes inbox storage while a huge author's fanout still needs bounded expansion. A read replica or cache can help tolerant reads, but it does not repair an undefined authorization contract. Explain which load each addition reduces and which new freshness or recovery policy it introduces.
Recover a projection without weakening access control
Suppose p7's visibility becomes private at revision 2 while the projection still carries revision 1. Under our strict policy, the serving path checks a source that can observe the required policy state before disclosing p7. If that check is unavailable, choose the failure response from the policy; do not serve a stale private post merely to fill the page. Returning fewer items or an error may be acceptable, but must be declared.
Repair the projection with version-aware updates and deletion markers. For a rebuild, explain how a consistent snapshot and change-stream position fit together, replay changes after that boundary, compare results and switch to the new generation. A timestamp alone is not automatically a safe change-stream checkpoint. Preserve the old read generation only while it remains compatible with the access policy.
Diagnose and transfer
Question: A feed read is slow because it scans thousands of candidates, most of which are now ineligible. Would adding a cache of the response solve the privacy problem?
Show answer and explanation
Answer: A cache can reduce repeated work only within its authorization and freshness contract. It cannot make stale permission safe. First bound candidate scanning and measure why candidates are rejected; repair the projection and define a policy-aware cache key or revalidation path. Keep strict policy decisions at an authority that can enforce them.
Question: Transfer the same idea to restaurant search. Which responsibility remains, and which does not?
Show answer and explanation
Answer: Candidate retrieval followed by exact eligibility checks remains. A follower-inbox fanout model does not automatically belong in restaurant search. Replace it with the actual spatial/text access paths and keep the shared distinction between a derived candidate and authoritative eligibility.
Continue with the weakest demonstrated boundary
Return to Delivery Framework if the request is incomplete. Choose indexing if the query has no bounded access path, caching if freshness is unclear, and sharding if routing and skew are the demonstrated limit. Study the deeper chapters before making guarantees about their mechanisms.
Primary references: PostgreSQL transactions, multicolumn indexes, and AWS transactional outbox. The feed design and numbers above are original teaching assumptions.