# How to Build a Scale-Proof Distributed Counter

**To build a scale-proof like counter, you must decouple user actions from database writes. Start with optimistic UI updates for instant feedback, log the event to a user-specific store, and process counts asynchronously using in-memory distributed buffers before periodically flushing aggregated totals to a persistent database.**

If I'm building a weekend side project, a single SQL `UPDATE` statement is fine. But at YouTube scale—where millions of users hit the like button simultaneously across the globe—that simple write becomes a massive architectural bottleneck. 

Here is how I would design a distributed system that handles this kind of write-heavy throughput without melting the database.


## Why can't I update the database immediately on every click?

Direct database updates fail at scale because high-volume concurrent writes to the same database row cause severe lock contention. To handle millions of writes per second, I must shield the database using an asynchronous, distributed processing layer.

Let's say a viral video goes live. Within seconds, ten thousand users hit the like button. If I try to execute `UPDATE videos SET likes = likes + 1 WHERE id = 123` ten thousand times simultaneously, the database has to lock that specific row for each transaction. This creates a massive queue, drives CPU usage to 100%, and eventually crashes the database.

To solve this, I decouple the user interface from the system of record. When you click that like button, the UI updates instantly. The application doesn't wait for a database roundtrip; it optimistically assumes success and increments the counter locally on your screen. 


## How do I design a system to handle millions of likes at scale?

To scale high-throughput counters, I split the operation into two parallel pipelines: a user-centric ledger and a video-centric aggregated counter. This decouples the identity tracking ("who liked what") from the high-frequency numerical aggregation ("how many likes total").

Once the client emits the request, I split the backend work into two independent flows:

*   **The User-Centric Flow**: I log the specific relationship (e.g., "User X liked Video Y"). This data is sharded horizontally by User ID, making it highly distributed and easy to scale. It ensures that if you refresh the page, I can look up your history and show that you already liked the video.
*   **The Video-Centric Flow**: I emit an asynchronous event—"User X liked Video Y"—into an ingestion pipeline to update the overall count. This pipeline is built to ingest a wild, constant stream of events without slowing down.

| Pipeline Component | Data Responsibility | Scale Strategy | Storage Medium |
| :--- | :--- | :--- | :--- |
| **User-Centric** | Tracks specific relationships ("User X liked Video Y"). | Sharded horizontally by User ID. | Relational DB or Key-Value Store. |
| **Video-Centric** | Tracks overall aggregations ("Video Y has N likes"). | Distributed in-memory counters & batching. | In-memory cache (Redis/Custom) -> Persistent DB. |


## How do I implement distributed in-memory counters?

Distributed in-memory counters ingest a stream of event messages across multiple parallel processes to track counts in local memory. A periodic background worker then sweeps these in-memory nodes, aggregates the counts, and flushes them to persistent storage in a single, batched database transaction.

Instead of hitting the database for every single event, I route the incoming stream of like events across multiple memory processes. Let's say I have five instances of a counter service running. When a "like" event occurs, it is routed to one of these instances, which increments an in-memory counter for that video. 

Every few seconds, a background process runs to reconcile these numbers. It reads all the distributed, in-memory counters across the processes, sums them up, and flushes the aggregated count to the persistent database. If a video receives 10,000 likes in three seconds across the fleet, the database experiences only a single write operation of `+10,000` instead of 10,000 individual writes. 

This is how I balance real-time user experiences with backend storage limits.


## FAQ

### What is an optimistic UI update and why is it used for likes?
An optimistic UI update is a frontend pattern where the client-side UI immediately reflects a successful state (e.g., highlighting the like button and incrementing the count) before the server confirms the action. I use it to ensure a fast, responsive user experience despite the asynchronous nature of the backend queue.

### How do distributed counters handle eventual consistency?
Because distributed counters rely on in-memory buffering and periodic database flushing, the public-facing like count is eventually consistent. While you will immediately see your own like, other users will see the total count update in small jumps every few seconds as background processes reconcile and write the batched counts to persistent storage.

### What happens to the counter if an in-memory node crashes?
If a node crashes before flushing, some in-flight counts can be lost. To prevent this, I run durable, append-only event streams (like Apache Kafka) behind the ingestion layer. If a worker node goes down, a new node can replay unprocessed events from the stream to reconstruct the correct counts before writing them to the database.