Social Feed Design: Fan-out, Ranking and Privacy

Social feeds: choose where to pay for the work

This exercise designs a feed for a hypothetical photo-sharing community. It does not describe Instagram's current databases or fan-out implementation. Meta's published discussion of Suggested Posts distinguishes candidate generation from candidate selection; that is useful background for separating retrieval from ranking here. Meta engineering: how Instagram suggests new content.

Start with the access pattern

Assume one million daily readers, ten feed requests per reader, and 100,000 new posts per day. The averages are approximately 116 feed requests and 1.16 posts per second. Average traffic hides the expensive part: follower counts are uneven, and readers arrive in bursts.

With 200 eligible followers per ordinary post, fully materializing the feed creates 20 million feed entries per day. At a hypothetical 24 bytes per entry, that is 480 MB before indexes, allocator overhead, replicas, and retention. One account with two million active followers can add two million entries from one post. This is a distribution problem, not a reason to select a database by brand.

Compare the work, not just the names

A push feed still requires a range read, authorization checks, and post hydration. Returning 20 posts is not literally constant work regardless of page size. A pull design can use per-author indexes and bounded parallel reads; it does not necessarily scan every post.

A break-even example

Suppose an author posts twice a day and has 200,000 active followers. Push requires 400,000 feed-entry writes daily. If only 20,000 of those followers visit daily, materializing for everybody wastes much of that work. If those 20,000 readers request ten pages each, pull introduces up to 200,000 author-list lookups before batching and caching.

The counts are not interchangeable: a batched write and a remote read have different costs. Use them to form a test. Replay the same follower distribution through both designs and compare write cost, read latency, and backlog age. A threshold based only on total follower count ignores posting frequency and active readership.

For this workload, choose push for ordinary authors and pull for authors whose expected fan-out exceeds a configured work budget. Keep the choice in author metadata so it can change without rewriting the client protocol.

Follow one post through the system

Commit the post and an outbox event together. A worker reads the event, enumerates eligible followers in resumable batches, and inserts (viewer_id, post_id) with a uniqueness constraint. If it crashes after batch 12, it can replay that batch without duplicating the post.

A feed request combines the viewer's inbox with recent posts from pull-mode authors, deduplicates by post ID, filters access, and ranks the remaining candidates. Media lives in object storage behind an appropriate delivery layer; the inbox contains references, not image bytes. Preserve a chronological fallback if the ranking service is unavailable.

Switching an author from push to pull needs an overlap period. Otherwise posts can disappear between the last materialized event and the first pull cutoff. During overlap, deduplication prevents a post appearing twice.

Privacy and deletion change the answer

Cached membership is not permission. A private account may remove a follower after a post ID has entered that follower's inbox. Check current access before returning the post, and enforce access on the media delivery path as well. A long-lived public image URL would undermine a correct feed API.

Use deletion events to clean materialized lists, but enforce a tombstone at read time so cleanup lag does not expose deleted posts. For availability, stale ranking scores may be acceptable; stale authorization decisions may not be.

Pagination must define a stable view

An offset can skip or repeat posts when new entries arrive. For a chronological feed, a cursor containing the last (created_at, post_id) pair supplies a deterministic tie-breaker. For a ranked feed, scores can change between pages. Choose a short-lived session snapshot or carry a ranking version and a bounded set of seen IDs. State the trade-off: a stable scrolling session is less fresh than a fully recomputed page.

A useful review checks three cases: a high-follower author posts during backlog, a blocked author remains in a cached inbox, and a new post arrives between page one and page two. Measure stale-post age and duplicate rate alongside latency; a fast response with missing or unauthorized posts is not a successful feed.