File Synchronization: Versions, Conflicts and Recovery

File synchronization: protect the user's changes first

This is a design exercise for a team file-sharing service. The architecture and numbers below are illustrative. Dropbox's published account of its Nucleus sync engine provides one relevant real-world lesson: stable file identifiers and explicit rules for offline edits matter as much as moving bytes. That account describes its 2020 design; it does not establish a current Dropbox or Google Drive storage stack. Dropbox engineering: rewriting the sync engine.

A concrete workload

Assume 10,000 users, two devices each, and 20 file edits per user per day. That gives 200,000 new versions per day, or about 2.3 per second on average. If a workday burst is 50 times the average, plan to test approximately 116 version commits per second. Upload bandwidth depends on changed bytes, so version count alone cannot size storage.

For a 12 MiB file split into three 4 MiB blocks, a same-length edit confined to the first block requires uploading 4 MiB rather than 12 MiB, excluding metadata. An insertion near the beginning can shift every fixed-size boundary, changing all three hashes. Content-defined chunking can preserve more boundaries, but adds computation and a more complex format. Measure representative files before choosing it; compressed archives can change broadly after a small logical edit.

Make a version a manifest

Store immutable content blocks separately from the namespace. A file record has a stable file_id, parent directory, current version, owner, and permissions. Each version points to an ordered manifest of block identifiers and lengths. Moving a file changes its parent, not its identity or bytes.

A commit has four steps:

1. Authenticate the user and check write permission for the file.
2. Upload missing blocks to temporary storage; verify their length and digest.
3. In one metadata transaction, compare base_version with the current version, publish the new manifest, and append a change event.
4. Acknowledge the committed version. Other devices fetch changes using a durable cursor.

Publish the manifest only after its referenced blocks meet the chosen durability policy. A process crash after block upload but before commit leaves an orphan, which is recoverable garbage. A published manifest pointing at a missing block is a user-visible data-loss incident. The order of operations therefore favors harmless storage leakage over broken files.

Walk through an offline conflict

Both devices start from version 7. A laptop changes the budget and commits version 8. An offline phone changes the same file, then reconnects with base_version=7.

A blind last-write-wins update would discard one user's work. Our server rejects the stale replacement and preserves the phone's upload as a conflict version. The client shows both versions and lets the user merge or choose. For an opaque spreadsheet or image, a generic byte merge is unsafe. A collaborative text editor would require a separate merge model.

Deletes also need versions. Retain a tombstone long enough for offline devices to learn that the file was deleted; otherwise an old device can resurrect it. When a device's change cursor is older than retained history, require a full reconciliation rather than silently treating its local tree as authoritative.

Deduplication has a permission boundary

For this exercise, deduplicate within an account or shared workspace. Knowing a block's hash must not grant permission to download it. A global “we already have that file” response can also disclose information about another customer's uploads. Avoid exposing that signal across tenants; storage savings do not justify weakening access controls.

Encryption changes the design too. Randomized client-side encryption prevents the server from recognizing equal plaintext chunks. Do not promise both unrestricted server-side deduplication and private end-to-end encrypted files without explaining the protocol and its leakage.

Recovery and a useful failure drill

Maintain a grace period before reclaiming unreferenced blocks. A cleanup process must account for retained versions, pending uploads, and committed manifests, not just the current file tree. Restoring metadata without corresponding blocks is not a successful backup restore.

A practical drill is to disconnect a client immediately after every step of a version commit, then retry with the same operation ID. Verify that there is at most one committed version for that operation and that every published block is readable. Separately test a rename concurrent with a delete, two offline replacements, permission revocation during upload, and reconnect after tombstone expiry.

Track conflict frequency, oldest unsynchronized cursor, orphan bytes, and failed integrity checks. Those measurements help decide whether the next investment should be faster transfer, better conflict handling, or recovery tooling. A fictional durability percentage cannot make that decision.