Object Storage (S3 Design)

File System vs Object Storage

Why not just store files on a really big hard drive?

Block Storage (Hard Drive): Good for OS, databases. Low latency. Hard to share across servers. Network File System (NFS): hierarchical folder structure. Good for sharing. Scale depends on the filesystem, metadata architecture, and workload.

Object Storage (S3): Bucket/key interface and HTTP APIs. Capacity and request limits still apply.

An illustrative architecture

The following three-layer model explains responsibilities. It is not a claim about unpublished Amazon S3 internals:

1. Front-end (API): Authenticates requests and routes traffic.
2. Metadata Service: "Where is my-photo.jpg?"
   • Maps Bucket + Key -> Volume ID + Offset.
   • Needs a highly scalable KV store (e.g., Cassandra, FoundationDB).
3. Storage Nodes: Dumb servers filled with hard drives.
   • Stores "Blobs" (Binary Large Objects).

Data Durability: Erasure Coding

Storing 3 copies of a 1GB file (Replication) uses 3GB. Expensive! Erasure Coding (EC) splits data into $N$ data chunks and $K$ parity chunks.

Example: Reed-Solomon (4, 2)

• Split 1GB file into 4 chunks (250MB each).
• Calculate 2 Parity chunks (250MB each).
• Total storage: $1GB + 0.5GB = 1.5GB$ (1.5x overhead).
• Recovery property: Any two of these six fragments may be unavailable and the object remains reconstructible. Surviving two drive failures also requires placing fragments on distinct failure domains.

[!NOTE] S3 Standard uses EC across at least 3 Availability Zones. It's designed for 99.999999999% (11 9s) durability.

Multipart Upload

How can a large object be uploaded with retryable parts?

1. Initiate: Get an UploadID.
2. Parallel Upload: Choose part sizes that fit the service's object and part-count limits. Upload permitted parts in parallel.
   • PUT /bucket/key?partNumber=1&uploadId=...
   • PUT /bucket/key?partNumber=2&uploadId=...
3. Retry: If Part 5 fails, retry just Part 5.
4. Complete: Send a request to stitch them together.

Consistency Models

S3 Strong Consistency (Since Dec 2020):

• Read-after-Write: If you PUT a new object and immediately GET it, you serve the new version.
• List Consistency: If you PUT and immediately LIST the bucket, the new object appears.

The user-visible consistency contract is documented by AWS; this article does not infer the private implementation. Strong object reads do not imply an atomic transaction across multiple object keys.

Code Example: Interaction

import boto3

s3 = boto3.client('s3')

# 1. Upload logic (simplified multipart)
def upload_large_file(bucket, key, file_path):
    config = boto3.s3.transfer.TransferConfig(
        multipart_threshold=100 * 1024 * 1024, # 100MB
        max_concurrency=10,
        use_threads=True
    )
    s3.upload_file(file_path, bucket, key, Config=config)

# 2. Presigned URLs (Secure sharing)
def generate_share_link(bucket, key, expiration=3600):
    url = s3.generate_presigned_url(
        'get_object',
        Params={'Bucket': bucket, 'Key': key},
        ExpiresIn=expiration
    )
    return url

# 3. Metadata Search (S3 Select / Athena)
# S3 isn't a database, but you can query CSV/JSON contents
def query_csv_content(bucket, key, query):
    resp = s3.select_object_content(
        Bucket=bucket,
        Key=key,
        ExpressionType='SQL',
        Expression=query, # "SELECT * FROM s3object s WHERE s.age > 30"
        InputSerialization={'CSV': {"FileHeaderInfo": "Use"}},
        OutputSerialization={'CSV': {}}
    )
    for event in resp['Payload']:
        if 'Records' in event:
            print(event['Records']['Payload'].decode('utf-8'))

Interview Tips 💡

• "How does S3 achieve 11 9s durability?" — Erasure Coding across multiple Availability Zones.
• "What involves latency in S3?" — First byte latency (TTFB) is higher than disk. Metadata lookup + gathering chunks.
• "Immutable Objects" — General-purpose object updates usually replace the object. Some service/storage-class APIs support additional operations; check the exact bucket and API before treating immutability as universal.
• "CDN Integration" — A CDN can help cacheable reads; measure request patterns, freshness needs, and current transfer charges.

Related Concepts

• Consistent Hashing (Used to place chunks)
• Distributed File Systems (e.g. HDFS)
• CDN Architecture

Source and practical check

Amazon S3 user guide documents the service interface and consistency behavior. For a large upload, test an interrupted part and an abandoned upload; configure cleanup for incomplete uploads. Retrying a part should not require restarting the entire object.