Video Processing: Resumable Uploads and Atomic Publication

Video processing: publish only a playable result

This exercise designs an upload-and-playback service. It does not describe YouTube's internal processing system. The main question is what happens when upload, encoding, or publication fails halfway through.

Establish the processing budget

Assume 100 uploads per day, each 20 minutes long, with four output renditions. That is 2,000 source minutes and 8,000 rendition-minutes daily. If a worker processes one rendition at 0.5 times real-time, the encoding load is 16,000 worker-minutes, or about 267 worker-hours per day.

Over a 24-hour period, the theoretical average is about 11.1 continuously busy workers. Over an eight-hour processing window it is 33.3. These are arithmetic consequences of a hypothetical measured speed, not encoder benchmarks. Add capacity for bursts, retries, I/O, source complexity, and other tasks. A queue can defer work; it cannot create processing capacity.

Finish the upload before starting the pipeline

Create an authenticated upload session with an owner, object key, size limit, expiry, and integrity requirements. The client uploads parts directly to object storage using bounded permissions. Persist completed part information so a disconnect does not force a complete restart.

Multipart upload separates initiation, part uploads, and completion. Unfinished uploads also need cleanup, since abandoned parts can consume storage. Provider rules and checksums belong in the implementation, not assumptions about what an ETag means. Amazon S3 multipart upload documentation.

A large HTTP request is not inherently impossible, but a resumable protocol limits the cost of interruptions. Trigger processing only after validated completion. A notification that a part arrived is not evidence that the full source exists.

Model tasks and dependencies explicitly

A practical pipeline is:

validated upload
  → inspect source
  → encode selected renditions
  → verify audio/video and segment alignment
  → assemble versioned manifests
  → publish playable version

Thumbnails, captions, and content review may run in parallel, but publication requirements must say which outputs are mandatory. An automatic content check is a decision input, not a universal guarantee of safety or rights clearance.

Give each task an identity such as (video_id, source_version, rendition, encoder_config_version). Retries write to the same temporary destination or to a uniquely tracked attempt. A uniqueness rule selects the accepted result. Persist task state and dependencies so a coordinator restart does not lose the workflow.

Split where decoding dependencies permit

Parallelizing a long video can reduce completion time, but arbitrary byte or timestamp cuts can break reference-frame dependencies. Workers need decodable inputs, and output renditions need aligned switching points. Choose closed GOPs or otherwise handle cross-boundary references deliberately. Audio timing and encoder delay also require validation.

Do not equate a ten-minute processing chunk with a playback segment. A worker can process a large section and produce many short media segments. Chunk size trades scheduling overhead against parallelism and wasted work on a retry.

Publish the manifest last

Suppose the 360p rendition finishes but the 1080p worker fails after writing half its segments. If the published manifest already references every intended output, players request nonexistent objects.

Instead, validate completed artifacts under an immutable versioned prefix. Publish a manifest containing only verified playable renditions, then atomically update the video's active version. The product can choose “play lower quality now” or “wait for the required quality.” Keep old versions until active clients have had time to finish or refresh.

HLS uses playlists to describe media and supports multiple container arrangements, including fragmented MP4. Four-second .ts files are one example, not a universal rule. A player selects variants based on its capabilities and observed conditions, and buffering remains possible. HLS, RFC 8216.

Storage and delivery have different lifetimes

Keep source media accessible while retries and expected re-encodes are likely. Moving it to an archival tier can save storage expense but may introduce retrieval delay and cost. Choose the tier from the recovery-time requirement rather than assuming all originals belong in cold storage.

Serve immutable output segments through a CDN with suitable authorization and cache keys. Long-tail videos may have low reuse; popular videos may share many cached segments. Neither implies a fixed 99% hit rate. The streaming design exercise shows how byte hit ratio affects origin bandwidth.

A failure drill should interrupt an upload, terminate an encoder, duplicate a task event, and fail publication after artifacts are ready. Verify that incomplete versions remain inaccessible, retrying does not create conflicting outputs, and a published manifest references readable objects. Track upload completion, time to first playable rendition, retry work, failed validations, and queue age to locate the actual bottleneck.