ML Model Serving

The Serving Challenge

Training a model is just the beginning. Serving it to millions of users is an infrastructure challenge.

Key Metrics:

• Throughput: Requests per second (RPS).
• Latency: Time to first token (TTFT) or total generation time.
• Cost: Measure cost per successful request or generated token at the required latency. Provider, region, accelerator, commitment, and utilization affect the result.

1. Serving Patterns

Real-Time (Online) Inference

• Latency target: Set a task-specific percentile target. A fraud score and a long text generation have different response budgets.
• Architecture: REST/gRPC API.
• Example: ChatGPT, Fraud Detection during checkout.
• Challenges: Auto-scaling GPUs based on traffic.

Batch Inference

• SLA: High latency (Hours/Days).
• Architecture: Cron Job / Airflow DAG.
• Example: Nightly product recommendation generation.
• Pros: High throughput, efficient GPU usage (packed batches).

Streaming Inference

• User experience: Deliver incremental output; measure time to first token and time between tokens separately.
• Architecture: Server-Sent Events (SSE) or WebSocket.
• Example: LLM token generation (typing effect).

2. Optimization Techniques

How to make models faster and cheaper?

Dynamic Batching

Process multiple requests in parallel on the GPU.

• Illustrative unbatched case: One request completes in 20 ms, or at most 50 requests/s without overlap.
• Illustrative batch of eight: Eight requests complete together in 25 ms, or 320 requests/s. Each request still waits about 25 ms plus queue time; dividing by eight describes amortized capacity, not request latency.
• Bound queue delay and compare p95/p99 latency at low as well as high traffic.

Tools: NVIDIA Triton Inference Server, TorchServe, vLLM.

Quantization (Shrinking Models)

Lower precision reduces the nominal bytes per stored weight: FP16 uses two bytes, INT8 one, and a packed 4-bit value half a byte. Total memory also includes scales, activations, runtime workspace, and KV cache.

• Speed changes depend on supported hardware kernels, batch size, and whether compute or memory is limiting.
• Validate model quality on representative tasks; lower precision does not guarantee negligible accuracy loss.
• GPTQ and AWQ are examples of low-bit quantization approaches, not universal GPU-count promises.

KV Caching (For LLMs)

Without caching, an autoregressive decoder may recompute keys and values for earlier tokens. KV Cache saves the Key/Value matrices of past tokens in GPU VRAM so we only compute the new token.

• Tradeoff: Increases VRAM usage significantly.

3. Deployment Infrastructure

Model Registry

Version control for weights (model.pt).

• Tools: MLflow, HuggingFace Hub, AWS S3.
• Canary deployment: Send a small traffic fraction to v2 and compare quality, latency, and errors. Blue/green instead maintains two environments and switches routing between them.

Serverless Inference

• CPU serverless functions: Can suit small models within memory, runtime, and package constraints. Check the offered hardware and cold-start behavior rather than assuming GPU support.
• Specialized: Modal, RunPod, Replicate. Spin up containers only when requests arrive.

Code Example: Triton Client

NVIDIA Triton supports configured batching and scheduling. The model configuration must enable compatible batching; replace tensor names and shapes below with those of your deployed model.

import tritonclient.http as httpclient
import numpy as np

# 1. Connect to Inference Server
client = httpclient.InferenceServerClient(url="localhost:8000")

# 2. Prepare Input (Batch Size 1)
input_data = np.array([[0.5, 0.2, 0.1]], dtype=np.float32)
inputs = [
    httpclient.InferInput("input_node", input_data.shape, "FP32")
]
inputs[0].set_data_from_numpy(input_data)

# 3. Request Prediction
results = client.infer(model_name="my_model", inputs=inputs)

# 4. Get Output
output_data = results.as_numpy("output_node")
print(f"Prediction: {output_data}")

Interview Tips 💡

• "CPU vs GPU?" — GPU for deep learning (Matrix Mult parallelization). CPU for decision trees (XGBoost) or very small models.
• "What is Cold Start?" — Loading weights, initializing the runtime, and compiling kernels can delay the first request. Measure this for your model and deployment rather than assuming a fixed duration.
• "Multi-Model Serving" — Loading 10 different models on 1 GPU. If VRAM fills up, swapping to CPU RAM kills performance. Use LoRA adapters to share base weights.

Related Concepts

• Container Architecture
• Load Balancing
• RAG Architecture

Source and validation exercise

NVIDIA Triton batching documentation explains dynamic batching and queue delay. Run the same input-length distribution with batching disabled and enabled. Report throughput alongside queue time, tail latency, accelerator memory, and model quality; an average cost per item cannot establish an interactive latency target.