Course

AI Inference Infrastructure: serving a model under real constraints

Name: AI Inference Infrastructure: serving a model under real constraints
Price: 1799 BDT

A generative model in production is a capacity problem before it is anything else. Budget the memory a model and its KV cache actually occupy, batch requests without starving any of them, and scale against demand rather than pretending accelerators are infinite.

38 lessons4h 1mEnglish

What you'll learn

Budget AI inference memory and capacity without pretending infrastructure is infinite
Separate the memory a model occupies from the memory its KV cache grows into as a conversation lengthens
Explain what continuous batching improves, and what it costs the request that was already waiting
Separate prompt prefill from autoregressive decode and choose a scheduling policy for the workload
Use prefix caching without confusing it with response caching or crossing a tenant trust boundary
Choose replicas or tensor parallelism by model fit, independent capacity, communication cost and failure scope
Evaluate model routing against explicit quality constraints and optimize cost per useful response
Define inference reliability in user terms and choose safe degradation under overload or model failure

Requirements

Cloud Infrastructure Foundations, or equivalent understanding of instances, scaling and failure domains
No machine learning background. This is about serving a model, not building one

Course content

38 lessons · 4h 1m

Serve AI inference

Fit model weights and active context into accelerator memory, then engineer batching, parallelism, queues and replicas against latency and cost.

0/7

Memory before speed

Budget model weights, KV cache and runtime workspace before estimating throughput.

Preview8m 2.

Context is live serving state

Relate active requests and sequence length to KV-cache memory.

Preview8m

Batching fills the GPU

Raise aggregate throughput while exposing queueing and time-to-first-token costs.

Replicate or shard

Use independent replicas for capacity and tensor parallelism when one model copy cannot fit.

Scale the queue, not the allocation

Prefer demand and latency signals over permanently allocated GPU memory.

Business case: inference economics

Balance quality, latency, throughput, reliability and accelerator cost per useful response.

Chapter check: AI inference

Check memory, batching, replicas, sharding and autoscaling signals.

Prefill, decode and scheduling

Trace one request through prompt prefill and autoregressive decode, then choose scheduling and admission policies for interactive and batch workloads.

0/5

One request, two phases

Separate prompt processing from token-by-token generation.

Preview8m 2.

First token is not the whole answer

Distinguish time to first token, inter-token latency and completion time.

Preview8m

Schedulers make tradeoffs

Prioritize prefill or decode without pretending every latency metric improves.

Admission and workload classes

Protect interactive traffic from long prompts, long answers and unbounded concurrency.

Chapter check: request scheduling

Check phase, latency and scheduling reasoning.

Prefix caching and isolation

Reuse computed KV state for repeated prompt prefixes without confusing semantic similarity with cache identity or crossing customer trust boundaries.

0/5

Repeat the prefix, not the work

Reuse warm KV blocks for an identical leading token sequence.

Preview8m 2.

Cache identity is exact

Key reuse by exact token history and execution identity rather than semantic similarity.

Preview7m

The cache does not speed decode

Apply prefix caching to prefill while preserving decode and output costs.

A cache is a trust boundary

Use tenant salts or namespaces to prevent cross-customer timing disclosure.

Chapter check: prefix caching

Check reuse, identity, latency and isolation reasoning.

Multi-GPU serving

Choose independent replicas or a cooperating shard group, then account for collective communication, interconnect topology and failure scope.

0/5

Four GPUs, two meanings

Distinguish four serving copies from four ranks serving one copy.

Preview8m 2.

Fit the large model

Use tensor parallelism when one complete weight set cannot fit on one GPU.

Preview8m

Every token needs the group

Trace collective exchange through every sharded token step.

Crossing the node boundary

Relate interconnect cost and rank failure to non-linear scaling.

Chapter check: multi-GPU serving

Check placement, collectives, scaling and failure reasoning.

Model routing economics

Route requests by workload need, enforce a quality gate, and measure cost per useful response instead of celebrating cheap failures.

0/5

One endpoint, many models

See a routing policy distribute one workload across two model tiers.

Preview8m 2.

A cheap request can be an expensive failure

Apply a quality gate before comparing inference cost.

Preview8m

Evaluate the workload, not the anecdote

Change the traffic mix and observe why routing results do not transfer automatically.

Constraints before optimization

Treat quality, capability and policy as gates rather than weighted decorations.

Chapter check: model routing

Check routing, workload evaluation and outcome-cost reasoning.

Reliability & graceful degradation

Define success in user terms, then choose retries, fallback or load shedding without hiding unsafe outputs.

0/5

A response is not success

Separate transport availability from timely, useful output.

Preview8m 2.

Retries can amplify failure

See why retrying an overloaded or unavailable primary can worsen the incident.

Preview8m

Fallback has a safety boundary

Relate fallback quality to the consequence of a wrong response.

Shed load to protect useful work

Refuse excess work before queues destroy every request deadline.

Chapter check: inference reliability

Check SLIs, retries, fallback and load shedding.

About the creator

Classytic