▲ SYSTEM
Inference infrastructure, training systems, distributed computing, and the software stack above hardware.
The KV Cache: A Definitive Engineering Analysis
Naive autoregressive inference recomputes the entire attention matrix at every decoding step. KV caching stores the immutable K and V projections instead, converting per-step O(t·d) cost to O(d) exactly — no approximation.
PagedAttention: How vLLM Borrowed Virtual Memory to Unlock GPU Serving at Scale
Static KV cache pre-allocation wastes 60–80% of GPU memory under production traffic patterns. PagedAttention applies OS-style virtual memory paging to the KV cache, enabling 3× higher concurrency on the same hardware.
Continuous Batching: The Scheduling Insight That Made LLM Serving Actually Work
Static batching blocks an entire batch until its longest sequence finishes. Moving the scheduling boundary from request to iteration frees GPU slots at every decode step, taking utilization from 20–35% to 70–85%.
The Missing Decision Guide: Parallelism Strategies in LLM Training and Inference
NVLink delivers 9–18× more bandwidth than InfiniBand NDR. Every parallelism decision is a negotiation with that ratio — this is the decision tree from hardware topology and model architecture to a concrete TP/PP/SP/DP configuration.
NCCL Collective Operations: The Architecture of Distributed Gradient Truth
AllReduce, ReduceScatter, AllGather, and Broadcast are the atomic grammar of distributed training, with derivable cost models. AllReduce decomposes exactly into ReduceScatter + AllGather — the identity that powers ZeRO and FSDP.
The Brutal Economics of LLM Inference
TTFT, TPOT, throughput, and cost per million tokens are the four numbers that determine whether your LLM infrastructure makes money. Prefill is compute-bound; decode is memory-bandwidth-bound by a factor of 295.
Writing Custom GPU Kernels in Triton: A Hands-On Guide for ML Engineers
CUDA forces simultaneous management of threads, warps, shared memory, and register pressure. Triton lets you write per-block logic in Python; the compiler handles vectorization, coalescing, and warp scheduling. Fused softmax end-to-end.
ZeRO vs FSDP: A Rigorous Dissection for Distributed Training Engineers
AdamW training consumes 16 bytes per parameter — 12 of them optimizer states. ZeRO and FSDP shard that triad differently: this is the quantitative account of what each does to your memory, communication bus, and iteration time.
Intra-node vs Inter-node Interconnects in Distributed Training
NVLink, NVSwitch, InfiniBand, and RoCE — the bandwidth and latency numbers that determine whether your distributed training job scales or stalls.
