GPU pricing — the metal underneath the tokens

Rent GPUs when utilization is high and predictable. A dedicated H100 at ~$3.50/hour beats per-token APIs the moment you push enough traffic through it — the crossover for a 70B-class model is typically a few hundred million tokens a month. Below that, serverless hosts win on pure cost and win harder on operations: no drivers, no queueing, no 3am pager.

The VRAM column is the decision. Model weights plus KV cache must fit: an 8B model at 4-bit quantization fits in 8–12 GB (a V100 laughs at it), a 70B at FP8 needs ~80 GB plus headroom (H100 territory), and 235B+ MoE models want 140 GB+ or multi-GPU tensor parallelism (H200/B200). When a "cheaper" GPU forces a smaller batch or tighter context, it stops being cheaper.

Spot vs on-demand vs cluster. Marketplaces are cheapest and least reliable; on-demand at Lambda/RunPod is the working default; Lambda's 1-Click Clusters cost ~40–50% more per hour than their on-demand rate and buy you coordinated multi-GPU capacity — worth it only for training runs or serving that truly needs the interconnect.

This page pairs with the LLM API pricing ledger — tokens for the API route, hours for the metal route. The same weekly agent maintains both.