Model choice and prompt design shape what each request costs. Infrastructure shapes what you pay whether requests arrive or not. Once teams move beyond hosted APIs to self-hosted models, fine-tuning or batch workloads, GPU capacity becomes one of the largest and least predictable items on the cloud bill.
The problems are familiar from earlier cloud FinOps, but sharper. GPUs are expensive per hour, capacity can be scarce, and idle accelerators cost the same as busy ones.
Plan capacity from workload shape
Start by describing workloads, not instances:
- Interactive inference — latency-sensitive, follows business hours and user traffic
- Batch inference — document processing, embeddings, scoring; flexible on timing
- Training and fine-tuning — bursty, often large, usually schedulable
- Evaluation runs — regular, predictable, often tied to releases
Each has a different demand curve. Capacity planning means estimating peak and baseline for each, then deciding which can share hardware and which can be moved in time.
Build the forecast from product plans: users in scope, expected requests per user, tokens per request and model size. Revisit it monthly. A rollout to a new region or a new use case on the same platform is a step change, not a trend.
Utilisation is the number that matters
A GPU at low utilisation is money spent on waiting. Common causes:
- One model per GPU when several small models could share it
- Instances sized for peak and left running overnight
- Development environments with dedicated accelerators nobody is using
- Poor batching, so requests are served one at a time
Levers that help:
- Request batching and inference servers designed for throughput
- Right-sizing — matching GPU type and memory to model size; the largest accelerator is rarely needed for every model
- Quantisation where evaluation shows quality holds
- Autoscaling on queue depth or latency, with scale-to-zero for development and low-traffic endpoints
- Scheduling batch and training jobs into off-peak windows on the same capacity
Measure utilisation per endpoint and per team. Cluster-level averages hide the idle pockets.
Reserved, on-demand and spot
Cloud providers offer several ways to buy capacity. The right mix depends on how confident your forecast is.
- Reserved or committed capacity suits the steady baseline — interactive inference you know will run all year. Lower unit price, less flexibility.
- On-demand covers growth and peaks above baseline while you learn the real demand curve.
- Spot or preemptible capacity suits interruptible work — batch inference, some training with checkpointing, evaluation runs.
A common pattern is to commit only to the baseline you have observed for a sustained period, run the variable layer on demand, and push interruptible work to spot. Review commitments before renewal with actual utilisation data, not last year’s assumptions.
Capacity availability matters too. For critical workloads, plan for fallback regions or instance types, and confirm data residency rules still hold when you fail over.
Chargeback makes cost visible to owners
Shared GPU clusters blur ownership. Without allocation, every team assumes someone else is responsible for the bill.
Make cost attributable:
- Tag every workload by team, product and environment from the first deploy
- Allocate shared cluster cost by GPU-hours or measured usage, not headcount
- Publish showback reports monthly so teams see their trend
- Move to chargeback once teams trust the numbers and have levers to act on them
Include idle cost explicitly. If a team reserves capacity it does not use, that should appear on its report.
Run it as a routine
GPU FinOps is a recurring meeting, not a project. A monthly review covering forecast versus actual, utilisation by endpoint, commitment coverage and the top three optimisation candidates keeps spend predictable. Pair every cost change with evaluation so savings do not quietly cost quality.
Where to start
- Inventory GPU workloads and classify them by shape
- Tag everything and produce a first showback report
- Find the lowest-utilisation endpoints and fix sizing or scheduling
- Base commitment decisions on observed baseline usage
Incubics builds AI platforms with utilisation telemetry and cost allocation in place from the first release.