Layer 5 — AI Infrastructure · Chapter 6
Performance, Cost, and Scaling
AI systems consume expensive computation unevenly. A short classification request and a long reasoning task may use the same model family but require radically different time and resources. Infrastructure turns these variable workloads into predictable service levels and sustainable economics.
The main performance measures
| Measure | Meaning | Why it matters |
|---|---|---|
| Time to first output | How long the user waits before the first token, frame, or partial result. | Shapes the feeling of responsiveness. |
| End-to-end latency | Total time until the task is complete. | Determines whether the workflow is practical. |
| Throughput | Amount of work completed per unit of time. | Affects capacity and cost at scale. |
| Utilization | How fully expensive compute is kept busy. | Low utilization wastes capital or cloud spending. |
| Availability | How consistently the service can accept and complete work. | Critical when AI becomes part of business operations. |
| Quality per dollar | Useful output achieved for the total cost. | Prevents optimization of speed or price at the expense of value. |
Why batching matters
Accelerators are more efficient when they process several compatible requests together. Batching improves throughput and utilization, but waiting to form a batch can increase latency. Infrastructure must balance individual responsiveness against total efficiency.
Caching
Repeated work can sometimes be reused. Infrastructure may cache retrieved documents, prompt prefixes, model states, embeddings, tool results, or final outputs. Caching is powerful when requests share common material, but it requires careful invalidation, privacy isolation, and version awareness.
Model and numerical optimization
- Quantization reduces numerical precision to save memory and computation.
- Pruning removes parts of a model judged less important.
- Distillation trains a smaller model to imitate a larger one.
- Parallelism divides model layers, parameters, or requests across devices.
- Speculative generation uses a faster model to propose work that a stronger model verifies.
- Sparsity and expert routing activate only part of a model for a particular input.
Scaling is not only adding machines
More hardware can increase capacity, but bottlenecks may move to memory, network communication, storage, retrieval, rate limits, or external tools. Effective scaling requires measuring the complete request path and designing for the dominant constraint.
The cost stack
The visible model fee may be only one part of total cost. A complete calculation can include:
- model inference or accelerator time;
- data storage, indexing, and transfer;
- retrieval, reranking, and embedding computation;
- tool and third-party API charges;
- logging, evaluation, monitoring, and security;
- human review and exception handling;
- engineering and operational labor;
- idle capacity reserved for peak demand.
Optimization should follow business value
The cheapest model is not economical if it creates more errors, human review, customer loss, or legal exposure. The strongest model is not economical if a smaller one can handle routine requests. Mature platforms use tiered models, confidence thresholds, escalation, and task-specific routing.
The objective is not minimum cost per token. It is minimum cost per acceptable outcome.