Deploying LLMs on a Budget

Not every feature needs a frontier model at temperature 0.7 with a 128k context window. Budget deployments mix model sizes, cache aggressively, and measure cost per successful task—not per token vanity.

Right-size the model

• Classification, routing, extraction → small or mid-tier models.
• Complex reasoning or codegen → larger model, optionally only after a cheap router approves.

Quantization and local inference

Tools like llama.cpp, vLLM, and ONNX runtimes support 4–8 bit weights. Quality loss varies by task—always run your eval set after quantizing.

Cache what repeats

Semantic caches (hash or embed similar queries) and prompt-prefix caching (where providers support it) cut bills for support bots and doc Q&A.

RAG beats giant contexts

Retrieving five good chunks into an 8k window often beats stuffing 100k tokens of noise. See fine-tuning vs RAG.

Observe spend

Log tokens in/out per endpoint, per user, per feature flag. Set budgets and alerts before launch—not after finance asks why the API bill tripled.