Set a latency budget before you pick a model
Most teams choose the most capable model and then discover their latency requirements. That's backwards.
Model selection conversations usually start with capability: which model gives the best outputs for this task? This is the wrong starting point for production systems. Start with the latency budget.
Every user-facing LLM call has an implicit latency requirement determined by the interaction pattern. A chat response can tolerate 2-4 seconds with streaming. An autocomplete suggestion cannot. A batch classification job can tolerate 30 seconds. A retrieval step that feeds another model call needs to be fast enough that the total pipeline stays under the user-facing budget.
Once you know the latency budget, you know the constraint space for model selection. A 500ms budget eliminates most large models regardless of their benchmark performance. A 10-second budget for a background task opens up the full range. Capability evaluation only makes sense within the constraint.
The other thing a latency budget forces is explicit conversation about streaming. If your budget is 3 seconds to first meaningful token and 8 seconds to completion, that's a different system design than 3 seconds to complete response. Settle this before architecture, not after.