> Most production AI applications aren't running 405B models. They're running 7B-70B models that need low latency and high throughput.

Really? At least for LLMs, most actual usage is concentrated on huge SOTA models. 1 trillion parameters or more. And LLMs seem to be the lion's share of AI compute demand.