MIT Technology Review: enterprise AI’s edge depends on infrastructure around the model

MIT Technology Review argues that as AI moves from experimentation into production, the central question is no longer only what a model can do, but whether a company can deliver that capability reliably, affordably and at scale. The publication calls the engineering system around the model a “serving harness,” covering architecture, memory management, scheduling and optimization.

The analysis says the workload should determine the architecture: an assistant requiring fast responses has different needs from a document-processing pipeline that runs overnight. Latency, concurrency, memory pressure, GPU utilization, cost and unpredictable traffic can matter more than small differences in accuracy. The publication recommends measuring the dominant bottleneck before optimizing; for example, the key-value cache can consume available GPU memory as concurrency and context length grow, without adding more compute solving the problem.