Microsoft details Azure Maia 300 architecture for model inference
🎧 Listen to this story
Microsoft detailed Azure Maia 300, its third generation of accelerators designed for cloud-scale inference, including agentic workloads with multi-step execution, long contexts and tool use. The company says the system combines software, silicon, racks, networking and datacenters co-designed to maximize useful tokens per watt and per dollar.
Maia 300 evolves the Software Defined Locally Accessed Dataflow (SDLA) architecture with NUMA memory distributed across chiplets, lower-precision formats and hardware-accelerated type conversion. Microsoft says each rack now contains 96 accelerators and that the scale-up domain reaches a superpod with 12,288 accelerators. The multi-chiplet architecture, using TSMC N2P for compute, increases FLOPs by 36%, provides 2.5 times more HBM bandwidth and delivers 14% more scale-up bandwidth.
Why it matters · editorial interpretation
For teams serving models with multi-step execution, long contexts, and tool use, the architecture could affect operating cost and efficiency. Microsoft’s claim that it maximizes useful tokens per watt and dollar highlights that inference performance depends not only on the chip, but also on co-designed memory, networking, and software.
Sources
Hardware