We worked on this project from mid-2023, with the main effort through the end of September 2023. NVIDIA GPUs get most of the attention in AI, but our client wanted to bring fast generative AI to Apple’s Mac Studio. In this case study we explain how we built a Metal inference engine that beats Apple’s own library on many operations.
About the customer

HippoML was an innovative startup that changed how AI compute works. They saw an opportunity to bring generative AI to lower hardware costs, and we worked closely together on several projects. NVIDIA has since acquired HippoML, so we can now share a few details about these projects.
This case study describes our first project together: making their inference stack run fast on Apple Silicon. In a later project, we brought FlashAttention to NVIDIA’s Hopper GPUs.
The challenge
Apple Silicon is an attractive target for AI inference. Mac Studio machines with the M2 Ultra chip offer a large unified memory at a fraction of the cost of a high-end GPU system. But the software to actually use that hardware for AI was not in good shape.
Apple ships machine learning kernels in Metal Performance Shaders (MPS) and the Metal Performance Shaders Graph. Quality varied a lot. Some kernels performed well, but many algorithms and configurations silently fell back to the CPU and were very slow. In one case we measured an MPS kernel that ran slower than the CPU baseline it was supposed to beat.
HippoML had already written some Metal kernels of their own. They excelled at the high-level design of their inference engine, but they lacked coverage for many kernel variants, and they needed implementations that make better use of the hardware. They asked us to build both.
Our approach
We started from an initial code drop: Metal shader sources for GEMM, host code in (Objective-)C++, and Python files for interfacing, benchmarks, and tests. We grew this into a well-structured inference engine.
Full operation coverage, tested against PyTorch
We implemented the full set of operations an inference engine needs. Flash Attention (including a specialized version for large head dims and causal mask support), GEMM and batched GEMM, convolution (including Winograd and transposed variants), ArgMax, GroupNorm, Index Add, NonZero, TopK, type casting, and more. Where the client asked for it, we also added residual-operation support to the kernels.
Every kernel is verified. We built a test suite for all kernels using PyTest, with PyTorch as the reference. The client could trust that a fast kernel is also a correct kernel. A benchmark suite compared our performance directly against MPS, also via PyTorch, so improvements were always measured against Apple’s library rather than assumed.
We improved the tooling needed to optimize Metal shaders. That included fixing bugs in an open-source disassembler for Metal shader binaries, and building a tool to dump Metal pipelines so they can be disassembled easily. Good tooling makes optimization work faster, for us and for the client’s own engineers.
Working as one team with the client
HippoML’s requests changed quickly, and we stayed flexible with them. We regularly asked them to share their expected tensor shapes, so we could make sure our implementations covered the real workloads rather than synthetic ones.
We wrote extensive in-code documentation so their engineers could understand both the implementations and the API. We sent weekly status updates to keep everyone aligned. After the project, we never got a question about API usage, nor negative feedback about the project structure.
The result
On the M2 Ultra chip, our kernels substantially outperform MPS. The client shared their own measurements in a blog post, Up to 80X Speedup in Multi-Head Attention on Apple Silicon:
- GEMM: 90% of theoretical peak performance at M=N=K=8192 (row-major layout), reached within the first two weeks of the project. Their own implementation reached about 80%; MPS about 75%.
- Attention: up to 75% of theoretical peak performance, up from 25-35% on MPS depending on parameters.
- Conv2D via implicit GEMM: 70% of theoretical peak performance, up from roughly 40% on MPS. Our Winograd-based Conv2D is slightly faster still.
- ArgMax: at least 2x faster than MPS. For one-dimensional ArgMax we measured 123 GB/s versus 0.17 GB/s on MPS, a 100x+ improvement over both MPS and the PyTorch CPU baseline. For two dimensions we measured 490-525 GB/s versus 90-225 GB/s. Beyond two dimensions we are 2-3x faster.
- GroupNorm: 2x to 13x faster than MPS, up to 10x (230 GB/s versus 17 GB/s) on specific configurations.
We also implemented Attention for large head dims by breaking up the computation across rows. This trades some latency for support of much larger head dims, and uses the M2’s large cache to free up shared memory for intermediate matrices. Final performance is slower than MPS for these cases, but the feature adds architectural support MPS does not have.
Together with the test suite, benchmark suite, CI, and documentation, the client received a maintainable engine, not just a pile of fast kernels.
We also implemented Attention for large head dims by breaking up the computation across rows. This trades some latency for support of much larger head dims, and uses the M2’s large cache to free up shared memory for intermediate matrices. Final performance is slower than MPS for these cases, but the feature adds architectural support MPS does not have.
Together with the test suite, benchmark suite, CI, and documentation, the client received a maintainable engine, not just a pile of fast kernels.
Key takeaways
- Apple’s stock ML libraries are not a solid basis for a serious inference product. Purpose-built Metal kernels deliver large speedups, and correctness comes from systematic testing.
- Asking the client for their real tensor shapes kept our work focused on the configurations that matter.
- Good documentation and weekly updates let the client take over the code without friction.
Did you enjoy reading about how we made machine learning faster on the Mac? You can read about more projects we worked on by checking out our other case studies.