AMD to Acquire Taalas for Model Specific AI Inference Chips

Today, AMD announced that it will acquire Taalas for a different kind of AI inference chip. Instead of having a highly programmable chip that can run many models and then using the programmability to change models, Taalas is the opposite.

This is one chip, or more likely, one set of chips, per model. If you want to change the model being run in a rack, you also have to change the chips, since these are model-specific AI inference chips. AMD to Acquire Taalas for Model-Specific AI Inference Chips The basic concept behind Taalas is that instead of loading almost all model weights from memory like HBM, and then using the programmable portions of chips to handle a model’s specific matrix and compute needs, it just burns the model into CMOS. The benefit of this is that performance gains can be enormous compared to other solutions designed to be more programmable. Taalas current-generation hardware is the HC1 technology demonstrator. The company showed the HC1 running Llama 3.1 8B and claims up to 17,000 tokens per second per user. Taalas lists the chip as TSMC 6nm with an 815 square millimeter die and 53 billion transistors. Taalas compares the HC1 against Nvidia H200 and B200, as well as Groq, SambaNova, and Cerebras, though those figures are Taalas’ own measurements. Realistically, however, that means many larger modern models will require multiple reticle-size chips just to hold an entire model, especially a large one. Imagine having to fabricate ten, dozens, or even more different types of chips, package them, put them onto cards, test everything along the way, and then get software to tie all of the parts together. What if you need 10 different chips, and there is an issue with the 10th when it comes back, leaving you with 9 batches of chips waiting for that last one? While that sounds scary, this is a super-fast and allegedly lower-cost way to do inference. Taalas says that changing weights, matrix dimensions, and other important bits only requires changing two mask layers. That means even if it needed many different chips per model, the number of expensive masks that would need to be produced is much lower than what would be used if you had to tape out two different GPUs, for example. A model-specific chip trades flexibility for efficiency.