This morning is an important one for AMD – perhaps the most important of the year. After almost a year and a half of build-up, and even longer for actual development, AMD is launching their next generation GPU/APU/AI accelerator family, the Instinct MI300 series. Based on AMD’s new CDNA 3 architecture, and combining it with AMD’s proven Zen 4 cores, AMD will be making a full-court press for the high-end GPU and accelerator market with their new product, aiming to lead in both big-metal HPC as well as the burgeoning market for generative AI training and inference.
Taking the stage for AMD’s launch event will be AMD CEO Dr. LIsa Su, as well as a numerous AMD executives and ecosystem partners, to detail, at last, AMD’s latest generation GPU architecture, and the many forms it will come in. With both the MI300X accelerator and MI300A APU, AMD is aiming to cover most of the accelerator market, whether clients just need a powerful GPU or a tightly-coupled GPU/CPU pairing.
The stakes for today’s announcement are significant. The market for generative AI is all but hardware constrained at the moment, much to the benefit of (and profits for) AMD’s rival NVIDIA. So AMD is hoping to capitalize on this moment to cut off a piece – perhaps a very big piece – of the market for generative AI accelerators. AMD has made breaking into the server space their highest priority over the last half-decade, and now, they believe, is their time to take a big piece of the server GPU market.
12:56PM EST – We’re here in San Jose for AMD’s final and most important launch event of the year: Advancing AI
12:57PM EST – Today AMD is making the eagerly anticipated launch of their next-generation MI300 series of accelerators
12:58PM EST – Including MI300A, their first chiplet-based server APU, and MI300X, their stab at the most powerful GPU/accelerator possible for the AI market
12:59PM EST – I’d say the event is being held in AMD’s backyard, but since AMD sold their campus here in the bay area several years ago, this is more like NVIDIA’s backyard. Which is fitting, given that AMD is looking to capture a piece of the highly profitable Generative AI market from NVIDIA
12:59PM EST – We’re supposed to start at 10am local time here – so in another minute or so
12:59PM EST – And hey, here we go. Right on time
01:00PM EST – Starting with an opening trailer
01:00PM EST – (And joining me on this morning’s live blog is the always-awesome Gavin Bonshor)
01:00PM EST – Advancing AI… together
01:01PM EST – And here’s AMD’s CEO, Dr. Lisa Su
01:01PM EST – Today “is all about AI”
01:01PM EST – And Lisa is diving right in
01:02PM EST – It’s only been just a bit over a year since ChatGPT was launched. And it’s turned the computing industry on its head rather quickly
01:02PM EST – AMD views AI as the single most transformative technology in the last 50 years
01:02PM EST – And with a rather quick adoption rate, despite being at the very beginning of the AI era
01:02PM EST – Lisa’s listing off some of the use cases for AI
01:03PM EST – And the key to it? Generative AI. Which requires significant investments in infrastructure
01:03PM EST – (Which NVIDIA has captured the lion’s share of thus far)
01:03PM EST – In 2023 AMD projected the CAGR for the AI market would be $350B by 2027
01:04PM EST – Now they think it’s going to be $400B+ by 2027
01:04PM EST – A greater than 70% compound annual growth rate
01:04PM EST – AMD’s AI strategy is centered around 3 big strategic priorities
01:05PM EST – A broad hardware portfolio, an open and proven software ecosystem, and partnerships to co-innovate with
01:05PM EST – (AMD has historically struggled with software in particular)
01:05PM EST – Now to products, starting with the cloud
01:06PM EST – Generative AI requires tens of thousands of accelerators at the high-end
01:06PM EST – The more compute, the better the model, the faster the answers
01:06PM EST – Launching today: AMD Instinct MI300X accelerator
01:06PM EST – “Highest performance accelerator in the world for generative AI”
01:07PM EST – CDNA 3 comes wiht a new compute engine, sparsity support, industry-leading memory bandwidth and capacity, etc
01:07PM EST – 3.4x more perf for BF16, 6.8x INT8 perf, 1.6x memory bandwidth
01:07PM EST – 153B transistors for MI300X
01:08PM EST – A dozen 5nm/6nm chiplets
01:08PM EST – 4 I/O Dies in the base layer
01:08PM EST – 256MB AMD Infinity Cache, Infinity Fabric Support, etc
01:08PM EST – 8 XCD compute dies stacked on top
01:08PM EST – 304 CDNA 3 compute units
01:08PM EST – Wired to the IODs via TSVs
01:09PM EST – And 8 stacks of HBM3 attached to the IODs, for 192GB of memory, 5.3 TB/second of bandwidth
01:09PM EST – And immediately jumping to the H100 comparisons
01:10PM EST – AMD has the advantage in memory capacity and bandwidth due to having more HBM stacks. And they think that’s going to help carry them to victory over H100
01:10PM EST – AMD finds they have the performance advantage in FlashAttention-2 and Llama 2 70B. At the kernel level in TFLOPS
01:11PM EST – And how does MI300X scale?
01:11PM EST – Comparing a single 8 accelerator server
01:12PM EST – Bloom 176B (throughput) and Llama 2 70B (latency) inference performance.
01:12PM EST – And now AMD’s first guest of many, Microsoft
01:13PM EST – MS CTO, Kevin Scott
01:14PM EST – Lisa is asking Kevin for his thoughts on where the industry is on this AI journey
01:15PM EST – Microsoft and AMD have been building the foundation for several years here
01:16PM EST – And MS will be offering MI300X Azure instances
01:16PM EST – MI300X VMs are available in preview today
01:17PM EST – (So MS apparently already has a meaningful quanity of the accelerators)
01:17PM EST – And that’s MS. Back to Lisa
01:17PM EST – Now talking about the Instinct platform
01:18PM EST – Which is based on an OCP (OAM) hardware design
01:18PM EST – (No fancy name for the platform, unlike HGX)
01:18PM EST – So here’s a whole 8-way MI300X board
01:18PM EST – Can be dropped into almost any OCP-compliant design
01:19PM EST – Making it easy to install MI300X
01:19PM EST – And making a point that AMD supports all of the same I/O and networking capabilities of the competition (but with better GPUs and memory, of course)
01:20PM EST – Customers are trying to maximize not just space, but capital expedetures and operational expedetures as well
01:20PM EST – On the OpEx side, more memory means being able to run either more models or bigger models
01:21PM EST – Which saves on CapEx expenses by buying fewer hardware units overall
01:21PM EST – And now for the next partner, Oracle. Karan Batta, the SVP of Oracle Cloud Infrastructure
01:22PM EST – Oracle is one of AMD’s major cloud computing customers
01:23PM EST – Oracle will be supporting MI300X as part of their bare metal compute offerings
01:23PM EST – And MI300X in a generative AI service that is in the works
01:24PM EST – Now on stage: AMD President Victor Peng to talk about software progress
01:25PM EST – AMD’s software stack is traditionally been their achilles heel, despite efforts to improve it. Peng’s big project has been to finally get things in order
01:25PM EST – Including building a unified AI software stack
01:25PM EST – Today’s focus is on ROCm, AMD’s GPU software stack
01:26PM EST – AMD has firmly attached their horse to open source, which they consider a huge benefit
01:26PM EST – Improving ROCm support for Radeon GPUs continues
01:26PM EST – ROMc 6 shipping later this month
01:27PM EST – It’s been optimized for generative AI, for MI300 and other hardware
01:27PM EST – “ROCm 6 delivers a quantum leap in performance and capability”
01:28PM EST – Software perf optimization example with LLMs
01:28PM EST – 2.6x from optimized libraries, 1.4x from HIP Graph, etc
01:28PM EST – This, combined with hardware changes, is how AMD is delivering 8x more GenAI perf on MI300X versus MI250 (with ROCm 5)
01:29PM EST – Recapping recent acquisitions as well, such as the nod AI compiler
01:30PM EST – And on the ecosystem level, AMD has an increasing number of partners
01:30PM EST – Hugging Face arguably being the most important, with 62K+ models up and running on AMD hardware
01:31PM EST – AMD GPUs will be supported in the OpenAI Triton 3.0 release
01:32PM EST – Now for more guests: Databricks, Essential AI, and Lamini
01:33PM EST – The four of them are having a short chat about the AI world and their experience with AMD
01:34PM EST – Talking about the development of major tools such as vLLM
01:34PM EST – Cost is a huge driver
01:36PM EST – It was very easy to incluide ROCm in Databricks’ stack
01:36PM EST – Meanwhile Essential AI is taking a full stack approach
01:37PM EST – The ease of use of AMD’s software was “very pleasant”
01:38PM EST – And finally, Lamini’s CEO, who has a PhD in Generative AI
01:39PM EST – Customers get to fully own their models
01:39PM EST – Imbuing LLNs with real knowledge
01:39PM EST – Had an AMD cloud in production for over the past year on MI210s/MI250s
01:40PM EST – Lamini has reached software parity with CUDA
01:41PM EST – Many of the genAI tools available today are open source
01:41PM EST – Many of them can run on ROCm today
01:43PM EST – AMD’s Instinct products are critical to supporting the future of business software
01:46PM EST – And that’s the mini-roundtable
01:47PM EST – Summing up the last 6 months of work on software
01:47PM EST – ROCm 6 shipping soon
01:47PM EST – 62K models running today, and more coming soon
01:48PM EST – And that’s a wrap for Victor Peng. Back to Lisa Su
01:49PM EST – And now for another guest spot: Meta
01:49PM EST – Ajit Mathews, Sr. Director of Engineering at Meta AI
01:50PM EST – Meta opened access to the Llama 2 model family in July
01:50PM EST – “An open approach leads to better and safer technology in the long-run”
01:51PM EST – Meta has been working with EPYC CPUs since 2019. And recently deployed Genoa at scale
01:51PM EST – But that partnership is much broader than CPUs
01:52PM EST – Been using the Instinct since 2020
01:53PM EST – And Meta is quite excited about MI300
01:53PM EST – Expanding their partnership to include Instinct in Facebook’s datacenters
01:53PM EST – MI300X is one of their fastest design-to-deploy projects
01:54PM EST – And Meta is pleased with the optimizations done for ROCm
01:55PM EST – (All of these guests are here for a reason: AMD wants to demonstate that their platform is ready. That customers are using it today and are having success with it)
01:55PM EST – Now another guest: Dell
01:56PM EST – Arthur Lewer, President of Core Business Operations for the Global Infrastrucutre Solutions Group
01:56PM EST – (Buying NVIDIA is the safe bet; AMD wants to demonstrate that buying AMD isn’t an unsafe bet)
01:57PM EST – Customers need a better solution than today’s ecosystem
01:58PM EST – Dell is announcing an update to the Poweredge 9680 servers. Now offering them with MI300X accelerators
01:58PM EST – Up to 8 accelerators in a box
01:58PM EST – Helping customers consolidate LLM training to fewer boxes
01:59PM EST – Ready to quote and taking orders today
02:01PM EST – And that’s Dell
02:02PM EST – And here’s another guest: Supermicro (we’ve now pivoted from cloud to enterprise)
02:02PM EST – Charles Liang, Founder, President, and CEO of Supermicro
02:03PM EST – Supermicro is a very important AMD server partner
02:05PM EST – What does Supermicro have planned for MI300X?
02:05PM EST – 8U air cooled system, and 4U system with liquid cooling
02:05PM EST – Up to 100kW racks of the latter
02:05PM EST – And that’s Supermicro
02:06PM EST – And another guest: Lenovo
02:06PM EST – Kirk Skaugen, President of Lenovo’s Infrastructure Solutions Group
02:07PM EST – Lenovo believes that genAI will be a hybrid approach
02:07PM EST – And AI will be needed at the edge
02:08PM EST – 70 AI-ready server and infrastructure products
02:09PM EST – Lenovo also has an AI innovators program for key verticals for simplifying things for customers
02:10PM EST – Lenovo thinks inference will be the dominate AI workload. Training only needs to happen once; inference happens all the time
02:11PM EST – Lenovo is bring MI300X to their ThinkSystem platform
02:11PM EST – And available as a service
02:12PM EST – And that’s Lenovo
02:13PM EST – And that’s still just the tip of the iceberg for the number of partners AMD has lined up for Mi300X
02:13PM EST – And now back to AMD with Forrest Norrod to talk about networking
02:14PM EST – The compute required to train the most advanced models has increased by leaps and bounds over the last decade
02:14PM EST – Leading AI clusters are tens-of-thousands of GPUs, and that will only increase
02:14PM EST – So AMD has worked to scale things up on multiple fronts
02:14PM EST – Internally with Infinity Fabric
02:15PM EST – Near-linear scaling performance as you increase the number of GPUs
02:15PM EST – AMD is extending access to Infinity Fabric to innovators and strategic partners across the industry
02:15PM EST – We’ll hear more about this initiative next year
02:16PM EST – Meanwhile the back-end network connecting the servers together is just as critical
02:16PM EST – And AMD believes that network needs to be open
02:17PM EST – And AMD is backing Ethernet (as opposed to InfiniBand)
02:17PM EST – And Ethernet is open
02:18PM EST – Now coming to the stage are a few netowrking leaders, including Arista, Broadcom, and Cisco
02:19PM EST – Having a panel discussion on Ethernet
02:21PM EST – What are the advantages of Ethernet for AI?
02:22PM EST – Majority of hyperscalers are using Ethernet or have a high desire to
02:23PM EST – The NIC is critical. People want choices
02:24PM EST – “We need to continue to innovate”
02:24PM EST – AI networks need to be open standards based. Customers need choices
02:25PM EST – Ultra Ethernet is a critical next step
02:26PM EST – https://www.anandtech.com/show/18965/ultra-ethernet-consortium-to-adapt-ethernet-for-ai-and-hpc-needs
02:28PM EST – UEC is solving a very important technical problem of modern RDMA at scale
02:28PM EST – And that’s the networking panel
02:28PM EST – Now on to high-performance computing (HPC)
02:29PM EST – Recapping AMD’s experience thus far, including the most recent MI250X
02:29PM EST – MI250X + EPYC had a coherent memory space, but still the GPU and CPU separated by a somewhat slow link
02:29PM EST – But now MI300A is here with a unified memory system
02:29PM EST – Volume production began earlier this quarter
02:30PM EST – MI300 architecture, but with 3 Zen 4 CCDs layered on top of some of the IODs
02:31PM EST – 128GB of HBM3 memory, 4 IODs, 6 XCDs, 3 CCDs
02:31PM EST – And truly unified memory, as both GPU and CPU tiles go through the shared IODs
02:32PM EST – Performance comparisons with H100
02:32PM EST – 1.8x the FP64 and FP32 (vector?) performance
02:33PM EST – 4x performnace on OpenFOAM with MI300A versus H100
02:33PM EST – Most of the improvement comes from unified memory, avoiding having to copy around memory before it can be used
02:34PM EST – 2x the perf-per-watt than Grace Hopper (unclear by what metric)
02:35PM EST – MI300A will be in the El Capitan supercomputer. Over 2 EFLOPS of FP64 compute
02:35PM EST – Now rolling a video from HPE and the Lawrence Livermore National Lab
02:35PM EST – “El Capitan will be the most capable AI machine”
02:36PM EST – El Capitan will be 16x faster than LLNL’s current supercomputer
02:37PM EST – And now another guest on stage: HPE
02:37PM EST – Trish Damkroger, SVP and Chief Product Officer
02:38PM EST – Frontier was great. El Capitan will be even better
02:39PM EST – AMD and HPE power a large number of the most power efficient supercomputers
02:40PM EST – (Poor Forrest is a bit tongue tied)
02:40PM EST – ElCap will have MI300A nodes with SlingShot fabric
02:41PM EST – One of the most capable AI systems in the world
02:41PM EST – Supercomputing is the foundation needed to run AI
02:42PM EST – And that’s HPE
02:43PM EST – MI300A: A new level of high-performance leadership
02:43PM EST – MI300A systems avaialble soon from partners around the world
02:43PM EST – (So it sounds like MI300A is trailing MI300X by a bit)
02:43PM EST – Now back to Lisa
02:44PM EST – To cap off the day: Advancing AI PCs
02:44PM EST – AMD started including NPUs this year with the Ryzen Mobile 7000 series. The first x86 company to do so
02:44PM EST – Using AMD’s XDNA architecture
02:45PM EST – A large computing array that is extremely performant and efficient
02:45PM EST – Shipped millions of NPU-enabled PCs this year
02:46PM EST – Showing off some of the software applications out there that offer AI acceleration
02:46PM EST – Adobe, Windows studio effects, etc
02:46PM EST – Announcing Ryzen AI 1.0 software for developers
02:46PM EST – So AMD’s software SDK is finally available
02:47PM EST – Deploy tained and quantized models using ONNX
02:47PM EST – Announcing Ryzen Mobile 8040 series processors
02:47PM EST – Hawk Point
02:47PM EST – This is (still) the Phoenix die
02:48PM EST – With one wrinkle: faster AI performance thanks to a higher clocked NPU
02:48PM EST – AMD’s own perf benchmarks show 1.4x over 7040 series
02:48PM EST – Now time for another guest: Microsoft
02:49PM EST – Pavan Davuluri, CVP for Windows and Devices
02:49PM EST – Talking about the work AMD and MS are doing together for client AI
02:50PM EST – Microsoft’s marquee project is Copilot
02:52PM EST – MS wants to be able to load-shift between the cloud and the client. Seamless computing between the two
02:52PM EST – Showing AMD’s NPU roadmap
02:53PM EST – Next-gen Strix Point processors in the works. Using a new NPU based on XDNA 2
02:53PM EST – Launching in 2024
02:53PM EST – XDNA 2 designed for “leadership” AI performance
02:53PM EST – AMD has silicon. So does MS
02:54PM EST – More than 3x the genAI perf (versus Hawk Point?)
02:55PM EST – And that’s AI on the PC
02:55PM EST – Now recapping today’s announcements
02:55PM EST – MI300X, shipping today. MI300A, in volume production
02:55PM EST – Ryzen Mobile 8040 Series, shipping now
02:56PM EST – “Today is an incredibly proud moment for AMD”
02:57PM EST – And that’s it for Lisa, and for today’s presentation
02:58PM EST – Thanks for joining us, and be sure to check out our expanded coverage of AMD’s announcements
02:58PM EST – https://www.anandtech.com/show/21177/amd-unveils-ryzen-8040-mobile-series-apus-hawk-point-with-zen-4-and-ryzen-ai
02:58PM EST – https://www.anandtech.com/show/21178/amd-widens-availability-of-ryzen-ai-software-for-developers-xdna-2-coming-with-strix-point-in-2024