Skip to main content

Nvidia and Microsoft are solving a big problem with Copilot+

The Surface Laptop running local AI models.
Luke Larsen / Digital Trends

When Microsoft announced Copilot+ PCs a few weeks back, one question reigned supreme: Why can’t I just run these AI applications on my GPU? At Computex 2024, Nvidia finally provided an answer.

Nvidia and Microsoft are working together on an Application Programming Interface (API) that will allow developers to run their AI-accelerated apps on RTX graphics cards. This includes the various Small Language Models (SLMs) that are part of the Copilot runtime, which are used as the basis for features like Recall and Live Captions.

With the toolkit, developers can allow apps to run locally on your GPU instead of the NPU. This opens up the door to not only more powerful AI applications, as the AI capabilities of GPUs are generally higher than NPUs, but also the ability to run on PCs that don’t currently fall under the Copilot+ umbrella.

Get your weekly teardown of the tech behind PC gaming
Check your inbox!

It’s a great move. Copilot+ PCs currently require a Neural Processing Unit (NPU) that’s capable of at least 40 Tera Operations Per Second (TOPS). At the moment, only the Snapdragon X Elite satisfies that criteria. Despite that, GPUs have much higher AI processing capabilities, with even low-end models reaching to 100 TOPS, and higher-end options scaling even higher.

In addition to running on the GPU, the new API adds retrieval-augmented generation (RAG) capabilities to the Copilot runtime. RAG gives the AI model access to specific information locally, allowing it to provide more helpful solutions. We saw RAG on full display with Nvidia’s Chat with RTX earlier this year.

Performance comparison with the RTX AI toolkit.

Outside of the API, Nvidia announced the RTX AI Toolkit at Computex. This developer suite, arriving in June, combines various tools and SDKs that allow developers to tune AI models for specific applications. Nvidia says that by using the RTX AI Toolkit, developers can make models four times faster and three times smaller compared to using open-source solutions.

We’re seeing a wave of tools that enable developers to build specific AI applications for end users. Some of that is already showing up in Copilot+ PCs, but I suspect we’ll see far more AI applications at this point next year. We have the hardware to run these apps, after all; now we just need the software.

Editors' Recommendations

Jacob Roach
Lead Reporter, PC Hardware
Jacob Roach is the lead reporter for PC hardware at Digital Trends. In addition to covering the latest PC components, from…
Microsoft is warping the PC industry into something unrecognizable
The Surface Laptop shown in front of a Copilot+ sign.

Microsoft has often taken a backseat in driving the direction of the PC industry. But that time is over.

Over the past couple of weeks, we've had a front row seat in seeing just how much power Microsoft has within the PC industry. Of course, developing the primary operating system used across millions (billions?) of devices gives you a lot of say, but the introduction of Copilot+ and the hardware announcements that followed show what can happen when Microsoft flexes its muscle.

Read more
The Intel we know is dead, but its new Lunar Lake chips are very much alive
An Intel executive holding a Lunar Lake CPU.

It’s not an exaggeration to call Lunar Lake radical. The upcoming chips don't showcase the Intel we’ve seen so many times before -- the one that pushes power, core counts, and clock speeds to their limit and wonders why it can’t manage a full day’s charge in a laptop. Lunar Lake is Intel admitting defeat, but not in a way that puts it down for the count. It’s clear that Lunar Lake, announced at Computex 2024, is a fresh start.

Intel has called it a “radical low-power architecture” previously, and that statement holds true. Everything in Lunar Lake is built around power efficiency. And with that new focus, Intel is shedding all of the tentpoles of its previous generations that don’t contribute -- even down to needing to manufacture the chip in its own fabs.
A new focus

Read more
Qualcomm is invading PCs, and says it’s ‘not going anywhere’
Qualcomm's CEO presenting Snapdragon X Elite CPUs at Computex 2024.

Qualcomm's Computex 2024 keynote was relatively uneventful. The company just went on a tear as Microsoft introduced Copilot+ PCs, so the hourlong presentation was a victory lap with executives from every major laptop brand. One note stood out, though -- Qualcomm's CEO Cristiano Amon made a commitment to PCs as a platform.

"We're in this neighborhood to stay," Amon said. "We're not going anywhere." The context here is that Amon announced that Snapdragon X chips will eventually come in all PC form factors. In the presentation, we saw 2-in-1s, laptops, mini PCs, and even desktops. One day, we might even see a handheld gaming PC powered by a Snapdragon CPU.

Read more