Skip to main content

Windows 11 will soon harness your GPU for generative AI

Following the introduction of Copilot, its latest smart assistant for Windows 11, Microsoft is yet again advancing the integration of generative AI with Windows. At the ongoing Ignite 2023 developer conference in Seattle, the company announced a partnership with Nvidia on TensorRT-LLM that promises to elevate user experiences on Windows desktops and laptops with RTX GPUs.

The new release is set to introduce support for new large language models, making demanding AI workloads more accessible. Particularly noteworthy is its compatibility with OpenAI’s Chat API, which enables local execution (rather than the cloud) on PCs and workstations with RTX GPUs starting at 8GB of VRAM.

Recommended Videos

Nvidia’s TensorRT-LLM library was released just last month and is said to help improve the performance of large language models (LLMs) using the Tensor Cores on RTX graphics cards. It provides developers with a Python API to define LLMs and build TensorRT engines faster without deep knowledge of C++ or CUDA.

With the release of TensorRT-LLM v0.6.0, navigating the complexities of custom generative AI projects will be simplified thanks to the introduction of AI Workbench. This is a unified toolkit facilitating the quick creation, testing, and customization of pretrained generative AI models and LLMs. The platform is also expected to enable developers to streamline collaboration and deployment, ensuring efficient and scalable model development.

A graph showing TensorRT-LLM inference performance on Windows 11.
Nvidia

Recognizing the importance of supporting AI developers, Nvidia and Microsoft are also releasing DirectML enhancements. These optimizations accelerate foundational AI models like Llama 2 and Stable Diffusion, providing developers with increased options for cross-vendor deployment and setting new standards for performance.

The new TensorRT-LLM library update also promises a substantial improvement in inference performance, with speeds up to five times faster. This update also expands support for additional popular LLMs, including Mistral 7B and Nemotron-3 8B, and extends the capabilities of fast and accurate local LLMs to a broader range of portable Windows devices.

The integration of TensorRT-LLM for Windows with OpenAI’s Chat API through a new wrapper will allow hundreds of AI-powered projects and applications to run locally on RTX-equipped PCs. This will potentially eliminate the need to rely on cloud services and ensure the security of private and proprietary data on Windows 11 PCs.

The future of AI on Windows 11 PCs still has a long way to go. With AI models becoming increasingly available and developers continuing to innovate, harnessing the power of Nvidia’s RTX GPUs could be a game-changer. However, it is too early to say whether this will be the final piece of the puzzle that Microsoft desperately needs to fully unlock the capabilities of AI on Windows PCs.

Kunal Khullar
Kunal Khullar is a computing writer at Digital Trends who contributes to various topics, including CPUs, GPUs, monitors, and…
AI could soon have a mind of its own. I spoke to experts to learn more about AGI
Clear Mannequin on Dark Blue Background.

Artificial intelligence (AI) may soon have a mind of its own, and many companies want to make that happen as soon as possible. Whether this is plausible remains to be seen; however, if achieved, we could move from the AI age to the AGI age in record time. 

The AI explosion of recent years may seem sudden to many, but the industry has been in constant development for several decades. As technology goes on, the evolution of AI has been rapid, and many in the industry are already looking toward the next big thing. That thing is Artificial General Intelligence (AGI), which currently remains a theoretical concept, but many believe will be the next wave in training AI to be autonomously intelligent. 

Read more
Harnessing AI: make Bitrix24’s your sales and marketing MVP
people in a meeting discussing

You’re about halfway through Q2 and your campaigns aren’t landing. Your team is tired. You’re staring down an end-of-quarter push with CRM fields still half-filled, a pile of call recordings no one wanted to transcribe, and one shared doc titled “Q2 Ideas (Pls Delete?).” It’s not that you’re not trying, you’re just tapped.

Enter CoPilot. No fanfare, no flashy onboarding webinars. It’s just there one morning, a new button inside Bitrix24. And somehow, it feels like the only teammate who hasn’t taken a vacation in the past year.
The unexpected power of AI that doesn’t shout “AI”
There’s a lot of hype in the sales and marketing tech world, AI this…neural that…but CoPilot doesn’t posture. It integrates quietly into the Bitrix24 ecosystem: CRM, chat, tasks, feeds, even site-building. It doesn’t try to reinvent your process, it shows up ready to assist with the one you already have.

Read more
Windows 11’s controversial AI Recall feature is coming to your Copilot+ PC very soon
The Surface Pro 11 on a white table in front of a window.

As AI strides on, it inevitably finds its way onto our personal devices, with tech giants announcing new features that rely on accessing our private information and media to serve us better. While some might find this useful, others are bound to find it creepy, and one such feature is Microsoft's controversial AI Recall, which takes screenshots of everything you do on a Copilot+ PC so it's easier to trace back your steps and find something specific later. After being announced last year, and then witnessing a few delays, Recall is finally rolling out to a broader group of Windows 11.

Microsoft recently announced Recall is coming to Windows 11 with the latest Release channel update with build 26100.3902 (KB5055627). The feature's availability in the Windows 11 Release Preview channel, which succeeds the Beta channel in the Windows Insider program, means it is in the initial phases of being available to a wider audience of folks who own Copilot+ PC. This category of PCs currently includes a whole wide range of laptops with specialized hardware in the form of a neural processing unit (NPU) dedicatedly for running AI tasks, though we might see desktops joining the club soon.

Read more