RTX Spark: Real-World Local AI ⭐

The appeal of local AI is obvious enough. Local AI models are free and don't incur any costs, usage-based or otherwise. They work offline. They're private, as your data is never shared with Big Tech, Big AI, or their partners. And they can be incredibly efficient, though the most efficient models that work with your computer's NPU are less common than those that rely on power-hungry GPUs like those found in RTX Spark computers.

But local AI is still more promise than reality because the models are much smaller and much less capable than cloud-based frontier models. AI evolves quickly, of course, and local AI benefits from this rapid progress too. But it's important to correctly frame the relative capabilities. That is, while local AI will never "close the gap" with cloud-based AI, it will be good enough depending on the use case.

This is the point behind hybrid AI, or what Microsoft calls Hybrid Intelligence: Use local AI as much as possible and only hand off tasks to cloud-based AI when required. In time, we hope, local AI will be good enough for enough tasks that cloud AI is rarely if ever needed.

It's a nice dream. But that's all it is today if you use AI to get real work done.

For one thing, hybrid AI requires a capable orchestrator, and we don't even have basic orchestrators for the most part right now. Microsoft is working to bring hybrid AI orchestration to tools like the GitHub Copilot app and even Copilot in Windows 11. And other Big Tech and Big AI companies--from Google to Anthropic and OpenAI and others--will no doubt roll out hybrid AI orchestration capabilities into their AI front-ends too. But that's not available today, certainly not broadly or in any mainstream AI assistant.

And that means we're stuck with a choice. We can use cloud AI, as we have been for the past three and a half years through whatever AI assistant or other AI front-end, incurring usage-based costs as we go and with the need to be online for the duration. Or we can suck it up and experiment with local AI. Which requires a lot of know-how, the right hardware, and unfamiliar tools.
Copilot+ PC got it right
It shouldn't be that way. If you're familiar with Copilot+ PCs, which I think of as the second-generation AI PCs, you may recall Microsoft claiming two and a half years ago that these new PCs shipped with over 40 so-called small language models (SLMs), which are smaller local AI models that, in this case, are each optimized for specific use cases. And if you use a Copilot+ PC, you might know that these models all need to be updated just like any other software, and that some need to be downloaded before specific features in individual apps will even work. For example, before you can use the Copilot+ PC-specific features in Microsoft Paint, you're prompted to download and install an Image Generation model from the Microsoft Store.

Say what you will about Copilot+ PCs, but this platform's local AI features are aimed at mainstream users...