Windows 11, RTX Spark, and Local AI: It’s Not Me, It’s Windows ⭐

I keep running into problems as I explore using local AI on RTX Spark, and I finally figured out what's wrong. It's Windows.

Remind me. What's the point of Windows again?

There's a lot that goes into that, of course, from providing a consistent graphical interface for all apps, inter-app content sharing capabilities, and global hardware abstraction with drivers, and there's a shared file system with whatever permissions. And when you download or copy a file to a Windows PC, that file is available to the system and all apps and services. It's not sandboxed or isolated in any way. If you download a DOCX file, for example, it's not only available to Microsoft Word, it's just a file in the file system.

But that's not how local AI models work. And I have no idea why.

Like DLLs, DOCX files, and drivers, local AI models are simply files and they do sit in the file system somewhere. Microsoft claims there are over 40 local AI models on every Copilot+ PC, but we're rarely confronted by what this means. It's a black box of sorts. The models are hidden somewhere and they're updated by Windows Update or the Microsoft Store, depending on the model. And whatever apps and services Microsoft makes available on these PCs know how to find those models and use them.

Where those models are in the file system shouldn't normally matter to the user, though I'm sure I could find the location(s) with a little research. What's important is that the software that needs to use the local AI models can find them. And it's logical to think that there might be a formal storage location for these models, or some system service that acts as a middleman between the models and the software that needs them.

There is not.

What this means in practice is that you can install your AI app of choice--Hermes, LM Studio, Clairvoyance, Windows Llama, or whatever else--use that app's AI model browsing functionality to find and then download a model, and then use it with that app. But if you install and use a different AI app, it cannot see the model you already downloaded with the first app. So if you want to use the same model, you will download it twice. Which can quickly become problematic as the sizes of these local models balloon, especially with RTX Spark computers that can have up to 128 GB of RAM and allocate most of that to the GPU that interacts with those models.

Think about this for a second. Especially right now, when this world is evolving quickly, you will want to experiment with different AI apps, different AI runtimes, and, most especially, different models. You will move back and forth between these things as you experiment, downloading and re-downloading AI apps and models as you go. And it won't take too long before the digital detritus of this activity starts taking over the PC's storage.

And so you may want to clean house. It would be easier if you knew where the models were located. And though it's possible (likely?) that uninstalling an...