Windows Copilot Runtime Isn’t a Runtime, and WPF is Back, Baby (Premium)

During Monday's Microsoft AI event, Pavan Davuluri threw out an unexpected term, Windows Copilot runtime, without really explaining it. Today, we have a bit more information about that, and some news about the new approach to "native" apps on Windows.

"We have infused AI into every layer of Windows, including a fundamental transformation of the OS itself to enable developers to accelerate AI development on Windows," Mr. Davuluri writes in a new Windows Developer Blog post today. "Windows Copilot Runtime has everything you need to build great AI experiences regardless of where you are on your AI journey – whether you are just getting started or already have your own models."

The Windows Copilot Runtime doesn't appear to be a runtime, despite the name (like .NET). Instead, it's better described as a platform that encompasses an API called the Windows Copilot Library, a set of AI frameworks (DirectML, ONNX Runtime, PyTorch, and WebNN), and so-called toolchains like Olive and AI Toolkit for Visual Studio Code.

Given my years-long attempts at understanding the future of Windows app development, I'm most interested in the Windows Copilot Library. This is described as a set of APIs powered by the 40+ on-device AI models that ship with Windows, including the new Phi-Silica model that was created specifically for Copilot+ PCs. (That 40+ number is rather impressive and unexpected.) These APIs work across the Windows shell, Win32 (desktop, but really WPF/NET presumably), and web apps, Microsoft says.

Interesting. But also perhaps not entirely accurate.

During the Build 2024 keynote today, Microsoft CEO Satya Nadella compared it to the Win32 API, and that's technically true in that it's a set of APIs. But without being able to see these APIs now—it's coming in June—I predict that none of this is literally native code, and that these libraries are higher-level abstractions that target .NET and web only.

From a functional perspective, the Windows Copilot Library will initially give developers APIs for Studio Effects, Live Captions Translations, OCR, Recall with User Activity, and Phi Silica. Then, a later release will add APIs for Vector Embeddings, Retrieval Augmented Generation (RAG), Text Summarization, and more.

It's not clear what Microsoft means by this, but the Windows Copilot Library will also give developers "no code effort" access to Studio

Effects like Creative filters, Portrait light, Eye contact teleprompter, Portrait blur, and Voice focus. That doesn't mean "no code," but ... it's unclear. Access to Live Caption's real-time translation capabilities is similarly a "no code effort."

The Windows Copilot Library will also help developers integrate their apps with the Recall feature that's exclusive to Copilot+ PCs. This support includes User Activity API that provides access to the underlying underlying vector database used by Recall. Microsoft apps like Outlook, PowerPoint, and Teams are using this API to provide...