We’re All Beta Testers Now (Premium)

For obvious reasons, I’ve been closely tracking how and when Microsoft updates Windows 11. And just last week, I wrote up how the software giant has issued new Windows 11 features every month since last September save one, in January. Explaining how Microsoft transitioned from “one feature update per year” to multiple new features every month is beyond my brain capacity right now, but one of the major changes it instituted to streamline this monthly outlay was formally revealed in late March: it now releases a preview version of each monthly update two weeks prior to the mainstream release.

There’s a lot of language to wade through here, but here’s the short version: Microsoft has long used a week-based system for scheduling software updates. Week B, the second week of a given month, is the week in which Patch Tuesday occurs, and that is when Microsoft issues “mandatory cumulative updates” for Windows and other systems, updates that can include both feature and security updates. Under the new system, Week D, the fourth week of the month, is now when Microsoft ships “optional non-security preview releases.” These are preview versions of the next month’s mandatory cumulative updates, and because they are optional, one needs to manually find and download them.

Keeping track of this stuff is hard enough, but the most recent optional non-security preview release, which Microsoft shipped in Week C last month, on April 25, includes a payload that I don’t feel was adequately explained, most likely on purpose. It arrives in the form of a new toggle in Windows Update called “Get the latest updates as soon as they’re available.” And there are literally three different places to find out more about this: the Microsoft Support document describing this update, a separate Microsoft Support document describing this feature, and a third Microsoft Support document about continuous innovation, Microsoft’s new name for “Windows as a Service.”

And yikes.

Among the many ways in which Microsoft can ship software code to its Windows 11-using customers is a technology called Controlled Feature Rollout (CFR). This technology is already being used in the Microsoft Edge Insider Program and the Windows Insider Program, and now it’s moving into what I’ll call “stable,” meaning the non-preview, supported version(s) of Windows 11 that are out in the world and used by hundreds of millions of mainstream users.

CFR is designed to roll out new features to the user base gradually, and only after each feature has “completed standard quality, reliability, and compliance tests and [Microsoft has] determined that the feature is at ship quality.” When that’s true, it will be rolled out to a “small randomly selected subset” of users. As feedback and telemetry come in and hopefully validates Microsoft’s belief that the feature in question is of high enough quality, the size of the subset of users who receive the feature grows, is further validated, and then grows again and again until everyone has it.

This sounds like a reasonable, quality-driven system on the surface. But it’s not, and it makes no sense for Windows 11 stable. After all, Microsoft already has a system for testing the quality of updates across an ever-growing set of users before pushing it to stable. It’s called the Windows Insider Program, and Microsoft just this past March expanded the program yet again by adding a new channel and “rebooting” another, a set of changes that I feel don’t make any sense at all.

The problem, of course, is that Microsoft doesn’t use the Windows Insider Program correctly. Instead of pushing new features logically through the various Insider Preview channels—starting with Canary or Dev and then moving through Beta and Release Preview before moving into stable—Microsoft has arbitrarily introduced new features throughout the Insider Preview tiers. And worse, it has introduced new features directly into stable without any testing. That’s what the “search pill” from last November/December was: a buggy new feature that had to be replaced months later because it was never tested properly.

About that search pill. It’s now clear that the search pill was an early example of a CFR. As I noted at the time, and this was important because I write a book documenting how to use Windows 11, the new search pill did not show up on all of my PCs that first month. Instead, it arrived in what appeared to be a random fashion and, over time, it slowly made its way to most and then all of them. It is very clear that this was a CFR. And that experience highlights the problem with this system: you don’t start testing new features in stable. You start testing them in Canary or Dev, move them into Beta and then Release Preview, and then you move them into stable. And if you want to use CFR to roll out those features then, God love you. The more testing the better.

As hinted at above, CFR is the second recent example of Microsoft screwing with Windows 11 stable. The other is that change to the monthly release schedule so that it includes a Week D preview update, a system Microsoft had been quietly employing since the previous September. Microsoft is using its mainstream Windows 11 users, hundreds of millions of people, as guinea pigs.

Why?

We must speculate because Microsoft will never admit to this. But there’s only one obvious conclusion: the Windows Insider Program is not active and engaged enough to give Microsoft the level and quality of feedback that it needs, and so it has turned to stable—again, with its hundreds of millions of users, now unsuspecting beta testers—to achieve test feature updates. (This is exactly what it’s doing with AI, too, with arguably the same level of risk, and few people using either system realize that they’re testing software for Microsoft for free.)

And that’s a tough one because the problems with the Insider Program are of Microsoft’s making. I mentioned A/B testing above, and that’s a good example: when someone joins the Windows Insider Program, they evaluate the available channels, determine which makes the most sense for them, and then they enroll one of their PCs in the program and install the software and test. The problem is that Microsoft changed the rules over time, and so 50 percent of the people who signed up for the Beta channel and expected to test whatever features aren’t getting those features to test. Disillusionment set in, engagement fell, and the Insider Program became less valuable. That Microsoft stranded many users’ PCs in some dead-end channel with no way out for several months didn’t help either. That was just one of many issues with the program.

But that’s how things go with this rendition of the Windows team. Continuous innovation is just a cute marketing term for a regular carpet bombing of new features, many of which were never tested before they reached our PCs. But if you look past all the invented new terms, what you see is the opposite of computer science. It’s starting to feel like we’re just a sociological experiment and that the real goal is to get us all to get us to give up and go use some other platform. It really seems that antagonistic and user-hostile.