Thurrott.com has seen an internal Microsoft memo that indicates that the software giant is readying a broader campaign to undercut this past week’s news from Consumer Reports. It also provides greater insight into why Microsoft believes the Consumer Reports recommendations are incorrect.
“It’s important for us to always learn more from our customers and how they view their ownership journey with our products,” the memo, from Microsoft corporate vice president Panos Panay reads. “Feedback like this [from Consumer Reports] stings, but pushes us to obsess more about our customers.”
Panay says that Microsoft will continue to “engage” with Consumer Reports and try to both learn from their survey and testing to improve things for customers and “reverse their findings.”
There are a number of interesting aspects to this memo. The first is a mention of “some quality issues” that the firm experienced with “the launch of Surface Book and Surface Pro 4.” I summarized these problems and Microsoft’s painfully slow response in my early 2016 article Welcome to Surfacegate. But it may be useful for everyone to understand how and why this happened.
Multiple senior Microsoft officials told me at the time that the issues were all Intel’s fault, and that the microprocessor giant had delivered its buggiest-ever product in the “Skylake” generation chipsets. Microsoft, first out of the gate with Skylake chips, thus got caught up by this unreliability, leading to a falling out with Intel. Microsoft’s recent ARM push with Windows 10 is a result of that falling out; the software giant believes that Intel needs a counter to its dominance and that, as of late 2016, AMD simply wasn’t up to the task.
Since then, however, another trusted source at Microsoft has provided with a different take on this story. Microsoft, I’m told, fabricated the story about Intel being at fault. The real problem was Surface-specific custom drivers and settings that the Microsoft hardware team cooked up.
The Skylake fiasco came to a head internally when Microsoft CEO Satya Nadella met with Lenovo last year and asked the firm, then the world’s biggest maker of PCs, how it was dealing with the Skylake reliability issues. Lenovo was confused. No one was having any issues, he was told. I assume this led to some interesting conversations between the members of the Microsoft senior leadership team.
But the net result was that Microsoft had to push out some existing designs quickly to get ahead of the reliability issues. The result was Surface Laptop, an uninspiring product whose origin is years old (Julie Larson-Green was still working at that part of Microsoft at the time), and the new Surface Pro, which is obviously just a minor update. More forward-leaning products like a new Surface Hub, code-named Aruba, and a mobile device code-named Andromeda, were pushed back, in the former’s case to 2019.
With that in mind, let’s return to this week’s memo.
In the wake of Surfacegate, Panay says that his team has “worked tirelessly” to improve Surface Book and Surface Pro 4 and to ensure that subsequent product launches—Surface Laptop and Surface Pro (2017), of course, but also Surface Studio and Surface Hub with Performance Base—launched with higher quality. The goal, which makes sense, was to get better each time.
“These improvements were unfortunately not reflected in the results of this [Consumer Reports] survey,” Panay claims, indicating that it was perhaps overly-weighted with angry Surface Book and Pro 4 customers.
He also suggests that what Consumer Reports calls a “failure” is perhaps overly-broad and that some incidents—like a frozen screen or unresponsive touch—are not “failures” but are rather just minor incidents that are easily rectified by the user.
“We take quality seriously,” he says, “conducting rigorous reliability testing during development to forecast failure and return rates, which are then continually viewed against [real world data] post-launch. We also regularly review other metrics to understand the experience we are providing to our customers and our findings show our products are in a much healthier place than noted by Consumer Reports.”
Here, again, things get interesting. Panay is about to offer some actual data to support his claims. And that is:
“Surface NPS [Net Promoter Score] is consistently higher than [that of] the OEMs” (e.g. the other PC makers).
And this is something I actually know a bit about.
Net Promoter Score does not measure reliability. It is, instead, a score, ranging from -100 to +100, that measures whether customers of a given product would recommend that product to others. It measures satisfaction, not reliability. And PC makers, like HP, religiously follow this score and use it to determine how to improve future versions of their products.
So this supports the contention that I made two days ago in Microsoft Mounts Its Defense of Surface Reliability: Customers who spend more on premium products tend to be more satisfied even when they are unreliable because they need to justify their own decision-making process. Whether you believe or not doesn’t matter. Because NPS does not measure reliability, which is what Consumer Reports is estimating. It measures customer satisfaction.
(On a side note, Panay never mentions Apple in this memo. I suspect that Apple’s NPS is higher than that of Surface, but no matter.)
But Panay does offer a bit of relevant data too.
He says that the rate of Surface “incidents per unit” (IPU) is “extremely low,” or less than 1 percent. This is across the Surface product portfolio, and inclusive of customer service calls. And this score has improved “with every device launch.”
He also notes that worldwide return rates “have consistently decreased over the past 12 months. The worst offender in the accompanying chart is Surface Book, which suffered from a 17 percent return rate post-launch, in late 2015. (Surface Pro 4 was roughly the same, at about 16 percent.)
Finally, Panay notes that several blogs have come to Microsoft’s defense despite the fact that there was, at that time, no real evidence suggesting that Consumer Reports wasn’t correct.
“[We] have put together a comprehensive set of data that reflects the strength of our quality and our customer sentiment, and will be working with partner organizations, including marketing, retail, and sales, to share that information broadly,” he writes.
So here are the key takeaways. The facts, if you will, that we can squeeze out of this communication.
One, that the Consumer Reports findings are largely skewed by Surface Book and Surface Pro 4 devices, which Microsoft’s internal data proves suffered from massive reliability issues over a long period of time.
Two, that Microsoft’s internal data shows that it essentially fixed those products.
Three, that subsequent Surface devices launched in far more reliable states. (This may explain why the firm has ignored USB-C/Thunderbolt 3, by the way.)
Four, that Surface customers—whatever the reason—are very happy with their purchases. (With the understanding that there is less correlation, I believe, between product reliability and customer satisfaction with premium products.)
And five, that Microsoft is determined to tell its story and to reassure its customers that it is serious about providing an excellent and high-quality experience to all Surface customers.
Put simply, Microsoft Surface has had some reliability issues. And Microsoft believes it has turned the corner on those issues.