Programming Windows: The Windows NT Death March (Premium)

In March 1991, the NT team hit a new milestone: Dog food came to the NT build lab. So not only were NT team members and a select small group elsewhere in Microsoft running on NT, the build lab was too. Microsoft was using NT to build NT.

Elsewhere, confusion reigned. NT architect Dave Cutler wanted to ship a lean, mean version of the product first and add new features later. But Bob Muglia wanted to NT to change the industry, to include new features big and small that would make the product better. Cutler referred to Muglia’s feature requests as “an endless list of shit.”

But some of Muglia’s features were great ideas. Key among them was a way to map the DOS 8.3 file name convention with the 255-character file names that were allowed by OS/2 and NTFS. (Anyone who’s used Windows 95 will remember this mapping, where a file name like Paul’s document.doc would appear in DOS as PAUL’S~1.DOC.)

It was the right thing to do, but this addition delayed the availability of NTFS, to July 1991, triggering another round of schedule rethinking. And some were pushing to drop features like NTFS in order to meet the then-current NT schedule, in which the product was to be code complete by Halloween. Surprisingly, Cutler intervened: They would make the new file system and it would ship in the first version of NT.

Cutler’s concerns about feature creep later took a humorous turn when Microsoft president Steve Ballmer said publicly that NT would feature something called “fault tolerance,” which was news to the NT team. He has misspoken, but two NT team members were inspired to make it happen.

Throughout 1991, the team’s dogfooding expanded. First to graphics, which replaced a “Lite” command-line-only version of NT, and to networking, in which the team finally abandoned the outdated LAN Manager software that had originated with OS/2. Instead, NT would run on both PCs and servers, with the latter providing shared access to files and applications. It was a great goal, but it made dogfooding difficult, because when a server crashed, it affected everyone on the team. The first networking-capable build of NT arrived in mid-August.

By that point, the NT code base was about 80 percent complete and it seemed like hitting code complete by Halloween was possible. But Paul Maritz, who had previously led Microsoft’s OS/2 efforts, identified a key missing piece: Security. He gathered the top leaders on the NT team and asked whether it was possible for him to store a spreadsheet in NT that only Bill Gates could access. The answer was no, so Maritz sent them back to the drawing board. NT needed to have the pervasive security features that business customers demanded.

Maritz had created a nightmare scenario for the NT team. But he also gave them more time, understanding that there was no way a more secure version of NT could be completed by the end of October. As a result, the NT team lost several more months, but it also created a system of trusted domains and pass-through authentication that would help form NT’s security reputation. NT was now scheduled for code completion in April 1992.

“Our track record for producing robust and reliable builds on time and without a lot of babysitting is not good,” Cutler wrote in an internal memo to the team. “There is very little margin for error.”

In mid-September, Cutler finalized an NT demo that would be shown at Comdex the next month. It was to be the first public showing of the product. And it was at this time that he used the word “showstoppers” to describe bugs that were serious enough to “stop the show,” or hold back the release of a build. The term had come from elsewhere within Microsoft, and it was the inspiration for the title of G. Pascal Zachary’s excellent book of the same name. It took Cutler 10 days to fix all the showstoppers in the build that Microsoft would show at Comdex, just four days later, on October 21.

NT was huge. The first version eventually consisted of over 6 million lines of code—“no one mind can comprehend it all,” Cutler said—and made a great case for using a CD-ROM instead of floppies for the setup media. The problem was that CD-ROMs were in short supply back in late 1992, so Microsoft had to arrange with Sony to ensure that they would have enough to bring the unfinished NT to Comdex.

It was worth the effort: Microsoft’s Comdex demo of NT was a smash success. Reviewers were overwhelmingly impressed, with PC Magazine declaring NT “nothing less than the modern reinvention of the operating system.” But the publication also called bullshit on Microsoft’s claims that NT would run MS-DOS applications. “Backwards [sic] compatibility is the first thing to go in the hectic final phase of development,” it claimed.

The NT team took that as a challenge. But Cutler had other issues: The Comdex demo ran only on the Intel 80386 chipset, undercutting his portability goals. Cutler called himself “Mr. Mips” and personally championed the version of NT that would run on that RISC chipset. He was afraid that an emphasis on the 386 would skew the amount of platform-dependent code in NT upwards from the 15 percent it currently represented.

“Screw Mips,” Bob Muglia said in response to this concern.

But as 1991 came to a close, Cutler engaged in his annual tradition of skiing at Whistler mountain in British Columbia, a location whose name would years later be chosen as the codename for the release of Windows NT that would finally replace DOS-based Windows. He was celebrating an important milestone: A unified Mips/Intel version of NT had shipped two days before Christmas. It was, Cutler said, “an important accomplishment.” And a nice retort to Muglia.

As 1992 began, Cutler felt that the NT team was hitting the “final run” of development. The product would be code complete in February, ready for application developers and customers in beta form in April, and could ship by the end of June.

It wasn’t to be. A number of factors—the key one being Windows application compatibility—delayed NT yet again. More features were added. The schedule slipped. And slipped. Finally, Cutler recognized the inevitable: NT wouldn’t be ready until the second half of 1992. They had entered into a death march.

In April, Microsoft invited application developers to a three-day developer conference, the NT PDC, at which it would help them “plot the next wave of application development for Windows.” It was set for early July and would feature both Bill Gates and NT architect Dave Cutler. So, Cutler cracked the whip, hoping to get NT ready for a beta release in mid-summer.

He was right to worry: The NT PDC wasn’t just sold out, it was oversold. Showgoers paid $795 to attend the San Francisco event, and they were expecting to walk away with a copy of NT. Those who couldn’t attend could order NT on disc for $69, and Microsoft quickly racked up tens of thousands of takers. The code needed to be ready for the July start of the show.

By May, things were starting to come together. The NTFS file system, which was buggy and slower than the MS-DOS and OS/2 file systems, was one of the final wrinkles, and it was still problematic heading into June, when the NT beta code had to be finalized. Lou Perazolli finally told the folks working on NFTS that if their code wasn’t satisfactory by July 1, NTFS would simply be cut from the product.

It never really came together. By June 15, the team had reached the “final countdown” on the release for the NT PDC. And then Cutler chose the June 29 build as the version that would be handed out to external developers. But this version would include a warning about NTFS and tell users to segregate their data from NTFS just in case. It was a demoralizing blow to the team.

That wasn’t the only issue with the beta release: Only half of the 70 most popular Windows and DOS programs ran successfully on NT at that time.

It didn’t matter: Over 4800 cheering developers attended the NT PDC, three times the amount expected. It was the largest audience ever for a Microsoft developer event.

But the accolades calmed down when the showgoers got home and actually started using the code. The consensus? “Too big, too slow,” the exact opposite of Cutler’s software development goals. “It wasn’t ready,” Paul Maritz said at the time. PC Magazine referred to NT as “a product only a developer could love.”

When Cutler had signed off on the NT beta, he had decreed that the final release date remained “the end of the year.” But the response to the beta indicated that more time was needed. So Maritz pushed the release back, yet again, to April 1993.

Cutler wasn’t entirely on board with this decision. He felt that NT was already where he wanted it, and he didn’t think a few more months would make a difference. All new operating systems performed slowly, he claimed. It would get better over time, with new hardware and with further refinements.

“We should have started our performance work sooner,” David Thompson said of the delay. The problem wasn’t that NT required twice as much memory and power as Windows on DOS. It required four times as much. Where most computers of the day shipped with 4 MB of RAM, NT really needed 16 MB. And RAM was expensive. At that time, a 16 MB RAM upgrade cost as much as the rest of the computer combined.

The push continued. Graphics guru Michael Abrash, who would later join Id Software to get the ambitious game Quake off the ground, joined Microsoft to fix NT’s graphics and dramatically speed performance. By October, Microsoft was finally ready to deliver the first true beta for Windows NT that was aimed at business customers and not just developers.

It was a big improvement over the PDC beta. This version was reliable and included a working version of NTFS. Performance, while not perfect, was much improved. Microsoft took in feedback, logged 2000 bugs, and began careening towards a second beta, then scheduled for January 1993. Further performance improvements ensued, both in graphics and in NTFS. Regarding the graphics, one NT team member told Zachary, “a miracle happened: Mike Abrash.”

On December 16, 1992, the NT leadership met again with Bill Gates for a progress report. Gates’ goal was simple: He wanted to “ship this thing.” He was willing to concede on NT’s 16 MB of RAM requirement—he had wanted it to work well with 8 MB of RAM—as long as the performance was satisfactory. Application speed had dramatically improved over the past several months. But no Windows applications ran as fast on NT as they did on MS-DOS.

Gates was apoplectic but Abrash saved the day:  He had “very good answers” to Gates’ questions and handled the co-founder’s anger with a poise that surprised the others. Gates trusted Abrash and gave him a list of things to fix. “Go ahead and change the code,” he told Abrash.

Gates was also upset about the speed of NTFS. But Cutler told him the team would get it there.  He agreed to another month of performance tuning.

It worked. When Gates met again with the team in early February 1993, NT had “turned the corner.” Windows applications still ran slower than on DOS, but only a little bit slower. And NTFS, finally, had overcome its performance problems. Gates blessed the shipment of NT. “There was no reason to hold this thing up,” he said.

It should have been a cause for celebration. But the team simply created a schedule—formally called the Windows NT 1.0 Ship Plan—for getting NT out the door. It called for a Beta 2 release on February 28, release to manufacturing on May 10, and a ship date of May 10, 1993.  Suddenly, it seemed like Windows NT was actually going to happen.

The NT team did hit the Beta 2 milestone on schedule. But after that release, the number of serious bugs started rising, not falling. So, the final release was pushed back for the umpteenth time so that they could focus on the bugs. By May 9, 1993, there was only one serious bug left in the queue. But when it was fixed, another one appeared. The cycle just kept repeating. Once again, development had turned into a death march.

Finally, on June 9, the NT team did it: There were no serious bugs in Windows NT, a state called “zero bugs.” Microsoft shipped the build as a third beta to customers, who noted the much-improved Windows application compatibility. On June 21, Cutler declared that the team had entered “the beginning of the final countdown.” He scheduled July 14 as the date for the “golden master” at which time NT would be written to disc and shipped off for manufacturing.

On July 1, the team issued yet another beta to just 50 customers, which it called a “release candidate” for obvious reasons. A second release candidate, for about 80 customers, followed a week later. Then, on July 15, 1993, Cutler declared that NT was in “escrow,” meaning that testing would continue but NT would not be further updated. “The only thing that is a showstopper is something that would cause us to pull back the final bits from manufacturing,” he told the team.

The next day, the team created the 509th build of Windows NT overall, and the 170th of 1993. “We have the final bits for release,” Cutler said.

(Fun aside: It is fascinating how many common terms were created or at least formalized during this time, from showstopper to release candidate to escrow to final bits. These terms have survived for about 30 years and will likely continue well into the future.)

During a July 23 meeting at which Cutler expected to sign-off on NT, a final bug came in: An Aldus programmer had found an obscure bug that occurred when Pagemaker for Windows printed a document of a specific length that mixed both text and graphics. The bug didn’t seem serious to anyone on the NT team, but several openly wondered what it meant that this had come in so late in the schedule: Were there other bugs like this out there?

They decided to hold off, and Microsoft engaged with Aldus to try and figure out the bug. After a lot of back and forth, they discovered that there were two bugs, one in Pagemaker and one in NT. It took just 10 minutes to fix the bug in NT. It was, as one NT team member described it, a “happy ending.”

With this out of the way, Cutler signed off on Windows NT 1.0—which was called version 3.1 to align it with the DOS-based Windows—on 2:30 pm PT on Monday, July 26, 1993.

“NT is officially released to manufacturing!” Cutler wrote in an unusually emotive email to the team. “Let me state that again—NT is released!!!!! This has been a long hard effort. You have all done a great job. Thank you all for your contribution, especially in the final three months where we managed to fix upwards of 200 bugs per day without serious regressions. It took an unparalleled effort on everyone’s part to accomplish this. We met or exceeded all our ship criteria!”

Windows NT was finally ready for the world.