Programming Windows: War Room (Premium)

Tensions ran high in January 2003 as the Windows team built Windows .NET Server 2003 each day, moving inexorably towards the shipping release.

Note: Much of this article was originally published in 2003, and it was based on unprecedented access to the teams responsible for building and shipping the product that became Windows Server 2003. However, I’ve edited it here to include information not found in the original article and to provide a conclusion to the story.

Developing Windows

One element about the NT family of operating systems—which evolved from Windows NT to Windows 2000, XP, and, now, Windows Server 2003—that has remained unchanged over the years, though the details have changed dramatically, is the build process. Somewhere deep in the bowels of Microsoft, virtually every day, at least one Windows product is compiled, or built, into executable code that can be tested internally by the dev teams. For Windows Server 2003, this process is consummated in Building 26 on Microsoft’s Redmond campus, where banks of PCs and CD duplicating machines churn almost constantly under the watchful eyes of several engineers.

The details of NT—excuse me, Windows—development have changed dramatically since the project first started in the late 1980s. “Back in the early days, we started with 6 people,” Microsoft Distinguished Engineer and Windows Server Architect Mark Lucovsky told me. “Now there are 5000 members of the Windows team, plus an additional 5000 contributing partners, generating over 50 million lines of code for Windows Server 2003. Getting all those people going in the same direction, cranking out code, is an enormous task. Building the results of their work, compiling and linking it into the executable and other components that make up a Windows CD is a 12- to 13-hour process that is done every day of the week. It’s an enormous task, the biggest software engineering task ever attempted. There are no other software projects like this.” And Microsoft compiles the whole thing—all 50+ million lines of code—almost every single day, he said. “We’re evolving the development environment all the time,” Lucovsky noted.

“When we turn the crank, we compile the whole thing,” he said. “We have to be able to reproduce the system at any point in time as well. So developers check-in code, we press a button, and out comes a system. We should be able to reproduce that [build] three years in the future, using the various tools, compilers, and scripts we used at that time.”

David Thompson, corporate vice president of the Windows Server Product Group at Microsoft, elaborated on the process. “The key here is that we built up the system over the years, advancing it in three dimensions,” he said. “First is the product itself. Second is the way we engineer the product. And third is the way we interact with a broader set of customers. The product evolution is pretty straightforward. The source code control system we use now is new because we really pushed the scale of the previous version with Windows 2000. Mark [Lucovsky] personally worked with a new system and introduced it post-2000. We bought some outside technology. We now do have a staged build [for the first time]. But every day the [staged builds] are rolled up into the total build. So we can scale but maintain, and we know where we are every day.”

Just eat it: Microsoft serves up dog food

Lucovsky reminisced a bit about the early days, when the first NT prototypes were built in his office with only a single person overseeing the process. That person would simply send out an email to the NT team when a new build was ready, and then 50 people or so would “eat their own dog food,” testing the build on their own systems and run stress tests. “I used to just walk around the building and write down the problems we found,” Lucovsky said. “That’s how it was [before Windows] NT 3.51. Now we have 16 build labs. Dave [Thompson] has his own [build lab] for the 1500 people he oversees. The main build lab cranks out the official build, which goes out to thousands of people daily. Notification is automatic, and it is sent out in multiple stages using the backbone servers across the campus. It’s all automated. Those little things have now scaled up.”

“We had a certain time of day [up to which time] we could check code in and then we stopped,” Thompson said. “After that, we threw the switch. Eventually, we grew the team to 85 people and serialized the process. [NT architect] Dave Cutler—who we all worked for—ran the build lab for about a week, and he required people to personally [make check-in requests]. He forced it into a mold. I sat in there for a while too. One day I accepted 85 check-ins, the most we had ever had to that point. Now we take in over 1000 every day. It’s a completely different scale. Even the whiteboard is electronic—web-based, actually—now.”

“There are no other software projects like this,” Lucovsky said, “but the one thing that’s remained constant [over the years] is how long it takes to build [Windows]. No matter which generation of the product, it takes 12 hours to compile and link the system.” Even with the increase in processing horsepower over the years, Windows has grown to match, and the development process has become far more sophisticated, so Microsoft does more code analysis as part of the daily build. “The CPUs in the build lab are pegged constantly for 12 hours,” he said. “We’ve adapted the process since Windows 2000. Now, we decompose the source [code] tree into independent source trees, and we use a new build environment. It’s a multi-machine environment that lets us turn the crank faster. But because of all the new code analysis, it still takes 12 hours.”

Dogfooding their code has always been a key requirement of the NT team, Thompson told me, and an integral component of Microsoft’s culture. “This is one of the things we’ve always done, back to the earliest days,” he said. “We were just joking about this today, actually. Back when we first got NT running on desktop [PCs], our email program wouldn’t run because it was a DOS application, and we didn’t have DOS compatibility mode working yet. So I ported our internal email app, WizMail, to Win32 so we would be able to use only NT systems.”

“When you are forced to use the system yourself, you see bugs and you see the performance issues,” Thompson added. “And in the case of NT, you’d go and find the person responsible for the problem and ask them to fix it.” One of Thompson’s primary responsibilities when he joined the NT team was to port the file system over to NT so that it could be used as the source code server. That required a moment of faith since NT was then using a prototype version of the NTFS file system. “The networking group took this very seriously,” he said, “and made sure it was ready for internal deployment. Once it was rolled out, we never stepped back. Obviously, if the file server goes down, it’s a disaster. So it was a big moment for us, getting over that hump.”

Later, as the development of Windows NT 4.0 wound down, Thompson’s team took on Active Directory (AD), Microsoft’s first directory service, which debuted publicly at the Professional Developers Conference (PDC) in 1996. “Before AD we had NT domains for our infrastructure,” he said, “and going to AD was even more complex. We deployed AD very early, first with our team, and then the wider Windows team. We threw the switch on Redmond [campus] AD in April 1997, [some four months before Windows NT 5.0/Windows 2000 Beta 1 shipped].”

Microsoft rolled out AD on-campus in stages, Thompson said, in stages. The campus went to a multi-forest AD topology with Windows Server 2003 last year. “We do a complete deployment internally, then push it out to the JDP [Joint Development Partners], testing over 250 usage scenarios. We get bug reports, feature feedback, and complex scenario testing that really proves the product.”

Windows Server 2003 hit 99.995 percent availability at the Release Candidate 1 (RC1) stage last summer, and the Microsoft.com website was fully deployed on Windows Server 2003 when RC2 rolled out in November 2002.

“Heavy usage internally and by close customers is key,” Thompson told me, “and we have a more mature view of what the product is now [compared to the early days]. We’re not just shipping bits in a box, but are also shipping a wide range of complementary tools, products, services, and documentation.” And Thompson explained that the teams working on Outlook 11, Exchange Server 2003 (“Titanium”), and Windows Server 2003 are all working much more closely together to implement complete end-to-end scenarios that meet customer needs. In the past, these products were basically developed in isolation.

Are you being served? A look at product maintenance

“Servicing has definitely matured over the years,” Lucovsky added. “We do a lot of work figuring out the right mix of service packs, hot-fixes, development branches, betas, and JDP customers for each product.”

“We’ve really extended the time that we service our products,” Thompson said, because when Microsoft ships a server product, customers may use it for up to ten years. So-called volume, or mainstream, service lasts seven years, but the company has constantly evolved the way it supplies updates and fixes over time. First, Microsoft has to be sure that bug fixes are applied to all of the applicable development branches. “Our work in rapidly addressing security vulnerabilities means that we now aggressively issue hot-fixes when we can,” Thompson noted. “It used to be that [service packs] were flexible. But customers made it clear that they wanted bug fixes only [in service packs]. That leads to an interesting question, though: what, exactly, is a bug? Is a missing feature a bug? [Windows] NT 4 SP3 was the end [of major new features in services packs]. After that, we went to Generally Deployable Releases [GDRs]. So SPs are called GDRs, and everyone can and should [install them]. They shouldn’t break applications.”

One side effect of trunk servicing is that Microsoft must maintain test environments for every permutation of its recent operating systems. That means that the final, or “gold” release of Windows 2000 is one branch for testing, Windows 2000 SP1 is another, Windows 2000 SP2 is another, and so on. “In our IT organization, we maintain a Windows 2000 infrastructure just so we can do live rollouts to Windows 2000 systems and test them in a production situation,” Thompson said. “It’s a big expense.”

Hot-fixes are treated as narrow releases that should fix only one specific problem and not affect other parts of the system. Thompson said that customers should generally only apply a hot-fix if they’re affected by the problem the fix addresses. However, security fixes are another issue altogether. “We expect all of our customers to install the security fixes,” he said, “so we are very careful with them, and do the right kind of testing. They are GDRs.”

Trunks, trees, and branches

As noted earlier, the various Windows versions require a series of product development code forks, where each different Windows product “branches” off the main development “trunk” over time. So each Windows release builds off the last, and at least two different versions—Windows Server 2003 and Longhorn, at the time of this writing—are in simultaneous development. Because Windows Server 2003 was split from XP, the server product basically builds on XP. Longhorn, a client release that will succeed XP in a few years, is actually building off the server branch code base, and not XP as you might expect.

“The mechanics of doing this are mind-numbing,” Lucovsky told me. “We have a main branch of code for the current Windows version, and that branch becomes the source base for hot-fixes and the next service pack. Once we spit out a service pack, that becomes a branch and now we have two branches we have to test for hot-fixes and service packs. We can’t tell customers to install, say, SP1 and then do this hot-fix. And this is going on for every [Windows] release, so some have 2 or 3 service packs, many hot-fixes, and many security fixes. Every one of these is a managed collection of 50 million lines of code. It’s a pretty big accounting issue. Additionally, for each main branch in active development, we also have roughly 16 team-level branches to allow team-level independence/parallelism while working on a common main line branch. Each team maintains a complete build lab environment that builds an entire release including the team’s latest changes and periodically integrates their tested changes back into the associated main branch so that others can see their tested work.”

Going to War: Triaging Bugs in the War Room

During the mad dash towards RTM, the heartbeat of the project is the War Room, where the War Team meets two to three times daily, five days a week, or six days a week now that Windows Server is in its final days of development.

“The War Team goes over reports and metrics to see where the project is at every day,” Thompson told us, an understated explanation that did little to prepare us for the horrors of the War Room. “Everything is automated now, but back then we came in and passed around a piece of paper that showed us how we were doing. There were, maybe, 15 to 20 people in the room. Now it’s very different.”

It sure is.

For Windows Server 2003, the War Room is run by Todd Wanke, who we eventually found to be an amazingly likeable guy. However, in the hour-long War Room sessions, Wanke rules with an iron fist, asking trusted lieutenants for advice here and there, but moving the process inexorably forward with little patience for excuses or, God forbid, product team members who don’t show up for War Team.

Here’s how it works. Every morning at 9:30 a.m., representatives from various Windows Server 2003 feature teams meet to triage bugs. They file into conference room 3243—whose exterior sign has been covered up by a handwritten note that reads “argument clinic”—in building 26. There’s a large conference table in the center of the room, but most of the participants have to stand, as the room is always overflowing with people. On the day we attended a War Team meeting—the first time any outsiders were allowed to view the inner sanctum for Windows Server, and only the second time overall during the entire development of NT and Windows (I assume the first was Showstopper author G. Pascal Zachary)—the team progressed through about 50 bugs, most of which were simple branding problems, though I’ve agreed not to discuss the specifics of any bugs discussed that day.

(Because we attended War Room very late in the development of the product, and the biggest outstanding issue was the last-minute name change from Windows .NET Server 2003 to Windows Server 2003.)

Every bug is logged in an incredible bug tracking system, each accompanied by a dizzying array of information about how the bug was found, which customers, if any, were affected, and a complete history of the efforts made to date to eradicate the problem. Wanke moved quickly through the bugs, calling out to members of specific feature teams to explain how the fixes were progressing. If there are one or more bugs in IIS, for example, a representative of the IIS team needs to be present to not only explain the merits of the bug, but whether customers are affected, how the fix might affect other parts of the system, and how soon it will be fixed. This late in the development process, bugs are often passed along, or “punted,” to the next Windows release—Longhorn—if they’re not sufficiently problematic.

The atmosphere in War Room is intimidating, and I spent most of my time in the room, silent and almost cowering, praying that Wanke wouldn’t turn his attention to me or my group. Heated argument and cursing are a given in War Room, and the penalty for not being on top of your bugs is swift and cruel ridicule from the other team members.

The most virulent treatment, naturally, is saved for those foolish enough to blow off a War Room meeting. On the day I attended, the UDDI feature group had four of its bugs punted to Longhorn because they had failed to show up for War Room. When someone argued that they should be given another day, Wanke simply said, “Fuck ’em. If it was that important, they would have been here.” Pressed again on the importance of these bugs, Wanke put an end to the discussion. “Tough shit, we’re shipping. They can fix it in Longhorn. Next bug.”

Once the hour-long meeting was over, we sat down and spoke with Wanke, who was almost a completely different person in private. “You run a mean meeting, Todd,” I told him, as we sat down.

Wanke’s background includes stints with NCR, America Honda, and an unspecified and mysterious-sounding security-related assignment as a US government contractor, and he’s been with Microsoft for nearly eight years. Before joining the Windows team, Wanke was one of the original architects of the Microsoft.com website and he spent three or four years as an “Internet guy” at the company before all of Microsoft found the Internet religion. In our meeting, Wanke explained how he fell into his new job, what he does now at Microsoft, and how the War Team works.

“My job is to manage the day-to-day operations with regards to shipping Windows,” he said. “I’m responsible for 8000 to 10,000 developers, program managers, and testers, and I have to make sure they’re doing the right things every day.”

War Team, he said, consists of a very broad set of people within the Windows team, all of whom are responsible for different areas of the project. They are test leads with responsibility for such things as TCP-IP and other low-level technologies, some developers, people that do the build every day, people that do build verification tests, and others. “Every area of the project is represented,” he told us. “The daily marching orders [for the Windows Server team] come from War Team, and also from the broad emails I send out. These emails are almost always Microsoft confidential, or even higher than that, emails that are very confidential and sent only to a much smaller group of people.”

As we witnessed, War Room is a very structured event, occurring at the same time every day and lasting exactly one hour. The team members look at the same bug system every day, and they often go over the same bugs until they are fixed. “If you’re not there, it’s not good,” he said.” Microsoft people have a strong sense of ownership for the product, and they want to make sure the right thing is happening. But if people aren’t there, I lay into them. I’m the ass-kicker.”

In addition to the morning War Room meeting, the Windows Server team holds an afternoon meeting from 2 to 3 p.m. and, if needed, another one from 5 to 6 p.m. The daily build usually starts at 4:30, but it can be delayed to 6, so this last meeting gives the team a chance to go over any final bug fixes that will be added to that day’s build. “The structure is very important,” he said, “and we need to know where the build is at all times. We look at the quality of the build, various stress levels, and all of the things that run overnight, anything that we need to follow up on. We get detailed reports, and review everything that goes into the project.”

Each of the feature teams also has its own War Room, so there could be as many as 50 such meetings each day, each going over a specific component of the system. These other War Room meetings occur at 8 a.m., every day. When a bug fix passes the local War Team process, it’s introduced at Wanke’s meeting. “They can’t come into War Room unless they’re fix-ready,” Wanke said. “They must be fix-ready.” Because there isn’t a single person making decisions, there is a system of checks and balances through which each bug fix passes before it’s introduced into the build.

The complexities of building Windows are staggering. “To simplify things, let’s say Windows consists of 100,000 files,” he said. “Usually, there are seven source code depots, each containing an exact replica of all of the sources, though at this point, we’re down to just one. Every development group has its own depot, so that when a developer writes a fix, he can compile it into the depot for testing. If the build compiles locally with his fix, they can test it there and then check it into the main depot in the main build lab.”

Not every build is successful, of course. Occasionally, Windows Server suffers from what Microsoft calls “build on the floor,” when a fix breaks some other part of the system, rendering the build unusable.

“That’s brutal,” Wanke told us. “There was a point about a year ago when we didn’t get a build out for seven days. We had to send an email to the product group executives at the company explaining the problem,” and the company entered into its private version of Defcon-5. “All the red flags went up,” he said. “It’s very ingrained in the developers not to break the build. They do their fix, do a buddy build, and then check it in. But they can’t go home. We’ve sent out calls at 3 a.m. when the build is broken, found the developer that broke it, and got him into work right then and fixed it immediately. The developers are on call 24 hours a day. There’s definitely an escalation process. A broken build is considered a critical, severity-1 problem.”

As the Windows Server 2003 development cycle wound down, the bug count fell dramatically, and the process was getting simpler each day. And then Microsoft announced the name change.

“We just have to live with that poor decision,” he told us. “They should have made it six months ago. Back then, we all agreed it was the right thing to do. But at this late stage—they brought in [CEO] Steve Ballmer to talk with all the War Teamers about why we made the change.”

The speed at which the team was able to fix all of the branding graphics, text, and registry entries in the system is a testament to the company’s dynamic process for fixing bugs, Wanke said. The problem was that several thousand changes needed to be made, and that would normally require several thousand new entries in the product’s bug tracking system.

“I went out and handpicked the three best developers on the team and said, ‘just go and fix it.’ One developer fixed over 7,000 references to [Windows] .NET Server. Let’s just say that there are people I trust, and people I don’t trust. I told these guys, ‘don’t tell me what you’re doing. Just do it.'”

Entering the home stretch

On the day that we attended War Room—January 21, 2003—Windows Server 2003 had hit an “absolute historic low” for bugs, according to Wanke. “We’re shutting down the project this week,” he said. “It’s done. We’re going to ship it.”

On that day, Windows Server 2003 had just 10 active bugs, not including those related to branding. “So let’s say there are about 150 outstanding issues to address,” Wanke told us. “Of that, we’ll fix about 100. All of the bugs are severity rated from 1 to 3, plus they get a priority rating. We have a few severity-1 bugs left to fix, and those all have to be fixed for us to ship.”

Wanke said that the server team had already fixed all of the known security vulnerabilities.

“We’re very happy about security,” he said. “It’s fun to see where we are [with security]. I’m personally very impressed with the work that went into it, the fixes, and the thought process. We all think it’s very secure. The [Trustworthy Computing] security push [last year] was a big milestone for us, and everything will be easier going forward because of it. It’s easier on the developers because they all have the same mindset and goals now, the same education about best practices. There used to be different methodologies between different groups. The security push unified it. Now it’s easier for everyone to communicate and see the end goal.”

With the completion of Windows Server 2003 development, the development team will enter a transitional period. First, the product will enter escrow, and the build process will be frozen. That build is then deployed around the campus, including in Microsoft’s corporate infrastructure. “That is the final build,” Wanke noted. “Then we sit on it for a period of time. During this time, there are no core fixes made to the product.” The escrow build will also be handed out to testers and JDP members, he said.”

If any issues do arise during the escrow period, the War Team makes case-by-case decisions about whether to fix the bugs. If a bug necessitates a kernel fix, a new build will be created, and escrow is reset.

“A change to a core component could delay RTM,” Wanke told us. “We run it before we customers to, and we have to run it a number of days before signing off on it. It’s a long haul.” Every feature team working on Windows Server 2003 must run the escrow build for 21 days without restarting before the build can be declared golden master and released to manufacturing.

But Wanke isn’t worried about the exact schedule, as the outcome is finally a foregone conclusion after years of work. His team is now preparing its RTM party—outside on one of the campus’ many soccer fields, weather permitting; inside a garage, if not—and Wanke has other RTM-related concerns he must address, including the launch venue.

“I’m working with the launch team to book a venue,” he said. “They need 95 percent confidence dates.” They’re also talking to OEMs to ensure systems are ready for launch, ISVs, marketing folks for signs and posters, and so on. “And I have to make sure that the 8000 people who deserve a ship award get one,” he added.

In the end, all this dedication will result in the more secure and reliable operating system Microsoft has ever created, and it’s impossible to overstate Wanke’s contribution to this project. “I basically haven’t missed a War Team in a year and a half, give or take a day or so for personal reasons,” he said, “every day, six days a week at the end of the schedule. We let people bring their kids in on Saturdays, it’s a family day. There’s no swearing allowed on Saturdays. But you still have to be there, and we still have to make a build.”

Though I understand that managing this project is a one-shot deal, due to the time and stress requirements, I just had to ask: Would Wanke run War Team on a future Windows version?

“No way,” he said, laughing. “No way.”

Launch

Weeks ahead of my campus visit, Microsoft’s Bob O’Brien told me where and when the firm would launch Windows Server 2003.

“The launch date will be April 24, 2003 in San Francisco,” he wrote. “Together with Visual Studio .NET, Microsoft will unveil the products and demonstrate how customers can quickly build, deploy and manage a complete server solution for connected platforms … I wanted to personally tell you to mark your calendar for April 24 to come and celebrate the much-anticipated launch!”

As the launch grew closer, Microsoft fine-tuned Windows Server 2003. It introduced a new browser “hardening” feature called Internet Explorer Enhanced Security Configuration (codenamed “IE hard”) at the last minute. It would be enabled by default and was designed to reduce the risk of attack from insecure web-based content.

“Windows Server 2003 provides customers with a more secure foundation out of the box, to help them build solutions that reduce potential areas for attack in the server,” the firm revealed in a press release. “The product will be shipped to customers in a locked-down state, with more than 20 services turned off by default or running with reduced privileges to help IT administrators run the most secure configurations … Internet Explorer technologies in Windows Server 2003 will have a default security setting of High.”

In late February, Microsoft released Windows Server 2003 build 3778 into escrow, hoping to make it the initial release. But the firm found “a small flaw … and the Microsoft development team together with the JDP decided to give it all a few more days (builds).” An anonymous source—highly placed in the server team—told me that Microsoft had long targeted March 12, 2003 as the target date for the RTM version of Windows Server 2003. But this new flaw delayed that release. And so a new RTM target date was set for March 19.

“This will not delay [our go-to-market] activities or the official release of the product targeted to be April 24, 2003,” the source added. “But it will cause minor problems for authorized replicators [ARs] as they will have one week less to produce the CD media that goes into the finished package.”

In mid-March, Microsoft began briefing the press about the coming launch of Windows Server 2003. They were “days away from RTM,” I was told on March 10th, and Microsoft was “on track to ship on April 24.” I was told that there was “no calendar date” for RTM—they were, of course, unaware of my more senior source—and Microsoft “didn’t miss a date.”

Regardless of the story, Microsoft launched Windows Server 2003 on April 24, 2003, as promised.

And I was surprised when Todd Wanke and Iain McDonald—the latter of whom you may remember from a story in Programming Windows: Are You Experienced? (Premium)—came up into the audience ahead of the event to find me and give me a “War Team” t-shirt and a Swiss watch, the Windows Server 2003 ship award. It was a truly special—and unwarranted—gift that I’ll never forget.