Copilot

Windows 11 version 23H2 introduces a new Artificial Intelligence (AI) companion called Copilot that combines the content creation capabilities and intelligence of Bing Chat with a limited selection of useful Windows setting configuration features.

In its initial release, Copilot in Windows 11 is available in preview in North America and parts of Asia and South America. But it will come to additional markets over time.

Copilot appears as a sidebar on the Windows 11 Desktop, and it provides a text-based interface by which you can ask questions and interact with the underlying AI capabilities.

Copilot in Windows 11 is just one of many copilots–AI-based assistants–that Microsoft is introducing across its various products and services. Underpinning this interface is a set of foundational AI-based services called Microsoft Copilot that is also used by Bing Chat, Bing Image Creator, the AI capabilities in Microsoft Edge, and Microsoft 365 Copilot for businesses (and, soon, for consumers too). This common foundation means that when third-party developers extend the capabilities of any of these offerings that this new functionality will work with all of these offerings.

A bit of history

To understand Copilot, one needs to understand how we got here. After all, this and the other new AI capabilities Microsoft is introducing in Windows 11 version 23H2 seemed to come out of nowhere. So here is a short slice of recent history that helps it.

In January 2023, Microsoft announced that it was expanding its multi-year, multi-billion-dollar investment in the pioneering AI firm OpenAI with the express aim of “completely transforming” every Microsoft product with AI capabilities. Previous to this, AI was seen as something futuristic. But OpenAI’s rapid AI advances had reached the point where Microsoft felt the technology could be used more broadly.

In February 2023, Microsoft announced Bing Chat, an “AI-powered copilot for the web,” built on OpenAI’s ChatGPT and a proprietary Microsoft AI model called Prometheus that it said provided “more relevant, timely and targeted results with improved safety.” Like ChatGPT, Bing Chat is what’s sometimes called a chatbot. It’s an alternative to traditional web search that uses a chat experience similar to text messaging that lets users easily refine complex searches using text-based prompts until they get what they’re looking for. Bing Chat can also summarize information and even generate written content: For example, it can help you compose an email or other correspondence, kickstart a blog post, resume, or other work, and even personalize the tone, length, phrasing, and other aspects of its output.

In March 2023, Microsoft announced Bing Image Creator, an AI-powered image creation tool based on OpenAI’s DALL-E technologies. As with that tool, users can simply describe the image they have in mind using a text prompt, and Bing Image Creator will generate multiple versions, and in a wide range of styles, from which to choose.

Bing Chat and Bing Image Creator are both available on the web and via the Microsoft Edge sidebar.

Also in March 2023, Microsoft announced Microsoft 365 Copilot, a “copilot for work,” for its commercial (business) customers. Unlike Bing Chat and Bing Image Creator, Microsoft 365 Copilot is not free. Indeed, at $30 per user per month on top of whatever other monthly Microsoft 365 license costs, it’s quite expensive. But it’s also very powerful, with hundreds of new capabilities across specific apps like Word, Excel, PowerPoint, Outlook, Teams, and more, and deep integration with an organization’s private corporate data. Microsoft 365 Copilot also includes a business chat capability, later named Bing Chat Enterprise, that brings the text-based prompt functionality of Bing Chat inside of organizations.

Then, in May 2023, Microsoft announced Windows Copilot, now simply called Copilot, or Copilot in Windows. As noted above, this feature appears as a sidebar on the Windows 11 Desktop and it provides all of the capabilities of Bing Chat combined with a handful of Windows 11-specific features that are described later in this chapter.

Finally, in September 2023, Microsoft announced the rebranding of Windows Copilot to Copilot (in Windows) and its schedule for rolling out this feature and Microsoft 365 Copilot for commercial customers. It also revealed that it would later offer a Microsoft 365 Copilot for consumers, but provided few details.

Microsoft rolled out Copilot to Windows 11 version 22H2 in preview form September/October 2023 and then shipped Windows 11 version 23H2 with Copilot enabled by default alongside other new AI-based features in apps like Clipchamp, Paint, Photos, and the Snipping Tool on October 31, 2023.

And here we are.

We cover the other new AI-based features in Windows 11 version 23H2 throughout the book where needed.

Get to know Copilot

Copilot is available by default in the Windows 11 Taskbar: It’s the colorful new icon to the right of the Search bar (and to the left of the Task view icon) in the default Taskbar layout.

When toggled on, Copilot displays as a sidebar on the right side of the Desktop. You can likewise hide Copilot by clicking its Taskbar icon again.

You can also toggle Copilot by typing WINKEY + C.

A couple of points about the Copilot sidebar: It is not resizable and it cannot be moved to the other side of the Desktop. Open windows will move aside to accommodate the sidebar but they can appear below it if you move them there. If you hide all open windows–either by clicking the “Show desktop” area at the far right of the Taskbar or by typing WINKEY + D, Copilot will remain visible on the right side of the display. The only way to hide it is to toggle it off.

You can easily remove the Copilot icon from the Taskbar if you don’t wish to use this feature or simply want its Taskbar real estate back and will use the keyboard shortcut instead. To do so, right-click the Taskbar, choose “Taskbar settings,” and then toggle “Copilot” to “Off” in Taskbar settings > Taskbar items.

Related to the above points, there is currently no way to use interfaces like Alt + Tab or Task view to switch the focus to Copilot. If you are using Copilot and switch to an app, the only way to switch back now is to click it with the mouse. (Or, if you have a multi-touch display, touch it.) This will change in a future update, Microsoft says.

Like Bing Chat, Copilot presents a textual, prompt-based chat experience by which you ask the underlying Microsoft Copilot just about anything. It uses AI to respond, and in many cases, it works much like a traditional web search engine like Google Search.

We explore these capabilities later in this chapter.

While Copilot is deceptively simple looking, this app has many UI components that can seem non-obvious until you get to know them. So let’s go on a quick tour. From top to bottom, you will find:

Refresh. This circular button at the top of the Copilot sidebar is used to reload it as if you had just opened the sidebar. This will clear any previous chats.

More options (“…”). This button displays a small menu of choices, among them a link to Copilot settings. There’s not much there yet, but you can expect that to change over time.

Chat history. Click this clock-like icon to view your chat history.

As with the similar History feature in a web browser, you can click any chat in the history to view it again and, if desired, continue the chat with follow-up questions. If you hover over an item here, you will see pop-up “Edit” (pencil) and “Delete” (trashcan) icons: The Edit choice is there so you can rename the conversation, not edit it.

Choose a conversation style. By default, Copilot is configured to use a “More Balanced” conversational style. But you can configure it to be “More Creative”–a good choice for creating fictional content that needs to be longer or more descriptive—or “More Precise,” which is ideal when you need short, objective answers to specific questions. You configure this using the large, three-way switch under “Choose a conversation style.”

Each conversation style has its own color scheme so you can tell at a glance which style you’re using.

Suggestions. Copilot displays a starter suggestion when it loads to help the shy or confused get started. It will also offer follow-up suggestions after it’s answered a question.

New topic. Click this button, found to the left of the “Ask me anything” box, to clear the previous chat and start a new conversation.

Ask me anything. This is the text box you use to interact with Copilot. It’s like the search box on Google.com or any other web search site.

Each query is limited to 2,000 characters, which should prove more than adequate for most people. But another limitation could be more challenging: You can engage in up to 30 conversations–or queries–per day because of Microsoft’s back-end expense of running this service.

Use microphone. If you’d rather use your voice, you can click the microphone in the upper-right of the “Ask me anything” box to ask a question or otherwise interact with Copilot.

Copilot does not natively support voice output, so it doesn’t (yet?) work quite like Amazon Alexa or Google Assistant. You can, of course, use Windows 11’s Narrator feature to read Copilot’s responses back to you. But it’s not appropriate as a general-purpose Copilot audio output feature. This will change, we think.

Add an image. Copilot can be used to create images using text prompts, but you can also use the “Add an image” button in the lower right of the “Ask me anything” box to ask Copilot to identify items in the image.

Both of these features are discussed in more detail later in this chapter.

Ask Copilot a simple question

Copilot can be used as an alternative to a traditional web search engine, where you can ask it simple and complex questions. But it also has some unique advantages. Your interactions with Copilot occur in the form of conversations in which you can ask follow-up questions to refine the answer or get more details. And because Copilot stays on-screen next to whatever applications you are using, you can use it to learn more about a topic you’re reading without needing to navigate away to another browser tab or app.

Here’s a simple example: You might ask Copilot the following:

What day of the week was Christmas 1985?

When you do so, Copilot will hand off this query to its own search engine, Bing, for the answer (which is Wednesday). That answer appears in real-time–it seems like Copilot is typing the answer as it responds—above the “Ask me anything” box in the upper two-thirds of the sidebar.

You can click “Stop responding” at any time to start over or ask another question.

As with web search, you don’t need to worry about capitalization, punctuation, or any other niceties. Indeed, you don’t even have to be polite, though adding the word “please” will differentiate you as a human.

Even with a simple query like this, you will notice two unique Copilot attributes: It’s chatty–even though you asked for something very specific, the answer it gave was quite verbose–and it cited the web sources it used to find the answer. That latter point is important because these early AI-based solutions are known to suffer from hallucinations–e.g. “mistakes”–and sometimes need to be fact-checked by their human overlords.

Remember to click “New topic” when you’re done with any conversation. Otherwise, Copilot will assume you’re still discussing the same topic. (If you don’t see it, mouse to the left of the “Ask me anything” box and it will appear.)

If the answer you get is too chatty for the topic, consider changing Copilot’s conversational style to “More Precise.”

Rewrite a question

While the conversational nature of Copilot makes it easy to ask follow-up questions, you can also rewrite a question if you need to refine it.

To do so, locate the question bubble in the chat stream, mouse-over it, and select the “Rewrite” item that appears.

Copilot will put that question back in the “Ask me anything box” so you can refine it.

Ask Copilot a complex question

You can also ask Copilot more complex queries. For example, you may want it to recommend an itinerary for an upcoming vacation. So you could ask it something like:

Can you give me a good itinerary for five days in Mexico City?

This type of prompt requires a longer and more detailed answer, and it takes longer to generate. But the basics are the same: As before, Copilot will cite its sources after responding.

It will eventually suggest some follow-up questions too. Remember: This is a conversation and you can keep it going.

And it will occasionally display an ad: Copilot is a free service, but the cost of the underlying AI queries in the cloud for Microsoft is quite high.

To use a follow-up suggestion, just click it. You don’t need to type it in yourself.

You can also ask follow-up questions of your own. For example:

Yeah, but is Mexico City safe?

As before, the answer can be quite chatty: Be prepared for a lot of information.

Create text-based content with Copilot

You can ask Copilot to create a wide range of written content, including creative content like jokes and stories.

So let’s start there. First, change the conversation mode to “More Creative.” Then use the following prompt:

Tell me a joke

Here, Copilot will respond with a random joke and provide some relevant follow-up suggestions like “Tell me another joke.”

As with any other query, you can ask Copilot to generate something more complex. For example:

Tell me a story about a princess in a far-away land

This will generate a unique–and lengthy–story, of course.

And the follow-up suggestions are particularly interesting as Copilot will prompt you so that you can choose how to continue the story.

Copilot’s creation capabilities aren’t limited to jokes and stories: You can also use Copilot to help you with your personal and work-oriented writing.

First, make sure Copilot’s conversation mode is set to “More Balanced” or “More Precise.”

For example:

Can you help me write an email complaining about bad customer service at a restaurant called Maximo Bistro?

The response will vary based on how specific the query was. In this case, it was general and so the AI had to guess what went wrong. But you can take this as a starting point and edit it yourself. Or you can refine the query by providing more detail.

Here’s another example.

Can you help me write a text message in which I will break up with my girlfriend because we live too far apart?

Remember, if it’s too wordy, consider the “More Precise” conversation mode. Also, don’t ever break up with someone over a text message.

Summarize a web page with Copilot

You can use Copilot to create a summary of any webpage.

Doing so is easier if you also use Microsoft Edge: While viewing the page in Edge, open Copilot and ask it:

Create a summary of this web page

You may not even need to type that: If you start a new topic when Microsoft Edge is open next to Copilot, it will usually suggest “Summarize the page in Microsoft Edge.”

Copilot will examine the page and respond with a bulleted-list summary of its contents.

If you use another web browser, you need to include the URL of the page in your query. For example:

Create a summary of this web page https://www.thurrott.com/cloud/291743/microsoft-word-celebrates-its-40th-anniversary

In this case, the results are less satisfactory: Copilot searches the web for the URL and then generates a single-paragraph summary of what it found, without the bulleted list.

Create an image with Copilot

You can ask Copilot to generate an image based on a text description you provide. Under the covers, this uses the same OpenAI DALL-E backend service as Bing Image Creator, and so the results will be similar.

Here’s a simple example.

Create an image of a Mexico City street scene in the style of Frida Kahlo

After a bit of time, Copilot will present the thumbnails for the four images it generated using your prompt. And as you can see, the results are quite impressive.

You can move on from here in a few different ways.

To view one of the images in its full size, click it. Microsoft Edge will open, navigate to Bing Image Creator on the web, and display the image you selected in a larger size.

From here, you can share, save, or download the image. Note that any images you generate with Copilot are saved to your Microsoft account and will be accessible from Bing Image Creator on the web. So you will never lose any images you generate with Copilot.

In this case, “share” means sharing to a Bing Image Creator collection associated with your Microsoft account.

You can also view the other pictures that Copilot generated for you by using the “Previous image result” (“<“) and “Next image result” (“>”) overlay buttons that appear over the full-sized image on the web.

You can also refine the image results. As always, Copilot will provide some suggestions for this. Or you can add your own follow-ups. For example:

Add a brilliant red, orange, and yellow sunset

In this case, Copilot will create four more image thumbnails that incorporate the additional information. To be clear, these images are new, not revisions of the original four images.

You can also specify the type of image you want by using terms like “watercolor painting,” “oil painting,” “photograph,” and the like. As always with Copilot, the more specific you are the better.

For example:

Create an oil painting of a Mexico City street scene with a brilliant red, orange, and yellow sunset in the style of Frida Kahlo

Copilot inherits some image creation limitations from Bing Image Creator. There are no aspect ratio choices, for example, and it will only create square images. Downloaded images are limited to a 1024 x 1024 resolution. And there are no editing features: OpenAI’s DALL-E, for example, lets you expand its images with generative frames. We expect these limitations to be quickly addressed as the technology improves.

Learn more about an image with Copilot

In addition to creating images, Copilot can also inspect an image you provide and potentially tell you more about it based on what questions you ask. That image can be on your PC or on the web.

To do so, click the “Add an image” button in the “Ask me anything” box. Copilot will prompt you to “paste an image or link” or upload an image from your PC.

You can also drag an image file from File Explorer on the Desktop directly onto the “Ask me anything” box.

However you choose to add the image, it will appear as a thumbnail in the “Ask me anything” box. Now, you must ask it a question about the image: Copilot won’t just examine it and tell you what it knows.

The choices here are limited. Copilot cannot tell you where a photo was taken, even if it has embedded location meta-data, like photos taken with a smartphone. Instead, you can only ask it about the content in the photo.

For example:

What type of tree is in the photo

This is a question that Copilot can answer reasonably accurately.

There are other limitations when it comes to identifying images. It cannot perform a reverse image search, for example, and cannot identify specific individuals in pictures.

Access some Windows features and settings with Copilot

When Microsoft announced Copilot, it claimed that it could be used to turn average, non-technical users into power users by exposing hidden features and settings. But that’s a bit of a stretch. Instead, Copilot has a very limited understanding of Windows 11 and it can be used to access only a short list of features and configure a small number of settings.

Here are some things to try.

Configure a setting

Copilot can help you configure a handful of common Windows 11 settings. For example, you can use it to mute the PC’s volume like so:

Mute volume

First, Copilot will prompt you to confirm this action.

Then, it will provide more information and a few follow-up suggestions.

And if you click “Yes” to perform this action, Copilot will of course follow up again to let you know it did as you asked.

You can also ask Copilot to configure other Windows 11 settings, like:

Turn on dark mode
Change wallpaper

Perform a common task

While Copilot cannot be used to launch individual apps, it can be used to start a handful of common tasks. For example:

Take a screenshot

This will launch the Snipping Tool and start its screenshot-taking process.

Other tasks you can accomplish with Copilot include:

Turn on Do Not Disturb
Set a focus timer for 30 minutes
Snap my windows

Launch a troubleshooter

As you may know, Windows 11 includes an integrated Troubleshooting platform that will try to automatically fix problems that occur on your PC. But you can also manually access its troubleshooters if you can see that something is wrong. And now you can do so with Copilot as well.

For example:

My audio isn’t working

Here, Copilot will offer to open the relevant troubleshooter.

You can also try related queries like the following to access other troubleshooters:

My camera isn’t working
I can’t download updates