Microsoft Announces its Mesh MR Platform

Posted on March 2, 2021 by Paul Thurrott in Windows 10 with 19 Comments

Microsoft Mesh is a new mixed-reality platform powered by Azure that allows people in different physical locations to share holographic experiences together.

“This has been the dream for mixed reality, the idea from the very beginning,” Microsoft Technical Fellow Alex Kipman said. “You can actually feel like you’re in the same place with someone sharing content or you can teleport from different mixed reality devices and be present with people even when you’re not physically together.”

Microsoft Mesh was featured during the Ignite 2021 keynote today, with Kipman appearing first in an immersive underwater experience, followed by several other environments. This is kind of odd when you think about it: Microsoft hadn’t heavily promoted its Augmented Reality (AR) or Windows Mixed Reality (WMR) solutions in over two years. But now the firm is (re)affirming that it views mixed-reality (MR) as “the fourth wave in computing [following] mainframes, PCs, and smartphones,” and claims that the technology is somehow going mainstream despite no evidence of that happening.

But here are the numbers it provides: 100’s of millions of people around the world have experienced MR through “the devices in your pocket,” by which it means Pokemon Go on smartphones. And over 50 percent of Fortune 500 companies have “deployed HoloLens and other Mixed Reality solutions,” meaning that Microsoft and its partners have collectively sold at least 250 of those headsets.

Fortunately, Microsoft is a bit more realistic about the challenges that developers face in bringing MR to the masses: Representing people in MR with appropriate realism requires a lot of time and resources, keeping a hologram stable in a shared MR space across time and device types is a non-trivial problem, it is hard to bring high-fidelity 3D models into MR to support the file formats our customers have, and synchronizing actions and expressions of people in a geographically distributed MR session is complex, it says.

These are the challenges that Microsoft intends to solve with Mesh. The platform is based on years of Microsoft research and development in areas it says range from hand and eye-tracking and HoloLens development to creating persistent holograms and artificial intelligence models that can create expressive avatars. Mesh is built on top of Microsoft Azure, and it will provide to developers “a full suite of AI-powered tools for avatars, session management, spatial rendering, synchronization across multiple users and holoportation to build collaborative solutions in mixed reality.”

Most important, perhaps, Mesh isn’t limited to just HoloLens and WMR headsets. Solutions built on this platform will also work on smartphones, tablets, and PCs too.

You can learn more about Microsoft Mesh from the Microsoft website. And developers interested in Mesh should check out this technical overview.

Tagged with , , , , ,

Join the discussion!


Don't have a login but want to join the conversation? Become a Thurrott Premium or Basic User to participate

Comments (19)

19 responses to “Microsoft Announces its Mesh MR Platform”

  1. Chris_Kez

    I am not a developer, but based on how little traction and interest we’ve seen in the years since Hololens and WMR launched, and knowing how difficult and complex this must be, I have very little faith that this will go anywhere. It seems like Apple, with it’s highly integrated stack, on top of extremely efficient chips, and massive developer base is much better positioned to succeed.

    Who is going to develop for this? Is Zoom going to use this to bring mixed reality to video calls? Not likely. Facebook? No way. Gaming companies? Surely they’d try to use Unity or some other gaming engine. I just can’t imagine that Microsoft has come up with something uniquely performant that is going to work across multiple chipsets and multiple vendors. If it is so great why aren’t they using it in a relatively simpler and more controlled context and bringing VR to Xbox? Demonstrate how easy it is to develop for, how robust it is, how big the market is, etc. in a space where they have a dedicated consumer base, engaged developers, and can control the hardware.

    I hate to be the naysayer here because I love this idea, and god help them if MR did explode and they missed out because they never tried, but it feels like they wasting time and money with this stuff.

    • ponsaelius

      In reply to Chris_Kez:

      Occulus is owned by Facebook and support for that was specifically mentioned. Facebook may be partner for this in the consumer space. Microsoft love partners.

    • wright_is

      In reply to Chris_Kez:

      You are forgetting Microsoft Teams and Microsoft 365. Integrating all that into a MR or VR experience is probably what they are after and for business solutions, not gaming or personal use, looking at the examples in the keynote.

      That said, the Mesh MR demo in the keynote seemed very childish and not very professional.

    • derylmccarty

      In reply to Chris_Kez: I fully understand the your point. But your expamples are what does that and virtual business meetings just don't cut it with me either. However, I can see HoloLens in this mix with virtual in person meetings when the meeting is actually exploring a cave or deep underwater or on the moon. Partly, I admit, I am using those examples because even when some explorer is alone in the dark exploring, it is probably a good thing to have a companion if only virtually to increase the sense of confidence in an otherwise long trek across the red planet.

      Go back to the Martian, if Tom Hanks had a Hololens, and a team of scientists one or two of them would have noticed that fire in an enclosed space with H2 would not turn out well. Now, granted, we'd have to work at the time lag, but as I am trekking across the darkside of the moon, and a geologist accompanies me virtually, he or she can stop and say, hey what's that? Pick it up if possible. Ah, never mind, its just a fossilized horshoe crab progenitor and we have a million of them in my back yard. A paleontologist might have a slightly different reaction. And Stanley Kubrick would just look around and say, "where the hell is the oblisk?"

      I could also posit that not just a lack of application software but HoloLens expense both in buying the item and the back end communication and logistics network is currently prohibitive. But, I suspect that the military applications and investment will be what will bring down prices and back end costs. Just like the USAF investment in jets like the Boeing 707 made that industry the megazillion dollar giant it is today.

      • wright_is

        In reply to derylmccarty:
        Go back to the Martian, if Tom Hanks had a Hololens

        Don't you mean Matt Damon? And a Hololens wouldn't have helped, considering he was cut off from the Earth at the time.

        For Mars the lag is something like 14 minutes average (varies from where you are between 4 minutes and 24 minutes, I think in the film it was closer to 20 minutes).

        On the dark side of the moon, there would be no radio to the Earth, radio is line-of-sight, so you'd need to put up satellites or a string of radio towers to get the signal back to the Earth side for transmission.

        The same with your cave or under water, normal radio signals don't work well through rock or water. I'm only sitting in my cellar in my home-office and I gon't get a signal down here (thank goodness for Voice over Wi-Fi on modern phones), the same with my Bluetooth earbuds and my smartphone, if I turn my body whilst walking and my arm comes between the phone and earbuds, the connection breaks up.

        • derylmccarty

          In reply to wright_is: yep, which is my point about the current costs of logistics and communications. If what MSFT programs HL for is office and meetings from one 6G city to another, I am not enamoured of their business thrust. But if, with the military investments, we are able to clear up the RF noise, and perhaps design secure micro-repeaters that we can string along in a cave or UW we can solve the issues you raise. But that won't happen unless we dream the dream of what could be. OTOH, as I noted, we still have an issue with that pesky Einstein fellow who only wants you to trudge along at or below the speed of light, not the speed of a philosopher (or SciFi writer) who dreams at Warp speed. And if you think your arm is an impediment to communications, try behind a rock when lead is flying or punching a radio signal not only over the farthest distance between Mars and Earth, but through the local G2 star. Side note: it is Matt Damon not Tom Hanks, my apologies for not double checking my memories. But I have kinda lost interest and caring about the Hollywood types who seem to spew in direct proportion to their net worth as opposed to IQ and education.

          • wright_is

            In reply to derylmccarty:

            The book is a million times better than the film - they seem to have cut out a lot of the funny bits in the book.

            The audio book is also excellent, I have the one read by R.C. Bray, but it has been re-released with Will Wheaton.

  2. IanYates82

    It seems doubly short-sighted of them to cancel Minecraft Earth... Make it the consumer entry to this way of thinking

    I appreciate it was maybe a bit more like people having to be in the same physical space and seeing shared holograms, but keeping that (the azure anchor thing I assume is tied up in Mesh too) and also allowing for this more virtual, physically separate, interaction, would be a great showcase for the technology.

    • eric_rasmussen

      In reply to IanYates82:

      Agreed, but this is unfortunately typical of Microsoft. They build something awesome that's ahead of its time and then when technology starts to finally catch up they cancel it. Being able to work on a Minecraft project together with my kids would be so awesome, it was the aspect of HoloLens I was most excited about.

      Once Apple releases a successful pair of AR glasses and really starts to make money on AR games and experiences, Microsoft will rediscover AR and scramble to release some half-assed solution for consumers. I don't understand it, but this somehow seems to be a repeating pattern for them.

    • Paul Thurrott

      Agreed. I feel like that should have kept going.
  3. wright_is

    I saw the TWiT commentary on the keynote. It looked horrendous. :-(

    The constant "popping" in the background and bubbling hearts and smilies and handclaps. No, just NO! If I was in such a meeting, I'd log out after 5 minutes and wait for the minutes of the meeting to be published...

    • Paul Thurrott

      When the keynote started, I complained on Twitter that there were distracting emojis "responses" (hearts, claps, etc.) floating up from the bottom right corner of the screen. Several people explained how one might disable that feature, which I had already done, my issue was that they were there in the first place. Flash forward a few minutes to Kipman's demo and now we have MR audience members floating emojis into the "air" and not only can we (the actual audience) not turn off those distractions, but they have audible popping sounds too. I mean this non-sarcastically: This is typical Microsoft. They try to do something nice for the audience but they just don't think it through. They don't finish the job, in other words.
  4. rbgaynor

    Microsoft Meh …

  5. Chris_Kez

    In reply to davidl:

    Thanks, I totally forgot about that.

  6. vladimir

    I have mixed feelings about this. I am a VR enthusiast and I love my WMR headset for gaming. But VR and holograms for meeting people? Participate in a party? Attend conferences? It doesn't bring any closer to the real experience than a flat 2d screen. It just makes it more awkward. I think most people hope that this situation is temporary and can't wait to be able to do these things for real again. What I am missing is the physical vicinity, the whispered comments during a presentation, the meet-ups, the chats during the breaks and at lunchtime, the professional networking that happens in the bars at the end of the day. Does anyone really want the post-covid world to change in the direction Microsoft is indicating?

  7. SyncMe

    I am pretty sure Apple announced Collaborative AR back in 2019. I will have to look to see what is different about Mesh.

    • Paul Thurrott

      What's different is that it supports non-Apple devices, for starters. Huge difference, really.
  8. Cosmocronos

    MESH was also the original name of Onedrive, and by far better too ...

    • Paul Thurrott

      Well. Windows Live Mesh was one of Microsoft's pre-OneDrive cloud storage services. But it didn't turn directly into OneDrive (SkyDrive first), and in fact had many more features (planned and implemented). And it wasn't the first such Microsoft service either: Windows Live FolderShare and Windows Live Sync both predated Windows Live Mesh.
  9. crunchyfrog

    Cool. This is going right next to my Duo and my Band 2.

  10. winner

    Surprised it isn't called "Windows Mesh Augmented Reality".

  11. angusmatheson

    This keynote was weird. Really, really weird. And it didn’t make sense. Windows Mesh is a new service. But instead of showing people doing real things with Mesh - they showed virtual squid, grown adults feeding Pokémon, virtual Jameson Cameron’s submarine, and people dancing like lunatics with giant people making psychedelic towers behind them. And no one in the whole conference was ever seen without a hololens on - do they think in the future we will wear them all the time? How could anyone at Microsoft watched the rehearsals for this and not say - hey guys let’s tweak this a little. VR/AR or mixed reality is really exiting, and I think will be a big part of the future. But this just makes it look nuts and like something no normal person would want to do. (They also never show you what you really see through the hololens - I’ve never used them but I understand it looks nothing like they show on screen with your entire field of vision immersed in the AR/VR)