Microsoft’s New iPhone App Helps the Vision Impaired Navigate the Real World

Posted on July 12, 2017 by Paul Thurrott in Mobile, iOS with 7 Comments

Microsoft today announced a new app for iPhone called Seeing AI that helps those with low vision navigate through the real world. Basically, it uses AI (artificial intelligence) to describe the people, text, and objects around you.

Like, seriously.

“Designed for the blind and low vision community, this ongoing research project harnesses the power of artificial intelligence to open up the visual world and describe nearby people, text, and objects,” a new post to the Microsoft Accessibility Blog explains. “The app uses artificial intelligence and the camera on your iPhone to perform a number of useful functions.”

Those functions include:

Reading documents. Seeing AI recognizes the structure of documents, such as headings, paragraphs, and lists, so you can skip through them easily using voiceover.

Identifying products. Seeing AI can identify products using barcodes and your iPhone’s camera.

Recognizing people. Seeing AI will provide a description of a person’s visual appearance, including their gender, facial expression, and other identifying characteristics.

Recognizing images within other apps. Seeing AI is available via the iOS Share interface so you can use it to recognize images in any app.

Seeing AI is essentially a research project, so it also includes some other experimental features, including a Scene channel that describes photos you capture with your iPhone’s camera.

“This app will be a game changer for many as it’s just simple to use,” Jenine Stanley, who is legally blind and reviews apps and technology for the American Council of the Blind’s tech radio show, told Microsoft.

You can learn more at the Seeing AI website.

Seeing AI is available for free from the Apple Apps Store in the United States, Canada, India, Hong Kong, New Zealand and Singapore. Microsoft says it will bring the app to more countries in the near future.


Tagged with

Join the discussion!


Don't have a login but want to join the conversation? Become a Thurrott Premium or Basic User to participate

Comments (7)

7 responses to “Microsoft’s New iPhone App Helps the Vision Impaired Navigate the Real World”

  1. skane2600

    I don't know how well it will work, but assistive technologies are always welcome.

    • evox81

      In reply to skane2600:

      I saw a demo of it on Bill Nye's (horrible) Netflix show... It was very impressive. Although I think what they demonstrated was a bit beyond what they're offering here, when he pointed it at a microscope, the Microsoft people acted really nervous and said "I'm not sure it will get that one, it's still learning" and the phone said "I think it's a microscope sitting on a stainless steel table in front of [something]"... It was great.

  2. Narg

    Played with it a little. It crashes a lot.

  3. Dont Fear the Future

    "Available in the United States, Canada, India, Hong Kong, New Zealand and Singapore."

    But I know where it WON'T be available: On Windows.

  4. Tyrone

    my mother is blind.. I downloaded the app for her but it is useless.. It requires her to see icons on the screen she can't see.. the only way it works is if the app is integrated with Cortana and you can verbally request that the app tell you what it is seeing or read a document for you. otherwise pointless.. truly great apps for blind people are apps like cortana or Siri or Amazon echo that can do things via a verbal request..

  5. mafleon

    My mobile battery is getting the drainage very fast when I am using this application if I use for 5 to 10 minutes my battery get drained more than 15 to 20%

    Development team need to see why it is taking so much battery power

    Also my application got crashed one time for which I have sent the report to the developers

    In any case this application is worth having with a blind people and it is a very good concept thanks to Mr Shekh

    I have seen his video more then One hear back and I just now got the app so I am very happy and excited to use this app . Got it from appvn ios store.