For some blind and low-vision people, AI glasses unlock a new independence

Open this photo in gallery:

Wes Ramage was born with a condition called optic nerve hypoplasia, an underdevelopment of the clusters of cells that relay signals from the retina to the brain. He can see objects, but no details. His family moved around a lot throughout Southern Ontario. As a kid with extremely limited vision, he had to devise ways of mapping his new surroundings each time. “Once I learned that environment, I was unstoppable,” he said.

Mr. Ramage could memorize layouts so well that he was comfortable riding a bike in his youth. But nothing in this world is static. One day, he was tearing along and slammed into a set of monkey bars that had been relocated unbeknownst to him, and went sailing over his front tire.

His desire to explore is unabated at age 43. These days, Mr. Ramage wears a pair of internet-connected glasses that are linked up to a large language model, the artificial intelligence technology that underlies ChatGPT and other chatbots. The hardware itself is Google Glass, the smart glasses first released a decade ago, and the company that put it all together is called Envision, which is based in the Netherlands.

When Mr. Ramage gives a voice command – such as “describe scene” – a camera embedded in the glasses snaps a picture. The image is then bounced to an AI model that interprets the scene and composes a description. Finally, a text-to-speech feature reads the description aloud through a tiny speaker behind Mr. Ramage’s ear. The whole process takes only a few seconds.

The device has become indispensable. “I wake up, have my coffee, grab my glasses, my keys, my wallet and I head out,” he said.

The leaps in artificial intelligence over the past year, particularly the combination of language processing and computer vision, have led to new and more advanced applications for people who are blind and low-vision. That includes glasses such as Mr. Ramage’s, along with AI-powered smartphone apps that go well beyond simple object recognition and can describe visual information in detail.

The technology is not perfect. Just as ChatGPT and other chatbots can make things up, these apps can occasionally hallucinate objects that aren’t there or simply misinterpret the world. But for some users, advances in AI are helping unlock a level of understanding and independence that was not available to them previously.

Mr. Ramage, who lives in Calgary, first came across Envision last year after searching the web for assistive technology. He has since found endless uses for the glasses. He has asked them about the colours of the clothes he’s wearing, called upon them for help finding empty chairs in public spaces, and used them to interpret the cooking instructions on a box of frozen pizza.

He relies on the glasses to read street signs, which is crucial when he is navigating public transit, particularly when route changes are posted on signage he can’t see. “Before, a bus would drive right by me because I’m standing at a yield sign,” he said. Now he can ask the glasses for a description to ensure he’s in the right place. “That brings a level of comfort I have not had in so many years.”

If he has misplaced something, Mr. Ramage can ask his glasses if the lost object is within view. He demonstrated this on a recent video call, prompting the glasses to locate his guide dog, a golden lab named Elm. “Dog, eleven o’clock,” came the reply from his glasses. “Dog, one o’clock,” the voice continued as Elm trotted over.

Envision was founded in 2017 by Karthik Mahadevan, who pursued the idea as a master’s student after giving a talk at a school for the blind in India on a visit home. What the kids most wanted, he heard, was independence. “I was speaking to many blind and low-vision people,” he said, “and what I understood was that for a lot of them, independence almost always meant access to information.”

The technology behind Envision has gone through different iterations, and relies in part on models from OpenAI, the company that created ChatGPT. The glasses have been able to interpret text for a while. But with the release earlier this year of OpenAI’s latest large language model, GPT-4, which can describe images, Envision was able to add the feature that narrates a user’s surroundings. The company is also testing an open-source AI model, but so far GPT-4 seems better at handling follow-up questions.

At first, “hallucinations were through the roof,” Mr. Mahadevan said. But OpenAI has done a lot of work behind the scenes this year to improve the model.

Open this photo in gallery:

Mr. Ramage, as an unpaid beta tester, has experienced a whole universe of flubs. His glasses have told him about toilets in dining rooms, menus on bathroom walls (including prices), and once mistook his mother for a pregnant woman because of how she was angled on a couch. A male relative was called a woman, possibly because he styles his hair in a man-bun.

When I met Mr. Ramage in Toronto recently, I asked him to have Envision describe me. The app accurately clocked me as a man with a beard, but mistook my plaid jacket for camouflage. “He is holding a cellphone in his hand,” the app said, correctly, “and appears to be taking a selfie,” it finished, incorrectly.

Later at a restaurant, Mr. Ramage asked his glasses to read the menu aloud. Then he cut to the chase: “Is there a chicken sandwich on the menu?” The app answered, correctly, that a chicken sandwich was on the menu.

The glasses are not cheap. Prices range from US$1,899 to US$3,499, and Mr. Ramage tapped a provincial program to cover part of the cost.

Fortunately, other options are available. Nayla Farah, who lives in Toronto, started using a smartphone app earlier this year called Be My AI that provides descriptions of photos she takes. “It is the closest thing to regaining vision,” she said. Ms. Farah, who grew up in Lebanon, was diagnosed with a degenerative eye condition when she was 12 years old and worked as an articling lawyer before she lost her sight completely. It became impossible for her to work in her home country, which is partly why she moved to Canada.

The app gives her a “reminiscence” of the time when she could see, as she put it. Ms. Farah once asked for a description of herself. The app listed off dark hair, light complexion, almond-shaped eyes and high cheekbones, all features that match the memory she has of her appearance. She has asked about the pictures lining the walls of her home, because she had forgotten where some of them were located, including a portrait she had painted herself years ago. She has asked for descriptions of the routes she has walked regularly for years, so she can grasp what the buildings look like and what kinds of trees surround her.

She has used the app to help find stores in malls, and to describe the scene outside her window when she hears a commotion. (She demonstrated on our call. “There is a building with a modern design,” the app said. “Inside the building, there is a raised concrete planter with shrubs and small trees. The planter is rectangular and has a rough texture.”) Mostly, she has found Be My AI to be accurate, although it has invented things before, such as when it insisted a pair of eyeglasses were resting on her table.

The app is provided for free by a U.S. company called Be My Eyes, which also offers a service for blind and low-vision people to connect with volunteers through live video calls when they are in need of help. Chief executive Michael Buckley cold-called OpenAI in January, thinking there might be an opportunity for the companies to work together. He learned that OpenAI was about to launch GPT-4 with vision capabilities, and Be My AI was born. For now, the company is able to access the model for free (some 70 per cent of blind and low-vision people are unemployed or underemployed, he said, and would have limited means of paying). “The truth is we’re in discussions about the long-term model between us,” Mr. Buckley said.

He doesn’t know yet how the app will affect his company’s original service – the one that pairs users with real human beings. Perhaps people will find they don’t need to call volunteers as often to read the expiration dates on cartons of milk or locate misplaced sweaters, though Mr. Buckley thinks there will continue to be uses for such a service. “Whether it’s because of trust, whether it’s because of loneliness or whether it’s because of a genuine need for human connection, sometimes people want a human,” he said. In the meantime, the company is looking to add a verbosity setting to the app. Be My AI occasionally waxes poetic when the user really just wants a yes or no answer.

There have been other hiccups, too. Earlier this year, an update by OpenAI caused an uproar among the app’s users: faces were inexplicably blurred in pictures, making descriptions impossible. “The community lost their minds,” Mr. Buckley said.

Ms. Farah didn’t lose her mind, but she was annoyed. “Depriving me of the details is very detrimental,” she said.

The change lasted for only a couple of days and likely stemmed from compliance concerns related to biometric information laws in the U.S. The app does not have facial recognition capabilities, but will provide facial descriptions, and the details are generally “slightly more robust” than those provided in OpenAI’s own ChatGPT, according to Mr. Buckley.

Still, these AI descriptions can lead to awkward moments. The app once described a middle-aged friend of Ms. Farah’s as an “elderly” woman. “This person was very, very offended,” she said.

And then there is the possibility of misgendering someone, as with Mr. Ramage’s man-bun-sporting relative. Many of us make assumptions about gender based on visual cues, even if our interpretation of those cues is wrong. Artificial intelligence applications inherit our biases, and suffer from the same problems.

But if the sighted among us are free to make conclusions about gender in their own heads, even if mistaken, why shouldn’t a blind person be free to do the same with the aid of AI? Mr. Buckley has discussed the topic of gender descriptions with users, and recalled that one woman told him, “I should have the same access to information as a sighted person, full stop.”

Google, which offers an accessibility app called Lookout, has been intentional when it comes to gender. The company consulted with non-binary blind and low-vision people and opted for a more descriptive approach in the latest version of the app. If a user asks for a description of an individual, Lookout will say “this person appears to have feminine characteristics” rather than definitively label that person as a woman.

“These are challenging areas,” said Sejal D’Souza, a product manager for Lookout. “We’re definitely trying to find that balance between being very helpful to our users, but also being sensitive to people’s preferences.” The response from users has been mixed, she said, as some would like more definitive answers. “As we learn more and as the model is improved, we can try to provide more useful information.”

For now, and perhaps for a long time ahead, AI is not set to replace other assistive devices or the ease of asking another person for help with trickier tasks. Mr. Ramage’s guide dog Elm, for example, is both less intelligent and much smarter than an AI application. He can see, feel and sense danger, and can draw on nearly two years of training.

Walking around Toronto with Mr. Ramage, trying to cross a chaotic intersection where drivers blithely stopped their cars in the middle of the crosswalk, Elm proved to be a calm, unshakeable companion, deftly leading his owner to the other side.