AI and XR: Redefining boundaries of reality

Jordan Cullen
Head of Product Marketing

TL;DR: This blog post explores the intersection of artificial intelligence (AI) and extended reality (XR) technologies. We dive into the AI principles that impact XR experiences, including machine learning, conversational AI, object recognition, and more. We’ll highlight applications of AI-powered XR across sectors, emphasizing the enhanced realism and interactivity that AI brings to these experiences. The blog concludes by acknowledging the challenges in implementing AI-powered XR and discussing potential solutions.

Artificial intelligence (AI) and extended reality (XR) devices have shifted how we interact with each other, our environments, workspaces, and daily lives. Separately, AI and XR are distinct technological categories with unique benefits. Together, they create experiences that are unlike anything the world has encountered.

In this blog, we’ll quickly define what artificial intelligence and extended reality are. Next, we’ll dive into their relationship, applications, and some of the challenges of AI-powered XR.

AI and XR defined

Artificial intelligence (AI)

AI is a field of computer science that focuses on creating systems and machines capable of performing tasks that typically require human intelligence. It works by simulating human-like cognitive functions, such as learning, reasoning, problem-solving, and decision-making.

Think of AI like a smart computer program. It helps machines or software, like a robot or video game, do things and act in ways that you would expect from a human. AI works by learning from an abundance of information and using it to make decisions.

We won’t be diving deeply into machine learning (ML), deep learning, neural networks, or the discipline of data science in this blog. Some will be covered briefly, but we recommend exploring these concepts and more on your own.

Extended reality (XR)

As shared in a recent Kontor blog, XR is the umbrella term that includes virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies. XR is a term and not a distinct concept or singular type of hardware or software.

Principles of AI impacting XR experiences

XR experiences are often enhanced when coupled with AI. In games, AI can alter how non-playable characters (NPC) interact with the gamer and even the overarching story. For maintenance workers, AI could recognize problems and share how to fix them across a range of machinery.

Head-mounted displays (HMD) like Meta Quest 3 use a combination of sophisticated hardware and software to apply AI to XR experiences. Below are some of the key concepts that impact its use in XR:

Machine learning (ML)

The ability of a computer system to understand and act on new scenarios or data independently, without human interference or programming.

Conversational AI

A type of AI that makes it possible for an individual to have a natural, human-like conversation with a virtual entity — a chatbot, game character, etc. Conversational AI is powered by a collection of technologies like natural language processing, machine learning, speech recognition, and more.

Natural language processing (NLP)

Gives computers the ability to understand both text and spoken words. NLP leverages deep and machine learning to process language, infer meaning, and deliver an answer. To draw a comparison to humans, computers have microphones to hear and software to read just as we have ears and eyes. NLP gives computers the ability to understand — a brain to determine what the collection of letters and words mean and the ability to formulate a coherent response.

Image and object recognition

HMDs leverage their hardware to scan environments and generate accurate 3D models of real-world objects. AI reviews the scan, identifies what objects are and their relative location, and uses this information to create interactive and realistic environments. Image and object recognition are what power experiences like that show a virtual pet jumping and playing on tables and chairs in the physical environment.

Predictive modeling

Uses data, past actions, probability, and other inputs to anticipate (predict) an outcome. It’s predictive modeling that informs dynamic and interactive XR experiences.

Applications of AI-powered XR

We’ve shared use cases of VR, AR, and MR spanning a variety of industries, most recently in our blog What’s the difference between virtual reality, augmented reality, and mixed reality? Across health care, manufacturing, architecture, construction, engineering, gaming, real estate, and others, XR has had an immediate impact.

But what about AI-powered extended reality applications? Below we’ll discuss how artificial intelligence impacts the use and adoption of XR devices and software. From training and simulation, through gaming, therapy, and more.

Training

Video by UPS | about.ups.com/us/en/newsroom

VR has been leveraged for training with documented success for years. For example, UPS has trained more than 20,000 drivers using VR and a driving simulator. Organizations using VR combine realistic experiences with hands-on and/or instructor-led classroom training. In fact, a study by PWC revealed that the use of VR for soft skills training was four times faster than in a traditional classroom setting.

When training in VR is coupled with AI, the simulated experiences become more realistic. The environments and NPCs understand a broader range of actions, react with more realistic responses, and can expose participants to a wider range of experiences that they could only previously read or hear about.

Video games

Image by Replika | replika.ai

Whether it’s a game of chess using an MR headset or a role-playing game in VR, AI has a role to play. It’s quite simple — AI used in VR creates a more immersive gaming experience. Working together, the two technologies can create and display interactive models of characters, game boards, environments, and more. AI can produce realistic-appearing content, entertaining stories that adapt from player decisions, adjusting difficulty based on skill, and more.

For instance, Replika gives the freedom to create a unique AI companion. In the game, players are encouraged to, “Teach Replika about the world and yourself, help it explore human relationships, and grow into a machine so beautiful that a soul would want to live in it.” Games like these have the potential to replace or improve on classics like The Sims or Second Life.

Entertainment

Image by Agence | transforms.ai/agence

Think about your favorite movie. No, really, think about your favorite movie of all time. Got it?

Now imagine being in that movie. Potentially as a character, but certainly as a participant. With AI and VR, this is a real possibility.

Artificial intelligence programs can ingest mountains of materials—films, scripts, interviews, source material, etc. In time, that will be enough to understand how to place you in the scene with VR being the delivery vehicle. Even if you’re just controlling a scene, the movie could play out in a completely different way. Viewers could do something as simple as swapping actors or something as complex as spoiling a secret kept from the main character.

At the point of publishing, AI can already alter film. Adding VR places the HMD wearer into the action.

A modern example that doesn’t change classic movies but has created a new type is Agence, “A dynamic film with a mind of its own.” Players control a world inhabited by AI-powered characters. The story unfolds based on the choices of the player, including if they decide to interact at all.

AI-powered XR challenges

While AI used in XR holds immense potential, their partnership presents some challenges. Data privacy and security are a top-of-mind concern for XR users because of the amount data and how HMDs and their software providers can collect data. The power of the devices themselves can also be limiting factor for adoption. Regardless, these aren’t insurmountable problems. They’re challenges that the XR community can address and overcome.

Privacy and security

VR experiences, particularly those powered by AI, often collect personal data from users. This information provides important context so the interactions feel personal. Without this context, the experience would be the same for every user.

But there is a trade off for data collection. People trade their data for a personalized experience. In return, they trust that the provider will store and process their personal information ethically and securely.

Safely securing and using personal data ethically is paramount for XR companies. This is true for XR software even when AI isn’t a feature. Many developers are up to the task, but new companies launch daily. It’s up to the users of AI-powered XR software to demand that their personal data are safe.

Processing power and efficiency

AI algorithms require substantial processing power, which can strain the hardware resources of XR devices. This can lead to increased heat generation, reduced battery life, and poor performance.

But the overall size and weight of the device limit the capacity for packing hardware within the frame. The device can’t be so large that it limits movement or feels awkward. And it can’t be too heavy, or it can cause pain in the neck or face.

Some companies have developed ways to offload some of the required processing power. For instance, the PlayStation VR2 leverages the processing power of the PlayStation 5 via a tethered connection so players can experience visually impressive and realistic VR games. Alternatively, the Lenovo ThinkReality VRX can connect to a PC to power graphics-intensive processes like cloud-based rendering.

For the VR2 and the VRX alike, both PlayStation and Lenovo, respectively, recognize the benefit of offloading computing power to a device that isn’t as restricted by size and weight. The current challenge for consumers is that a secondary device is often, but not always, needed to experience the full power of XR. In a non-enterprise environment, people might perceive this added cost as a barrier.

Future HMD developers will need to factor in the processing power required for AI-dependent XR applications. There is significant potential in XR leveraging cloud computing for heavy computational tasks. Processes like on-the-fly AI-generated environments and scenarios that would otherwise be impossible for HMD onboard hardware, will become widely accessible.

Conclusion

Artificial intelligence will undoubtedly continue to play a major role in our lives. AI will impact entertainment, our work, cars, medical care, education, and so much more. The impact of AI on virtual reality and other XR platforms is just a part of the broader implications.

How we choose to include AI-powered software and the hardware necessary in future XR applications and devices is a story yet to be written. What we know is there has been significant progress in the XR space. From AI companions to entertainment, training, productivity and more, the XR industry is adopting AI faster than many others.

The current challenges at the intersection of AI and VR, AR, and MR are solvable. New systems on a chip (SoC) like the Snapdragon XR2 Gen 2 are making XR headsets faster, more performant, and power efficient. These chips will continue to evolve, improve, and make on-device AI better and more natural than we can imagine.

The organizations who choose to adopt the emerging era of AI-powered XR, now, will be positioned to lead the market of the future. The consumers who choose to learn and experience AI-powered XR now, as it emerges, will be leaps and bounds ahead of their peers and colleagues. Those who don’t will be find themselves just like the folks who doubted the internet’s staying power in its infancy.

About Kontor

Kontor is a premium accessories brand for virtual and augmented reality. We enable extended use times and less distractions from the physical world through accessories that deliver supreme wearability.

Our team was born out of Resmed, a leading medical device manufacturer that transforms care for people with sleep and breathing conditions. We have over 30 years of anthropometric design observation and expertise and uniquely understand facial geometry and skin science.

If you’d like to discuss partnering with Kontor, send us an email using the “Contact us” button at the base of our Partnerships page and we’ll be in touch.