Is Mixed Reality Worth a Try?

Scientists estimate that 80% to 90% of the information processed by humans occurs through the sense of sight. For design engineers contending with the gap that exists between the physical and digital worlds, this can be a problem. Developers of mixed reality (MR) systems, however, aim to give product designers the means to bridge this gap by integrating the two forms of data in an environment that plays to the strengths of humans’ ability to digest visual information.

How do they plan on doing this? By returning to the basics. The advantage in mixed reality systems lies in the form in which information is presented and the context in which it is applied. To achieve this, the developers of these systems have turned to existing and proven technologies: virtual reality (VR) and augmented reality (AR). What is of significance is how the two are combined.

Two Ways of Viewing It

As if a technology that represents a mix of two systems is not confusing enough, the MR label has two meanings. It can be used as an independent concept or to classify types of reality technologies.

The broader interpretation, referenced in the reality-virtuality continuum, describes the coverage of all possible variations and compositions of real and virtual objects. The continuum ranges from a completely real and natural environment to a completely virtual environment.

Developers espousing the independent concept, however, see MR as a combination of the best features of virtual reality and augmented reality. This approach produces visualizations in which physical and digital objects co-exist and interact with each other in real time.

A Question of Context

Similar to AR, MR superimposes digital content (e.g., text, 3D graphics, sound and video) onto the backdrop of a real-world environment. By placing information in context and allowing the user to simultaneously see elements of both worlds, MR helmet-mounted displays (HMDs) greatly enhance the user’s ability to consume and process information.

The element that sets MR systems apart from AR systems is that MR content appears in the form of a hologram within the actual real-world environment, an approach that eliminates the constraints of conventional 2D visualizations.

With MR headsets and gloves, designers can use hand gestures and voice commands to manipulate and interact with their designs as if they were real objects, walking around and looking inside the design. In addition, MR allows users to see the real world alongside virtual objects anchored to a defined point, helping users to treat them as real objects.

Until recently, Microsoft’s HoloLens smart glasses dominated the MR market, but an HMD from Magic Leap blends holographic data with the real world in a way that some claim represents a significant advance toward the provision of true spatial context for the user, aiming to place full-sized, 3D digital data into the physical world.

What’s the Appeal?

Even after hearing about some of MR’s more impressive features, design engineers inevitably may ask: How will MR enhance the product design process? What will the hybrid visualization technology let engineers do that they can’t already achieve with more established enhanced-reality tools?

MR system advocates say that by taking the best of AR and VR, the hybrid systems deliver richer interactions, enhanced efficiency and greater flexibility.

“The idea that you can look at a complex system and modify it can be particularly appealing,” says Todd Zielinski, senior director of electrical engineering for the Bresslergroup, a design consultancy based in Philadelphia. “AR allows you to see and potentially diagnose problems or add information in a spatial context, while MR implies you can now interact with that system in a more meaningful way in real time.”

MR’s efficiency entails streamlined design processes, greatly enabled by the VR element of the MR system.

Very early in the design process, when there are few physical objects accessible to serve as design aids, VR is instrumental in helping design engineers to view their designs before they are made and experience key design concepts. Engineers can take their CAD designs, merge that geometry with the contextual geometry of neighboring components and digital human models, and gain a real sense of how users will interact with the new product.

“Interaction modeling is examined and tested early in the process, which helps significantly in defining the operation of the product, and what features it must have—and how they work—in hardware and software,” says Zielinski. “This helps minimize the time spent on product development due to missed or incorrect requirements by pushing system modeling earlier in the design process.”

Once engineers immerse themselves with the products in their chosen context, they can conduct engineering reviews, which normally would not occur until after parts are available. This effectively lets designers jump months, or years, into the future. If any shortcomings emerge in the design surface, development teams can evaluate mitigating options or redesign the product and then determine if the changes address the shortcomings.

These advantages help development teams reduce the amount of time their products are in design because a virtual prototype can be created in short order. “This is a very fast, iterative process—even faster than 3D printing of prototypes,” says David Francis, chief marketing officer for Theorem Solutions.

In addition, MR systems promise to smooth the transition from design to manufacturing, holding out the potential for an end-to-end efficiency improvement. “The design can be checked [so] that it can actually be assembled,” says Francis. “Can you get a wrench or screwdriver into a restricted space? Can the material be formed as the designer has designed? Can a welder actually get the equipment into that space to create a proper weld? Is the design safe? With mixed reality, you can use the actual tools on the holographic/virtual product.”

An MR Workflow

Typically, an engineer begins working with an MR system by taking the relevant 3D CAD files and preparing them for use in a virtual environment. This means translating them into a lightweight format suitable for use on an HMD, smart glasses or a tablet.

To accomplish this, the designer processes the 3D CAD models, optimizes and decimates them so that the MR system can visualize them. The system then displays the data in immersive 3D.

Moving Toward Greater Collaboration

These immersive 3D visualizations open the door for greater collaboration among development team members, supporting an interactive quality previously unheard of before the introduction of VR technology.

“The real power comes when multiple users are viewing the data collaboratively,” says Francis. “For example, Theorem’s software allows multiple engineers using MR headsets to view the data in the same session. The assembly appears as a hologram that can be interacted with by all participants—although one at a time. If they are in a remote location, then they appear as an avatar to their colleagues. VR, MR or desktop devices can all be used concurrently. Discussions around the design can be annotated with video/audio or text remarks, which are accessible following the review.”

As impressive as this sounds, how do MR systems stack up against pure VR systems for collaboration?

For example, when handling large assemblies like airplanes, can MR headsets deliver the same level of performance as a VR-based cave automatic virtual environment (CAVE) or powerwall systems?

In these applications, MR systems can fall short. “Massive assemblies such as entire airplanes or ships can be modeled and viewed in mixed reality, and through collaboration, all parties can be involved regardless of which device they have or where they are,” says Francis. “One point on the large assemblies, however, is that mixed reality displays data in context with the environment. If you are in an office building, displaying an entire airplane would not be practical. In this example, VR would be better.”

Another area of concern is whether HMDs such as Magic Leap One and smart glasses like HoloLens 2 can support a multi-user experience adequate to satisfy engineers’ expectations in design review applications? How do MR systems compare with the VR-based CAVE and powerwall systems? Does the MR collaborative viewing experience for design reviews scale up?

The answer to these questions is: It all depends on the application and the support infrastructure.

“When employed by individual engineers, HMDs provide highly immersive visualization and experiential evaluation of service and assembly requirements during product development and process planning,” says Eric Kam, manufacturing business channel marketing and alliances director at the ESI Group. “For buyoff reviews or other gated milestones in an engineering lifecycle, HMDs can make it difficult for teams to participate in virtual reviews. Collaboration between engineers in a common virtual environment, but each using their own HMD stations, is a way to allow collaboration. However, this too has its limits.”

That said, a lot of the MR system’s performance is determined by its support systems. The number of MR system users able to view the digital product representation or digital twin during a design review depends on the capacity of its network and servers. If these two elements are up to the task, then a distributed development team can often expect a satisfactory experience, regardless of location.

Challenges and Shortcomings

There is no doubt that digital reality technologies in general, and MR systems in particular, have much to offer design engineers. An honest look at the technologies involved indicates that all of the options require improvement.

What’s holding these systems back? The tug of war between cost, compute resources and ergonomics tops the list of reasons for the slow adoption rate.

“Existing MR systems do not seem to penetrate far enough into potential low-cost applications, which would drive wider adoption,” says Zielinski. “The bulk of applications centers on higher-power, Linux-based systems and have not effectively integrated with the mainstream of lower-cost IoT applications.”

Additionally, the number of device types supporting MR points to the fact that the technology still hasn’t undergone the shakedown process that culls out unwieldy designs.

“Current hardware solutions are uncomfortable to wear due to where the center of gravity is, weight and in some cases excessive heat,” says Theorem Solutions’ Francis. “Moreover, many people do not like to wear them because they are neither aesthetically pleasing nor are they able to look at other people without the device impeding their view.”

Microsoft considered these shortcomings when it designed HoloLens 2, and aimed to address all of the issues. Only time and the market will tell how successful it was.

As for performance, MR systems face challenges in meeting the high standards of design engineers, specifically in the areas of data handling and presentation.

“To have interaction with the elements overlaid in AR to be useful in an MR context, the latency between the action and a visible result must be low enough to have a meaningful connection between the content and the user,” says Zielinski. “This implies that the connected device that we wish to interact with must have a reasonably small communication latency, generally implying reasonable bandwidth, small communications packet size and a processor capable of supporting the application and backhaul infrastructure. This doesn’t mean that the processor necessarily has to be more expensive, but the scalability and latency of the architecture has to be considered seriously.”

In information presentation, MR systems still wrestle with issues like field of view and resolution. “For operator visibility, if the field of view is too narrow compared to real visibility, then some conclusions from a review might skew in the wrong direction,” says ESI Group’s Kam.

As for resolution, to correctly address many of the challenges confronting development teams, the visual fidelity and image sharpness need to be very high.

What’s Next?

It is likely that MR will follow in the footsteps of VR, carving out a niche in mainstream consumer entertainment systems first, and then using that position in the market as a springboard for design/development, industrial and healthcare applications. At this point, early adopters will move into a second wave of ecosystems as standardized methods become more available.

More ESI Coverage

More Theorem Solutions Coverage

Digitally Transforming the Design Review Process

In only a short time, ATM manufacturer NCR (National Cash Registers) have revolutionized their design review and prototyping processes with the help of Theorem-XR.