Can augmented reality help operators improve overall machine performance? This is the question Caterpillar Inc. is looking to answer through its research and development efforts into the use of augmented reality (AR) machine control.
The idea behind the system is to simplify the cab environment by using AR in place of physical operator interface components such as displays, gauges and push buttons. Instead, the operator wears a pair of AR glasses which create computer-generated images of the various screens and functions a driver will need to operate the machine. 
With the AR technology, operators can glance to their right to view the rearview camera display.Caterpillar Inc.
“The way I try to describe this to a person who hasn’t been around machines very often is [to have them], think about their car. I’m going to leave the steering wheel, gas pedal and brake pedal, everything else we’re going to eliminate…and replace with augmented reality components,” explains Lonny Johnson, Senior Engineer at Caterpillar Inc.
Creating a more customizable user interface
The Caterpillar system is designed so that when operators put on a pair of AR glasses, computer-generated images of the necessary components for machine operation are displayed within their field of view. These images are visible only once a user’s gaze is in the correct position so as not to distract the driver or impair visibility outside of the vehicle.
Currently, the AR machine controls appear when an operator looks just below the cab’s windshield at the area where a physical dashboard would be in a traditional machine. Johnson says this is due to the limitations of the AR hardware’s field of view. As it exists today, the majority of AR hardware only allows a 30-degree field of view. Ideally, he says it needs to get to an 85-degree field of view in order to allow the computer-generated images to appear in operators’ peripheral site lines. Doing that would ensure visibility out to the job site is not obstructed and operator comfort is optimized. He notes there is some hardware coming out soon which will support a 50-degree field of view.
For its proof of concept, Caterpillar chose to use the Microsoft HoloLens which Johnson says was due to the fact it has a robust surface mapping system. This helps provide better imagery and an easier design interface. He says other technologies, such as an AR headset using a standard cell phone camera, would require vision tracking and processing of the camera image to lock the AR content into the 3D space. To effectively use a standard cell phone camera, it would be necessary to have a vision target, such as a large sticker image, which the Microsoft HoloLens technology does not require. Additionally, he says constantly running a cell phone camera and processing the imagery would be a large drain on the battery and generate a lot of heat, neither of which are conducive to the goal of helping increase operator productivity. 
Caterpillar's AR system creates a computer-generated image of the various buttons and displays an operator will need for machine operation.Caterpillar Inc.
He notes that the next generation of Caterpillar machines will have an electronic on-board data network capable of supporting the data flow from the AR headset to the machine.
Another benefit of the HoloLens technology is that it includes a gaze interface which allows machine functions seen with the AR glasses to be selected via a joystick button. This enables operators to keep their hand on a single control interface at all times, helping improve productivity by ensuring they are always at the ready to select a machine function.
Johnson notes that by having computer-generated images of machine displays and controls in place of physical ones, it is much easier to customize the interface. For instance, the current iteration of the AR system generates a 7-in. (17.8 cm) backup camera display; but if suddenly a 21-in. (53.3 cm) or even 30-in. (76.2 cm) display was wanted, that change could quickly be made within the software to have a larger display generated for the user. That new display could then also be moved about within an operator’s field of view to the most desired location without worrying about where and how to mount it, as would be the case with a physical display. “It’s really about [building] an interface that allows the operators to make it customizable to their needs, what they want it to be.
“In a typical system, when changing the user interface, you’re talking about months to make radical changes,” says Johnson. “With this augmented reality interface, I’m talking days to weeks. I can make radical changes in the interface and we can go out and test it and get feedback.”
In addition to being customizable to individual operators, the AR system is also capable of remembering user settings and preferences, shortening machine startup time. “There are many people working on how do you communicate ‘better operating practices’ to operators for coaching,” says Johnson. “AR is a new, more intuitive method to communicate information, and I personally believe that includes coaching information.”
The first phase of this AR research project for Caterpillar was implementing the system on a standalone cab. Next year it will be put on a functional machine. “Once we put this on a machine, there’s going to be a lot of testing,” concludes Johnson.
