From ultrasound to ultra rugged

Operator engagement has shifted dramatically in the past five years with the continual advancement of smart phone and mobile device interfaces. The automotive industry is typically the first to implement new ways for vehicle operators to interact with on-board electronics and features, and then the technology transitions to the more demanding on-highway market and eventually to the rugged off-highway applications. In rare instances, inspiration and technological adaptations come from seemingly unrelated industries such as the medical field.

The multi-gesture interfaces of today’s mobile devices—which have evolved from keypads to touchscreens with pinch-zoom and touch-point dependent commands—are reminiscent of the various sequence codes (commonly referred to as “combos”) for video game controllers to activate a desired result. This is the next level of human-machine interaction. The technology is quickly spreading in popularity for electronic equipment controls, and on July 18, 2013, Grayhill Inc. (company information, 10056433), LaGrange, IL, launched its multi-touch human interface device (MT-HID) for its Instinct Touch Technology control system that can be used in any application where instinctive and intuitive finger movements can be used to initiate commands.

The software

Instinct Touch Technology combines Grayhill’s Instinct Gesture Recognition Library (product information, 10927023) with its own HID to track movements on the device’s touch pad with up to five touch points at one time. The software tool converts sensor data into gestures to give any device functionality similar to that of an iPhone.

The Instinct software allows its customers to customize their own unique library of gestures. “There are no standard settings because each application is different. In our library you can fine tune the recognition process and adjust parameters, such as the amount of time a position should be held before it is recognized as a gesture,” says Rob Kerner, Research and Development Manager, Grayhill. The library is split into two different types of gestures. Event gestures refer to those that occur without a direction or magnitude, such as taps and tap-and-holds. Tracking gestures, on the other hand, allow an operator to manipulate a level, rotate or zoom.

Tracking gestures can be completed in 2D or 3D mode. In 2D mode the software gives you X and Y axes, rotation and scale. “If, for example, you are using one finger and move it along the interface surface, it will give you the X and Y data and how much movement is occurring. It’s up to the programmer to qualify the received data based on the number of touches, so you can customize it for the application,” Kerner says.

In the 3D mode, the software has the X, Y and Z axes, and is able to rotate around each axis, but Grayhill does not see the 3D mode as a benefit to the off-road operator. “That mode is more geared toward the medical field where they may want to rotate a brain scan and manipulate it in a 3D space,” says Kerner. “2D mode, however, could be used with navigation, interacting with the ISO bus monitor, or for reviewing a site survey data map.”

The hardware

Grayhill’s Series T2 Multi-Touch Control Wheel (MTCW, product information, 10963279) can interpret several different finger gestures for machine command and image manipulation on its 2.75-inch diameter surface. The single unit system features the combined functionality of a multi-touch touchpad, a virtual 4 to 256 position rotary encoder which reads the operator’s finger motion when run around the circumference of the wheel, a virtual four-position joystick and a proximity sensor.

“This product came from a need we saw in the medical industry. Currently ultrasound machines have a trackball-type device where bacteria are likely to collect because you can’t clean underneath it. This is a solution that is fully sealed, a flat surface that can be easily cleaned.

“We wanted to start the technology development in the medical industry because of how controlled the environment is,” explains Kerner. “But we can see a path to get the technology to off-road and to have multi-touch technology in the cab,” says Jason Kandik, Marketing Manager, Grayhill. Also included in the Instinct Touch Technology suite of products is the Multi-Touch Ring Encoder (MTRE) which Grayhill brought to bauma 2013 inMunich,Germany. “When we showed the interface technology at bauma, several people confirmed that they too could see applications for its use in enclosed cabs,” Kandik says.

With an increase in device functionality comes the opportunity to consolidate operator control panels into simpler displays and organizations for a sleeker, cleaner interface for the operator with plenty else to monitor. The simpler design with fewer components also brings cost savings and easier panel design and system integration for a design engineer.

“Control simplification is a trend we’re seeing in a lot of our environments, and I think that’s why we’ve had success with our Vehicle Display Controller, as it controls display rotation and directional control in one device. Operators need space to rest their arms; you can’t have every surface covered in a button or switch,” Kandik says.

Another way to simplify control design is to hinge its functionality on the context of the vehicle’s operating parameters: How fast is the vehicle traveling? In what mode is the vehicle running? How hot is the engine compartment? “If the machine is traveling at 40 mph cruising speed, the same button could do something completely different than when the machine is traveling at 4 mph on a job site,” Kerner confirms.

The path ahead

Currently Grayhill is working toward a better solution for agriculture and construction equipment operators with gloved hands, a common issue with the sensitivity setting with touch-based interfaces. “A touch surface is nothing more than a series of proximity sensors which you interpolate to find position. The sensors are tuned to a sensitivity distance comparable to the level of the surface where contact is made. The technology’s sensitivity distance can be set beyond the surface so the glove is taken into consideration, but as soon as the operator removes the glove, the sensitivity is off again,” explains Kerner. “What we are considering is combining the proximity sensor technology with a pressure sensor so the system feels the physical contact and can search for the position to solve the problem with gloved hands.”

Another way to enhance the interaction with advanced controls for off-road operators is to program in haptic feedback for certain touches, functions and warnings. The haptic feedback could present the sensation of pushing a button, even on a flat surface. “You can vibrate a surface in such a way that it gives the impression that there are detents in the buttons,” Kerner confirms.

 “Another idea we’re looking into is to put touch surfaces on traditional controls like handles and joystick grips. The operator is already physically touching the surface to interact with the control, so there wouldn’t be additional moving parts to consider. That potentially solves the problem with touch precision during the strong shocks and vibrations operators experience in off-road vehicle cabs,” Kerner says.

Grayhill sees agriculture as its best introductory industry for the technology once it is ready for an off-road application. “The cabs are well isolated and the automation technology for hands free steering is already implemented and accepted,” says Kandik.

The greatest hurdle will be operator acceptance, as many operators are used to resistance feedback when they switch gears, flip a switch or push a button. The physical indicator has become an expectation in order for the operator to be confident that an action was completed. It is for that same reason that electronic joystick control manufacturers currently build in resistance to give the operators a similar sensation to mechanical systems while still being able to offer increased system functionality and intelligent system efficiencies.

Conversely, as a new generation of equipment operators filter into the industry with a comfort and familiarity with video game controls, the need for haptic feedback will be less necessary for an operator to feel comfortable with electronic control technology.