Of Pixels and Voxels and nervous messes

In 1995 I was employed in exhibit development at Discovery World, a museum of science, economics and technology. My job led to my involvement in several vastly different technologies and scientific fields, from hydraulics and lasers to electricity and health. One particularly interesting piece of technology with which I became involved was a “Haptic” interface, called “The Phantom.” Haptic, from the Greek, means touch, and the Phantom was intended to provide the user with virtual sense of touch via a single finger tip.

It looked much like a miniature architect’s lamp, an arm with several degrees of freedom, terminating in a thimble-like cup at the end, which, in turn, was attached to an armature governed by priceless little motors and sensors. The entire design intended to allow the user to move their finger as freely in space as any of their other digits, until they encountered a virtual obstacle. This might be something as simple as a simulated piece of paper, or sandpaper, or perhaps something more complex, a billiard ball, or banana, a wrist, or a wrist with a pulse.

Via the thimble, the controlling computer system could convey texture, viscosity, pressure, vibrations, movement — the entire range of things we can feel with our fingers, albeit not heat nor cold nor the pin-prick of pain. But one might pluck an invisible guitar string, and feel its harmonics, or palpate the back of a virtual patient.

My group were unsure just what we would have the device simulate, nor how we would allow a visiting public to interact with it, given the inherent fragility of the device (and the largely reckless tendencies of the public). But as this was very new technology, having just been invented a year earlier by an MIT grad student, there was a scholarly conference about it, held near MIT, in suburban Boston, and I was to attend. In fact, when I received my conference credentials I was pleasantly bemused to see that I was credited, on MIT stationary, as Doctor Nic Bernstein. Doctor indeed!

Upon arrival at the conference assembly I was greeted by a curious assortment of engineers, scientists, investigators, doctors, physicists. Oh, and a three-star General from the US Army; Defense Advanced Research Projects Agency (DARPA), the same people who invented the Internet. Other than myself, and a geologist from Australia, everyone else there was, in some way or another, in the pocket of this General, something I pointed out during our plenary introductions. At the next meal break, said General sought me out as a dining companion. How, he needed to know, was I not also on his payroll?

There were several obvious implementations of the Phantom being discussed, such as a medical school using them, along with surgical dummies, to help physicians learn proper technique for administering epidermal injections & draws.

The Australian geologist was working with seismic stimulators to probe for deeply buried oil & gas deposits. This being done by amassing the vast amounts of three dimensional data produced by seismic stimulation — essentially carefully calibrated “shakers” attached by outrigger arms to long, low trucks, like massive insects, which would slowly advance along a grid work, shake the ground a bit, raise, advance some more, lower, shake, etc. Once a full grid had been worked, and the data assembled into a three dimensional model, the investigator would probe through the data, feeling his or her way along veins of ore or into voids filled with gas or oil; each substance represented with a different virtual viscosity.

During a field trip to the labs of the Mitsubishi Heavy Industry corporation, a friendly scientist showed me the system they were developing to help orthopaedists feel their way around (“appreciate” in the parlance) the knee joint of a prospective surgical subject, prior to wielding an actual knife.

Here’s how it was done. The patient would receive a scan — PET, CAT, MRI, whichever technology would best image the tissues involved — and the data would be loaded into a computer model. Rather than the pixels (Picture Elements) we think of from the two dimensional world of television or video, or printing, this data were rendered into Voxels, Volumetric Elements. In addition to the X, Y, & Z coordinates of a datum, there was also information on the density of the matter, rendered to the “viewer” as viscosity or resistance. A doctor could thus feel around the back side of a kneecap, for example, to appreciate the condition of the soft tissues there (if any remained), such as cartilage or muscle, each rendered in a different haptic manner.

It was fascinating. This was 1995 remember, long before these sort of things were depicted as routine in movies and on telly.