irtual reality, or VR, made its debut during NASA and Department of Defense experiments in the 1970s, but recent science-fiction stories, including The Matrix, have taken the concept one step further: simulations so realistic you don't know you're in them. Such things are indeed possible, although the technical challenges are formidable. The five senses of the human body actually break down into a number of sub-senses, each of which has to be fooled in a different way. Follow me, and I'll show you how.
Strictly speaking, the earliest form of virtual reality was auditory, and first appeared in 1960 when Americans began broadcasting music in stereo. For the first time ever, an ordinary person could slip on a pair of headphones and feel literally surrounded by the music, as if they were sitting in the audience or even in the midst of the band itself. Fooling the ear in this way is not difficult; we hear in stereo, with a little organic microphone inside each of our ears. Since the speed of sound is finite and our ears are not both on the same side of our heads, sound waves take longer to reach one ear than the other, and our brains decode this information to determine the direction each sound must have come from. Pretty cool, eh? Sounds also grow fainter as the source becomes more distant, a fact on which experimental stereo musicians were quick to capitalize, creating music and other sounds that seemed to fly around the room and even right through the head of the of the listener. Today, virtual-reality games like Half-Life use the same principle to create an immersive sound environment through the speakers (or headphones) of your PC.
Fooling the eye is slightly more involved. We see in stereo as well as hearing in it, and in fact the three-dimensional world we perceive around us is really a kind of VR simulation running in our heads, reconstructed from the slightly different images reaching the right and left eye. Looking at a single flat TV screen does not reproduce this effect at all; this is why VR headsets place a separate screen in front of each eye, showing slightly different views of the same scene. But this introduces another problem, which is evident if you lean forward to inspect these words closely on your computer screen. See all those dots, the pixels? Any successful VR is going to have to exceed the resolution of your eye, so that the colors show up as smooth shapes and textures rather than just a bunch of dots. You can see this in any old-fashioned paper photograph: the chemical grains that record the color are microscopic in size and cannot be seen without a magnifying glass. The VR displays would also need to reproduce all the colors of the real world and to update the scenery fast enough that the eye would not perceive any flickering, and without the sort of smearing and ghosting you see sometimes on your laptop monitor and even on movie screens.
The promise of cyberpunk comes to pass
Unfortunately, during the 1990s, as computers became more powerful, VR aficionados began to notice a new problem, especially in simulations where the user's point of view was changing rapidly. The problem? Virtual reality makes people sick. This is not too surprising when you consider that boats and airplanes and spaceships also make people sick. Why? Because our inner ear provides a strong sense of balance and motion, which allows you (for example) to navigate through a familiar room with your eyes closed, without bumping into anything and without falling down. Under normal circumstances, you generally know where you are, and which way is up. This is definitely a good thing—without this sense you'd have a tough time in the world—but when the inner ear (known to scientists as the vestibular system) tells a different story than the eyes, the brain doesn't know which sense to believe and becomes confused. That sick-to-your-stomach feeling is like pain: a generalized warning from your brain to quit whatever you're doing, to pause and get your equilibrium back. This is probably a good idea if you're climbing a tree or running across the open prairie, but in a bouncing aircraft or VR simulation it simply causes pointless suffering.
Amusingly, NASA studies have shown that even fish get motion sickness and will sometimes barf if you hold a swaying photograph outside their tank. Do your fish a favor, though, by not trying this at home.
In the most elaborate virtual realities, like military flight simulators, the problem is partly addressed with an elaborate sort of rocking chair that swings side to side as well as back and forth. This device can reproduce certain kinds of swaying motions or tilt the user backward to simulate a sense of acceleration. Some very expensive simulators can even rise and fall a few meters, to reproduce some effects of flight turbulence and tricky maneuvering. But these half measures are no substitute for the real thing, and the pilots who use them still often complain of "cyber-sickness." To solve this problem, the Defense Department is turning, in limited experiments, to direct stimulation of the nerves of the inner ear, using electromagnetic waves from a special headset. This may sound dangerous, but it's not really so different from the way the nerves are stimulated naturally or the way our eyes respond to light. Anyway, systems like this can completely invert the problem of cyber-sickness by creating a very distinct sense of motion or rotation when the eyes know perfectly well that the body is not moving. Keep your barf bag handy, yeah. But when the eyes and ears and vestibular system are finally all giving the same information to the brain, the VR illusion becomes eerily effective.
Easy enough so far? Good, because the sense of touch is a lot more difficult to handle. Our sense of touch is really about half a dozen different senses. Even the vestibular system is often lumped in with it, along with separate systems for sensing hot, cold, extreme cold and pressure. And pain, which is itself a complex phenomenon we don't fully understand. And there is also proporesis, the body's own sensation of the position of its limbs.
The sensation of hot and cold can be provided with probes on the skin, and there are "haptic" or touch interfaces with can reproduce sensations like softness and hardness and to some extent even texture. However, these haptic sensations are not applied to the hands or body directly, but instead to a sort of mechanized wand, which can be tapped or dragged on virtual surfaces to provide the appropriate sensations. Imagine running around the world, tapping on things with a wooden spoon, and you get a pretty good idea what this feels like. To provide the same sensations directly to the skin would require some very complicated technology, almost to the level of Star Trek's holodeck. Other sensations, like proporesis, cannot be mimicked through physical means at all. Instead, you've got to get in and stimulate the nerves directly, as with the vestibular system. And if you've got to do that anyway—feed complex signals directly into the nervous system—then it's probably simpler to do the whole VR thing Matrix style, and simply jam a big probe directly into the subject's brain, with a mesh of fine wires spreading out into the visual cortex, the auditory cortex, the centers of movement and sensation. It's the cyberpunk dream: wire the brain!
Some people may recoil in horror at this idea, but in fact we're already in an age when handicapped people have wires connected to certain critical nerves. The most striking example is the cochlear implant, an artificial device that allows deaf people to hear, but there has even been limited, experimental success in getting blind people to see using a grid of electrodes attached to the visual cortex. For non-handicapped people to adopt similar technologies may require a pretty big leap of faith, but you never know: the lure of VR, of enhanced reality, of computer-mediated supersenses augmenting or replacing your own, may be difficult to resist.
Real or not, the Matrix still matters
Believe it or not, there is a step that goes beyond even this: giving up the body entirely, giving up even the brain itself, and simply uploading your consciousness into a computer as a piece of software. In science fiction, this idea dates back at least as far as Joe Haldeman's novel World Apart (1983), William Gibson's Neuromancer (1984) and the 1985 debut of Max Headroom: 20 Minutes Into the Future on Britain's BBC 4 network. It's now a standard science-fictional trope—almost a cliche—which just goes to show that people find the idea interesting, and even appealing on some level. Of course, giving up the brain is the final, irrevocable step; at that point you're stuck in VR for good, unless you want to download or copy yourself into a robot body. As in The Matrix, whole generations of people could be born and live and die without ever realizing the true nature of the world they inhabited. Which really does lead to a lot of slippery questions about the nature of reality and whether the nature of reality matters more than our subjective experience of it.
But while science fiction constantly warns us that machine intelligences must someday rise up to destroy their creators, MIT scientist Hans Moravec answers this speculation with with amusement. In countless enthusiastic interviews, he points out that the hyperintelligent machines of the future could easily create virtual environments, and even virtual humans to inhabit them, without humans being any the wiser. And since the capacity of these computers will be nearly infinite, and since the future itself is infinite, where the past (especially the recorded past) is finite, he notes that far more time will be spent by humans in these virtual environments than in the real ones from which we sprang. Historical reenactments of the real world will therefore outnumber the one and only real real world by a considerable margin.
So hang onto your hats, my friends, because if that observation is true, then the odds indicate —overwhelmingly!—that you and I and everyone we know are living in The Matrix right now, and always have been, and always will be. If we were never physical beings to begin with, there is no place for us to escape to! Still, if you glance out the window, or run your hand along the surface of your desk, you do kind of have to admire the workmanship. Whether it's "real" or not pales in comparison to the much more important question: Are we having a good time?
Wil McCarthy is a rocket guidance engineer, robot designer, science-fiction author and occasional aquanaut. He has contributed to three interplanetary spacecraft, five communication and weather satellites, a line of landmine-clearing robots and some other "really cool stuff" he can't tell us about. His short writings have graced the pages of Analog, Asimov's, Wired, Nature and other major publications, and his book-length works include the New York Times notable Bloom, The Collapsium and most recently The Wellstone and a related nonfiction book, Hacking Matter.
