Humankind 2.0

a book in progress...
Meditations on the future of technology and society...
...to be published in China in 2016

These are raw notes taken during and after conversations between piero scaruffi and Jinxia Niu of Shezhang Magazine (Hangzhou, China). Jinxia will publish the full interviews in Chinese in her magazine. I thought of posting on my website the English notes that, while incomplete, contain most of the ideas that we discussed.
(Copyright © 2016 Piero Scaruffi | Terms of use )

Back to the Table of Contents

Virtual Reality: History, Trends and Future

(See also the slide presentation)

Narnia: Can we start with a short history of Virtual Reality?

piero:

The story of virtual reality dates back to the 1960s. The CIA and two military agencies funded the first virtual-reality system, built in 1969 at the University of Utah by one of the great pioneers of computer graphics, Ivan Sutherland. He connected a head-mounted display to a computer and then programmed the computer to send images to the display. It took almost 20 years to find a practical applications. Again, it was funded by the military: in 1986 Thomas Furness, working at an airforce base, designed a simulation system that allowed a pilot to fly a plane through a computer-simulated landscape by moving his head and his hand. At approximately the same time the NASA center in Silicon Valley funded the project by Michael McGreevy for "virtual planetary exploration and the more general system designed by Scott Fisher that incorporated the first "dataglove". In 1985 Jaron Lanier established VPL Research at his house in Palo Alto, the first company to sell Virtual Reality products. These pioneers were visionaries but they worked on hardware that was not for the masses. Real progress came from the world of computer games. A MUD (Multi-User Dungeon) is a computer game played by many simultaneous users on different computers, all of them connected to the same virtual world. There were predecessors but the term was coined at Essex University by a student, Roy Trubshaw, and his game became popular in 1980 when the university joined the Internet. For a while Britain was the leader in this field with games such as "MIST" (1986) and "AberMUD" (1989), followed by Denmark with "DikuMUD" (1991); but the virtual worlds were text-based, not graphical. In 1986 Lucasfilm launched "Habitat", a social virtual world in which each user was represented by an "avatar"; and this was a graphical MUD. In 1990 "GemStone III" was launched in Missouri, a graphical MUD that would spread on CompuServe, Prodigy and America OnLine (AOL). The world-wide web and the browser (introduced in 1991) helped turn this kind of game into a serious sociological project. In 1992 the University of Illinois at Chicago demonstrated the CAVE, a virtual-reality environment whose walls and floor were basically just very large screens, and many universities would install a CAVE system in the following decade. In 1994 Ron Britvich in southern California created WebWorld, later renamed AlphaWorld, in which people could communicate, travel and build; and in Silicon Valley a former hippy named Bruce Damer formed the Contact Consortium that in 1996 launched 3D virtual-reality environments such as a virtual town and a virtual university. At that time (about 1996) a new kind of "massively multiplayer game" appeared, the MMORPG. The genre was invented in 1996 by Korean game "Baramue Nara" ("Baram" in the USA), followed by "Meridian 59" (1996), developed by brothers Andrew and Chris Kirmse in Virginia, and "Ultima Online" in 1997, developed by Electronic Arts' game designer Richard Garriott who also coined the term MMORPG. The emphasis here is on having many players at the same time. EVE Online, introduced in 2003 by Simon & Schuster Interactive, is a MMORPG that can have tens of thousands of players at the same time). Today the most famous MMORPG is "World of Warcraft", launched in 2004 by Blizzard Entertainment. The technology of MMORPG was focused on letting many players play at the same time, but indirectly it laid the foundations for the software of virtual-reality. So the history of virtual reality is another funny history, that begins with military applications and then shifts into the subculture of videogames.

Narnia: There are announcements every week in VR such as near-eye light-field technology (for example from Nvidia and Magic Leap), Facebook's acquisition of Oculus, Apple's acquisition of Faceshift, Google Cardboard... What does VR mean for ordinary people?

piero:

Virtual Reality is definitely here to stay. Finally, after 30 years of "flops", the devices for virtual reality are available at reasonable prices. Sega released the Sega VR in 1993, Nintendo's Virtual Boy was introduced in 1995, etc. In 1995 Future Vision Technologies, a spinoff of the University of Illinois at Urbana-Champaign, developed a head-mounted display for the consumer market, the Stuntmaster, and in the same year a company based in Sacramento marketed the iGlasses goggles. They all had the same problems: a true stereo display made of two high-resolution color LCD screens coupled with motion tracking was too expensive and caused serious motion sickness. They all failed in those days, but Samsung's Gear VR was the #1 high-tech toy of Christmas 2015. Between 1995 and 2015 two things happened: 1. the cost of LCD screens declined (thanks mainly to Hitachi's In Plane Switching design and Samsung's Multi-domain design, so that in 2007 for the first time LCD TV sets surpassed CRT TV sets in worldwide sales) and the cost of 3D motion capture declined (Microsoft Kinect came out in 2010); and 2. movies such as "The Matrix" (1999) popularized the idea of life in a simulated world and therefore inspired a new generation to live inside virtual worlds. (Few people seem to know that "The Matrix" is a Hollywood remake of a German movie made 26 years earlier by the great director Rainer Fassbinder, "World on a Wire").

There is impressive progress in the technology. Near-eye light-field technology creates a much more "real" impression when you are exploring a virtual world. There is so much progress that probably virtual reality is a field that will become obsolete in a few years. In a few years the hardware will have changed completely. We will look back at the Oculus Rift and the Gear VR the same way that today we laugh at the tape deck of the 1990s. In fact, that is precisely Google's strategy: Google Cardboard is a step towards making your smartphone the virtual-reality device of choice, just like the smartphone has become the photograph-taking device of choice for most people. I am not sure that the smartphone (as it is today) is the right vehicle for virtual reality, but i think that in five years the hardware for virtual reality will be much simpler. I suspect that Apple will be, as usual, the company that makes everybody look obsolete: in 2013 they acquired the Israeli company PrimeSense, most famous for designing the motion-detection technology of Microsoft's Kinect in 2010; in 2015 they acquired the German company Metaio, a Volkswagen spin-off specializing in augmented reality; in 2015 they acquired the Swiss company FaceShift, a spinoff from Lausanne's Polytechnique Federale, whose technology captures the user's facial expressions in real time and creates an animated avatar; and in 2016 they hired Doug Bowman, a world expert of 3D graphics.

Since 2014/15 the industry split in two fields. One field is focused on gaming. Entertainment has been the first mass-market for virtual reality. The other applications (mainly marketing and simulation) generate much less in revenues. The research firm TrendForce predicts that in 2016 the industry will sell 14 million virtual-reality devices and mostly for gaming. Facebook acquired Oculus for this kind of virtual reality. Sony's PlayStation VR (Project Morpheus) will plug into a PlayStation game console. The other field disagrees because it is more interested in office productivity, i.e. in selling virtual reality to the office, not to the consumer. This field is more interested in "augmented reality" and is best represented by Microsoft's Hololens. This device is an evolution of the Windows computers that have been sitting on my desk since 1982: it "sits" directly in front of my eyes and its user interface replaces the mouse with my gaze and the mouse click with the motion of my finger. The desktop computer allowed me to walk around the "dataverse" (my personal data and remote data) and Microsoft's productivity tools allowed me to work on those data using keyboard and mouse. This new type of computer allows me to walk around three-dimensional virtual objects and to work on them using gaze, gesture and voice. Oculus' Rift needs to be plugged into a host computer and, to do the same things, it should be supplemented with gesture recognition (like Leap Motion or Kinect) and stereoscopic cameras. It was not designed for the office. It was designed for the home, and more specifically for a kid's bedroom.

The term "augmented reality" was coined in 1990 at Boeing in Seattle by Tom Caudell. The event that changed the course of this field was probably the development in 1999 of an open-source platform called ARToolKit at University of Washington in Seattle by two foreigners, Hirokazu Kato from Japan) and Mark Billinghurst from New Zealand. This platform was ported to mobile phones already in 2005. In 2008 the general public saw augmented reality in television for the first time: during a CNN show the public a three-dimensional virtual image of reporter Jessica Yellin "magically" materialized in the CNN studio.

Meta, founded in 2012 in Redwood City by Meron Gribetz, introduced the first augmented reality system in 2014, see-through glasses that allowed wearers to move and manipulate 3D content using hand gestures. In 2017 it introduced an augmented-reality user interface for desktop computers. In 2017 Mira, founded in Los Angeles by three students from University of Southern California in Los Angeles, demonstrated a cheap, wireless, see-through, augmented-reality headset powered by a smartphone.

Robert Xiao at Carnegie Mellon University is working on "Desktopography", a device that one can hang from the ceiling like a light bulb to project images on a surface and then let the user move them around the surface. For example, you can project or map onto a desk and then you can manipulate it with your fingers.

Augmented reality can also be applied in the factory: in 2014 a Los Angeles company, Daqri, launched its Smart Helmet, an Android-powered wearable that creates augmented reality for industrial and construction workers. For example one can show visually how to do something, instead of simply providing written or verbal instructions. In 2015 they acquired ARToolworks. They also acquired Melon, a Los Angeles startup that developed a wearable that reads brain waves.

Augmented Reality can be more disruptive and pervasive than Virtual Reality because AR turns the whole landscape into a blank canvas: you can place objects anywhere and then your friends can find them. The world becomes a wall on which you can write and display what you want and you can decide who will see it. AR became popular when PokemonGo placed creatures in the urban landscape that only people with the AR app could see (and capture). In 2016 Microsoft placed a digital sculpture in the Bellevue Arts Museum of Seattle. Only people equipped with Hololens could see it: they could see a flock of words flying in the room. This was a powerful demonstration of what ordinary people can do with AR. If we augment a chat application like Wechat or Whatsapp or Facebook, you will be able to post "virtual" messages or flowers in front of someone's door or in her street. Nobody will see them unless they are wearing the AR glasses and you gave permission to them. This will be a new way to communicate, not email and not chat but something else altogether. Apple's iPhones of 2017 came with AR capabilities.

Augmented reality has the potential to revolutionize the world the way the World-wide Web did. It can create a visual Wikipedia for every single place and object in the world, a geographically distributed web of webs. You look at a place and you can see the history, science, art, etc related to it, and every person can add more information about it. You look at an appliance and you can see the operating manual, the warranty, the technology behind it, famous movie scenes with that kind of appliance, the biography of the person who invented that kind of appliance, similar appliances on the market and so on. Wherever you look, you will open a universe of information. Augmented reality glasses will turn our flat world of information into a multidimensional universe of universes.

In 2016 Intel introduced a 3D camera called RealSense ZR300 that can scan things and create a digital 3D copy, as well as react to gestures like Microsoft's Kinect does.

Intel's Project Alloy is one of the most interesting projects underway. It will be the first stand-alone headset: no PC/smartphone required. Facebook Oculus and HTC Vive must be tethered to a computer. Samsung Gear VR and Google Cardboard require a smartphone. Alloy will be the first one that comes equipped with its own computer. Intel is partnering with Microsoft like it did in the old days of the "Wintel" alliance for the PC and is opening up the specifications of the project to other manufacturers, just like they did in the 1980s for the PC.

Augmented reality is not only for the office. In fact, its first success story has been the apps that let you "try" clothes or makeup (lipsticks, eye shadows, gels and creams). ModiFace, yet another offshoot of the University of Toronto, where a lot of the influential research on machine learning has been done, shows you how the makeup will look on your face. You can move your head, smile and wink. Its founder, Parham Aarabi, used a technology that he developed at Stanford for military recognition of lip movements. Image Metrics, founded in 2000 in Manchester, became famous in 2007 when it created a 3D hologram (a visual clone) of a deceased actor that performed "live" on a stage. Apple acquired, Faceshift, which is a spinoff project from Lausanne's Polytechnique Federale, another company in this business, of capturing the user's facial expressions in real time and creating an animated avatar of the user.

Augmented reality can also be used for fun, like Blippar, a smartphone app for "visual discovery". It mixes image recognition and augmented reality. You point your smartphone's camera at an object and the phone displays something about that object, like a visual and almost three-dimensional encyclopedia.

Augmented Reality will also be very important in hospitals. A surgeon needs to see what the machines are seeing, and needs to see all the relevant information about the patient. For example, in 2017 Scopis Medical in Germany and Cambridge Consultants in Boston announced systems based on the same principle. First the doctors can use MRI and CT scans to build up three-dimensional images of the organs of the patient. Then the surgeon can use Microsoft Hololens to "see" inside the patient during the operation. In April 2016 a London surgeon, Shafi Ahmed, broadcast live a 360-degree view of an operation. Students equipped with Samsung Gear or Google Cardboard were able to immerse themselves in the operation as if they were physically in the same room. In 2017 the French firm Revinax supplied the augmented-reality system (based on Microsoft Hololens) that aided in a real live surgery at the hospital of Montpellier: the surgeon was able to learn by watching the same kind of operation as performed by another surgeon. Touch Surgery, founded by two former surgeons in 2012 in London, develops a similar system.

At the end of the day, however, you always need a Hollywood movie or a videogame to make a technology popular, to turn it into a commodity. What truly made augmented reality popular was Pokemon Go, a mobile game introduced in 2016 by Niantic (a startup incubated by Google in 2010) and Nintendo. In fact, we forget that the top-selling toy of 2010 was Eyepet for the Sony Playstation, introduced just one year earlier. That toy came with a projection camera that projected a virtual pet capable of interacting with the children. It wasn't called "augmented reality" in those days but that's what it was.

I see virtual-reality devices like the Oculus Rift as the continuation of 3D television and movies. I don't think those were success stories. I don't think that 3D glasses or virtual-reality headsets are "friendly" user interfaces. They are very unfriendly. Most of the online comments are about motion sickness, not about how great the experience is. I think that only fans of games are willing to be tortured by these devices for hours and hours. Everybody else probably prefers to watch a soccer game on an old black and white tvset rather than on a 3D tvset that forces you to wear annoying glasses. These virtual-reeality headsets are also quite antisocial, an isolated, individual experience. Black and white television was a family experience. Maybe not everybody wants the "family experience" but many people prefer a communal experience of watching something together, as opposed to each person being immersed in its own virtual reality.

Virtual reality is much more than just games. The experience of "watching" content can be completely different. We live in a world in which the "attention span" has been constantly reduced. The younger generations are proud that they are "multi-tasking" all the time. And the older generations complain that this multi-tasking and this short attention span are creating the "shallows" (the title of an influential book by Nicholas Carr), people with only a shallow perception of the world, people with a very superficial understanding of the issues, people who cannot read a poem or a philosophical essay because or anything else that requires concentration. Virtual reality can be the perfect antidote to this trend because virtual reality is a medium that "forces" absolute focus on the content. Virtual Reality opens a new dimension of human-machine interaction, one in which the machine is not a distraction but an aid to concentrate. We had interaction based on keyboards, mouse, voice and touch. Virtual reality will allow for interaction based on everything that we do: gazes, gestures, body movement, and some day maybe even breathing and thinking. We used to say "i watched a movie" but in the future the verb will not be "watch" but "experience".

Narnia: what are the impediments to broader adoption?

piero:

Content. There is very little that you can do with virtual reality. We have seen the first wave of 360-degree cameras, stereoscopic 180-degree 3D cameras, etc like Jaunt VR, Matterport, Lucid VR, Google's Jump, Intel's RealSense camera, but we still don't have the GoPro of virtual reality. When we get the equivalent of a GoPro, contents that you can experience in virtual reality will skyrocket. Possibly the most important news of 2015 was that the New York Times distributed one million Google cardboards to the subscribers of its digital edition so that they can experience stories in virtual reality via a free smartphone app. One reason that content is still scarce is, as usual, the lack of a standard. If you buy one VR system and then change your mind later, you need re-purchase all the content that you bought. In 2016 GoPro has indeed come out with an "end-to-end" platform for virtual reality: the six-camera Omni VR coupled with the LiveVR wireless streaming tool and the GoPro VR video channel, plus partnerships with 100+ developers.

Then there is still the problem of motion sickness. They say that faster processors will solve the problem, because movement and reaction will be better synchronized. Qualcomm's Snapdragon 820 for Android smartphones (2016) is probably the first of a new generation of processors that target virtual reality apps on smartphones. Project Tango is a joint effort by Google, Intel, Qualcomm and Infineon to create Kinect-like sensors for smartphones that will be much better at tracking the position in space (the project's manager is Johnny Lee, whom Google stole from Microsoft where he was the leader of the Kinect team and most of the technology comes from Google's acquisition of Motorola). Google is funding an "app incubator" for Tango devices.

What is missing is the ability to use "augmented reality" at home, in the office, in the store. Tango will solve that problem because it will provide a "location-sensing system". The GPS is great for telling you where you are in a city, but not where you are in a building, and not where you are in a room. The GPS knows that you are at 120 Main St, but not that you are sitting on the couch of the dentist's lobby. When you are inside a building, your GPS stops being useful. Tango uses different technologies to understand where you are and to follow your movements. The goal is to achieve an accuracy of one centimeter, i.e. being able to tell whether you are leaning against the wall, sitting on the chair by the wall or standing in front of the desk. The application is not only "augmented reality". For example, Tango will be able to measure the size of an object simply by calculating the distance between two places where you press a button: no more measuring tapes!

[Narnia inserts here an Editor's Note: in 2016 Lenovo announced that it would make the first Project Tango smartphone]

Narnia: but this is a new way to interact with the machine, isn't it?

piero:

Actually, one of the benefits of this boom of Virtual Reality is that we are witnessing the first major revolution in human-machine interfaces since the iPhone. Microsoft HoloLens and Israeli-based Lumus offer see-through displays for augmented reality. Survios, an offshoot of the University of Southern California's Mixed Reality Lab, offers an immersive headset that is also capable of tracking the user's physical movements (basically Oculus + Kinect). Various versions of "optical touch" (that project a screen on an object and then track the movement of the fingers on the projection) are turning every object into an input device. Zhen Liu, a Chinese-born graduate from Harbin's Institute of Technology, has a startup in Singapore that makes the Touchjet Pond, a system that turns every surface (a table, a wall, a floor) into a touch-screen. Israeli-based Lumio turns any surface into a keyboard. Measuring neural activity has become more affordable and is giving us wearables that can determine one's state of mind, from South Korea's SOSO (tested in schools to determine children's concentration) to Israel's ElMindA.

This new human-machine interface is actually hindered by the current device, the bulky headset. The headset is not the solution but the problem. In 2017 a Seattle-based startup, Innovega (whose main investor was Tencent), introduced eMacula, a pair of regular-size glasses and contact lenses that incorporate a screen for virtual reality. eMacula makes you think of another possible application of virtual and augmented reality: helping people read. If they can project games on your contact lenses, maybe they can also project a page that you hold in your hand. It should actually be easier. It may help people with some eye diseases better than magnifying lenses. Sony, Samsung and Google have filed patents for contact lenses to be used in AR/VR systems.

Narnia: What other applications besides videogames?

piero:

There are already some practical applications of VR to marketing. London's Framestore "transports" people into Marriott Hotels and made a famous app for Volvo that is probably the future of car marketing: you can "test drive" the car from your living room. YourCampus360/YouVisit (Florida, 2008) creates "walking guided tours" of hotels, fashion shows, etc. Outlyer VR (Los Angeles, 2015) aims to create interactive advertisements for any product. In particular, VR has been used to market homes by companies such as Matterport (Mountain View, 2010), VR Global (New York, 2014) and Arch Virtual (Wisconsin, 2014). They use VR to create virtual copies of real-world spaces; virtual environments that prospective buyers can experienced and changed.

In 2014 British fashion multinational TopShop produced a 360-degree view of a yearly fashion. In 2015 Jaunt produced a 360-degree video recording of a Paul McCartney concert in San Francisco. In both cases we can download the app, using Oculus, we can attend the show as if it was happening now and here. I just tried to purchase tickets for a classical music concert and it's all sold out. Of course, i can just listen to the music from a CD: why do we want to go to a symphony hall to listen to the same music? The experience is different. The "live" experience is different from being at home. Unfortunately, there is a limited number of tickets available: only 1,000 people can enjoy that live experience. Ditto for a soccer game. These are three different experiences: 1. watching the game hours or days after it took place, after i already know who won; 2. watching the game live on tv; 3. watching the game live at the stadium. The experience is different. The problem with 3 is that there is a limited number of tickets available. Only 50,000 people can enjoy the experience of the live game. Virtual Reality will allow the symphony hall and the stadium to sell millions of tickets for the exact same seat. The day that we can create virtual reality in real time is still far in the future, but some day it will become possible: 360 degree cameras will photograph in real time what is going on in the stadium, software will recreate it on your computer and the VR "gear" (which hopefully will be more user-friendly than today's head-mounted displays) will "transport" you in the stadium and give you the feeling of being there with all the other spectators.

Another spin-off from the Swiss Federal Institute of Technology, which has now relocated to San Francisco, is called MindMaze, and makes devices that help victims of strokes (or, in general, of brain injuries) to rehabilitate, to learn again how to use their body. This is a case in which virtual reality is used to "re-train" the brain when the brain has "forgotten" how to do something, for example how to move paralyzed limbs. This device is a complex mixture of electroencephalographic sensors (that "read" your mind) and motion-capture sensors. There are many health care applications. For example, San Francisco-based Vivid Vision (founded in 2014 by Manish Gupta, Tuan Tran and James Blaha) is using VR for repairing eye conditions, and Palo Alto-based Moodru (founded in 2013 by Tony Burton) is using VR to control anxiety and phobias.

Narnia: You mentioned that virtual reality forces people to "concentrate" instead of being distracted by the environment. So the ideal application would be something that requires concentration?

piero:

The original application (flight simulation) was definitely a task that requires concentration. Videogames require concentration. Virtual reality is indeed the technology of concentration. In fact, i wouldn't be surprised if someone came out with a virtual reality system that helps Buddhist-style meditation. One could create a virtual world of pure nature or even super-nature (clouds, galaxies, etc) the same way that the new-age music of the Western world in the 1980s was creating an electronic sound evoking other worlds for the purpose of relaxation and meditation. But of course there are many tasks that require "concentration", starting with research and study. Virtual reality is not only a way to reproduce the classroom but also a way to reproduce a classroom without distractions. Obviously augmented reality, that mixes the real world and the virtual world, can still offer plenty of "distractions" but a world whose existence and dynamics depends on the movement of your hands and your eyes is much more likely to focus your attention. This could be the future of "distance learning" but it is also already used in California schools. Stanford has experimented with the system of zSpace, established in 2007 in Sunnyvale and originally funded by the CIA. Alchemy VR in London and especially Immersive VR Education, a spin-off of the Waterford Institute of Technology in Ireland (that has created programs like LectureVR) have experience in applying virtual reality to the educational curriculum. In 2015 Google launched Pioneer Expeditions, a program in which Google employees install (for free) a virtual-reality system in selected schools and teach the teachers how to use it. This system takes the children to places like Maya ruins or Mars, so the lesson becomes more exciting and much more powerful.

Narnia: What is missing from today's virtual reality?

piero:

If we get motion sickness, it means that it is not so real. It doesn't feel real. Personally, i don't feel that it is real because it only simulates two of the senses, the two that our old technologies can handle: sight and hearing. We invented ways to store and manipulate images and sounds. Virtual reality technology is an evolution of those old technologies: Edison's grammophone and Daguerre's daguerreotype. We forgot that we have 5 senses, not 2. We don't have a practical way to store and transmit smell, taste and touch. A boyfriend cannot kiss his girlfriend over Wechat. He can text her a lot of emoticons and email her a romantic song but he cannot send her the feeling of the kiss. And she cannot send him the perfume of her skin and hair. And neither can taste the food that the other is eating. Adrian Cheok founded the Mixed Reality Lab in Singapore to develop techniques that would enable people to broadcast smell, taste and touch over the Internet. Personally, i don't miss taste so much as smell and touch. A natural environment is not the same without the strong perfume of the flowers and the strong feeling that you get from the wind. If virtual reality drops me into a beautiful landscape but i cannot smell its flowers and feel the wind, i am missing 50% of the experience.

In 2016 Emma Yann Zhang, a student at City University London, demonstrated a device that can broadcast a kiss. In 2017 Vasqo announced a device to release scent.

Narnia: You said the progress in VR is rapidly. That's why all the big companies want it. Why can VR post real and fast progress?

piero:

Both the hardware and software of VR are here today. And they are improving very rapidly. VR, like any other technology, can be applied in different fields, and different fields will have different growth rates. People need to remember that what we have now is "three-dimensional 360-degree video" not "true VR". And three-dimensional 360-degree video is already useful in fields like videogames and soon will be useful in fields like journalism and education, but not much more. But it is not "true VR", and it tends to be "passive", not "active". True VR should be active: you "control" the real world, not just "experience" it.

Narnia: How long you think it will take for VR to become something really big, something truly disruptive for our society"? S

piero:

VR is one of many technologies. Each is disruptive in some way. The smartphone changed a lot of things but didn't change the way i play soccer or the way i eat. It changed the way i communicate and organize. In the short term VR will not disrupt much: just videogames and slide presentations. It will improve things like simulation, training and education. It will take a long time before we can create a virtual world that you can smell and taste. In the next 10 years VR will have the same impact that cinema had in its first 10 years. When you go to the movie theater, you get immersed in the movie, right? You "feel" the story. Sometimes you laugh, sometimes you cry. You forget that you are in a movie theater and that there are people around you. VR gives you an even stronger feeling of being "somewhere else". That's already today, but the technology makes you a little sick after a while. In 10 years it will be much more realistic. The software to create worlds exists today so hopefully in 10 years it will be normal to create worlds on social media (Facebook, WeChat, whatever will dominate in 10 years) and share them with friends. Just one click and your avatar enters my world. So in the short term VR will disrupt some things (cinema, journalism, education) but 10 years is not enough to get full VR. It took 26 years for the car industry to introduce the electrical starter (before that, the driver had to start the car with a crank, and that's why drivers were typically strong men). It took 37 years to add the radio to the car, and 45 years to get air conditioning in the car. The car was disruptive from the beginning, but the full power of car transportation came when the car became comfortable. Then the car really reshaped society, with people moving outside the city center because it was so nice and easy to drive a car. It will take a lot of progress in hardware technology for VR to become truly pervasive in communications.

Narnia: What are the short-term opportunities? Can you give some suggestions to the investors and startups?

piero:

The short-term opportunities are the devices to experience VR (headsets), motion sensors, software for 3D Modeling and 3D display, and videogames. As far as applications go, i think the first fields to benefit will be marketing (eg marketing homes and cars) and business presentations (Microsoft HoloLens). In the 2020 timeframe i expect the transition from passive 3D 360-degree video to "true" active VR in which you "control" the real world, not just "experience" it. This will require a new generation of motion sensors. It will enable applications like: 3D CAD/CAM/Printing, Avatars for health care, Avatars for investment. And total immersion media will become much more appealing: Journalism, Cinema, Training/education/simulation. Then i expect by 2025 we should have brainwave sensors. But we have to be careful not to exaggerate. In 2007 Gartner Group predicted that 80% of internet users will have an avatar in a virtual world like Second Life by... 2011! Today is 2016 and not even 1% of internet users have an avatar in a virtual world. In 2013 kzero.co.uk, that specializes in studies about virtual reality, forecasted 2014 sales of virtual reality headsets "to exceed 200,000". Well, there was no virtual-reality headset available in 2014 for the mass market. The Samsung Gear VR and all its main competitors came out in late 2015. The Gear VR "sold out" the first day in Korea, but later we learned that only 2,000 units were available. The Oculus Rift finally went on sale in January 2016 but at a price more than twice what people expected. Statista.com predicts that in 2016 Samsung will sell 5 million Gear VR. I think Samsung will be happy if it sells 500,000 units. So we have a lot of exaggeration, as usual. But there may be something big coming soon. It has never happened in history that a company is valued at $4.5 billion before it even introduces a product. Magic Leap is worth that much today, and Alibaba is one of the investors in Magic Leap. They must have seen something impressive.

And hopefully venture capitalists know what they are doing. According to Digi-Capital, in the 12 months from April 2015 to March 2016 venture capitalists invested $1.7 billion in virtual reality, $1.2 billion just in the first quarter of 2016. And there were already four unicorns: Magic Leap, Oculus, Blippar, MindMaze.

Narnia: what does virtual reality mean for our social lives? Will it isolate people?

piero:

The story of computers in the last 40 years is a story of social and antisocial technologies that influence each other. You can begin the story with the invention of email in 1972 by Ray Tomlinson. Email allowed people to exchange messages instantaneously and for free. When the personal computer became pervasive in the early 1980s (basically after the introduction of the IBM PC), email started replacing the traditional mail. The handwritten letter is perhaps the most important "technology" and social phenomenon that has disappeared over your lifetime. When thee first Chinese immigrants moved to California, it would take almost a month for their families to learn that they had arrived safely. That was the time that a letter would take to travel across the ocean. Email was revolutionary because it allowed a message to cross the ocean in a few milliseconds. And for free: unlimited messages to unlimited recipients. That was social progress: ordinary people were able to keep in touch much more frequently and much more easily than before. However, the same technology that gave us personal computers with email also gave us videogames and many other forms of digital entertainment that, typically, turned computers into an antisocial medium: the younger generation spent more and more time alone with their digital device, and "multitasking" (homework, videogame, television, email, music, etc). But then the same technology gave us social media like Facebook and Wechat. Now there is too much "social" life on the Internet. Inevitably, it becomes more superficial ("shallow"). I have 5,000 friends on Facebook but how many would help me if i get in financial or health trouble? So computers have always made us more antisocial while they were making us more social; and that's because socializing via computers is not based on mutual benefits but just on chatting; and the only thing you can do is praise your "friends". Facebook has a Like button but not a Dislike button: real friends have the courage to criticize you when you make a mistake, not just praise you all the time. Real friends share experiences, and the most important ones are often the negative ones. If we are stuck together in a cold place with no food, we will probably remember that experience forever. So, again, a computer technology that, in theory, was meant to help people socialize ends up being antisocial: it devalues friendship, it turns friendship into a commodity. Worse: Facebook makes money out of your friendships. Your social life becomes someone else's business model.

Virtual reality is another case in which the antisocial life and the social life collide, except that in this case the technology was born antisocial. In 2003 Linden Lab launched "Second Life", a virtual world accessible via the Internet (it was basically a MMORPG) I had an avatar in Second Life, and my avatar had its own life and friends (who were avatars of other people, people unknown to me). Your avatar can be a very different person from your real self. It was an interesting psychological experiment: what would you like to be if you lived in another world, if you were not watched by your family and friends, if you could forget about your duties and obligations? Where would you travel if you could travel incognito? Second Life could be the future of social media. In the future you will be able to create your own world and live in it alone (a game that children frequently play) or create a world in collaboration with your "friends" (either friends whom you personally know or friends who are just avatars of strangers). I suspect that both uses of virtual reality will provide a deeper experience. Socializing in a virtual world (in a world that doesn't exist) may sound bizarre, but for some people, especially the introverted ones, it may constitute a happier experience than socializing on Wechat or Facebook.

If you only think about money and games, you miss the point of what Virtual Reality can achieve. Nonny de la Pena at the University of Southern California is pioneering a new kind of journalism that she calls "immersive journalism". Her website is http://www.immersivejournalism.com/ She debuted her virtual-reality technique in 2012 with "Hunger in Los Angeles". She uses virtual reality to make you "feel" how it feels to be in places where something terrible is happening. If i tell you that people are dying in Syria, it is just a sentence with a number; but if i show you the people dying in the streets of a Syrian city, the effect is much stronger. That's her "Project Syria", presented at the Sundance Festival in 2014. As i said, virtual reality helps you concentrate. It can also help people concentrate on world news in a way that a written story doesn't. The "reader" becomes an "observer", and therefore is much more likely to "feel" the story instead of simply "hearing" it.

Narnia: The boom of virtual reality represents the fear of people to socialize in the real world, so they socialize in a world that is not real?

piero:

Actually, it is very interesting that, at the same time that we are having the boom of virtual reality, we are also having the boom of "live streaming", of people who stream their daily life in real time. Call it "hyper-reality". In january 2016 Periscope announced that 100 million live broadcasts have been made by its users (in just one year). During 2015 Periscope ranked in the top 10 most popular social-networking apps on both Google Play and the App Store. Periscope, New York-based YouNow, Meerkat (also only one year old), Facebook Live are all very popular. "Live streaming" means that you are telling everybody what you are doing and they can comment on it. In virtual reality you hide yourself, you become someone else, while in live streaming you are "naked in front of everybody"; one the opposite of the other; and they are becoming popular at the same time. Or maybe they complement each other: one extreme complements the other.

Narnia: Virtual reality is presented as something that the masses can only use, but you are saying that ordinary people should also be able to produce it?

piero:

Today the media are mostly emphasizing the devices to "consume" virtual reality (either for fun or for work), not the tools that will help you create your own virtual world. I am more excited about the creative part. Today the main platforms for virtual-reality creators are sold by veterans of the gaming world who have been offering engines to create 3D games for a decade or more: Unreal Engine 4 (North Carolina, 1998), Razer OSVR or Open Source Virtual Reality (San Diego, 1998), EON Reality (Sweden, 1998), Worldviz (Santa Barbara, 2002), Unity 5 (San Francisco, 2005), Improbable (London, 2012), one of the most sophisticated, as well as Autodesk's Stingray (2015), built around the Bitsquid technology that Autodesk acquired in 2014. Their platforms were originally conceived for game design (the first 3D-graphics consumer market) but the new releases are increasingly general-purpose. Of course, the fact that the world of VR hardware is so diversified (Oculus, Samsung, etc) makes it difficult to provide a universal platofm that supports all head-mounted displays, all 3D displays, all motion sensors, etc.

At the end of 2016 Unity introduced a new tool named EditorVR that transports Unity game designers into virtual reality (they can create worlds while wearing a VR headset).

What we need is an operating system for creating and running virtual worlds. Improbable (founded in 2012 in London by game designer Herman Narula and Robert Whitehead) hired Sam Kalnins (the chief designer of Hangouts at Google) and Eric Molitor (from Amazon) and developed the cloud platform SpatialOS for videogames. But the same platform can be used to simulate entire cities: in 2017 Improbable unveiled a project to simulate the town of Cambridge.

Penrose Studios demonstrated in early 2016 its first virtual-reality film, "The Rose And I", which is an adaptation of a popular children's book, "The Little Prince". It's a short film but it is one of the first attempts at using virtual reality for a different kind of storytelling, unlike 3D movies that were simply a visual gimmick. Penrose Studios was established in 2015 in San Francisco by Eugene Chung, who has experience with both Pixar (animation) and Oculus (VR headsets).

As a writer, i am actually very excited about the potential of virtual reality for future "writers". A novelist or a poet uses a keyboard (or a pen!) to create a world. In the future she will have tools to create a virtual world in which her "readers" can experience the story or the poem. This should be more intense than writing and reading. It should create a different kind of bond between writer and reader (or, if you prefer, producer and consumer). The writer will become the equivalent of the "guide" or the "scout". I can take you to see amazing nature in the Sierra Nevada of California, and in that case i am your guide/scout. Some day i will create my own world of nature (in the computer or in the cloud) and guide you into it. I am sure it will not be as beautiful as real lakes and waterfalls, but it will be my world. Every parent knows that her or his children are not the most beautiful nor the most intelligent in the world, but they are her/his children, children that she/he raised. Maybe we will have a similar feeling for the virtual worlds that we created. Virtual reality tools can potentially empower us to become gods creators of new worlds.

I will escort you into my world and we will socialize in it. "Life" will not be about going to a place that preexists (a park, a shopping mall, a stadium) but about going to the places that we design.

Narnia: What are the dangers of Virtual Reality?

piero:

Every technology has a downside. Unfortunately, we find out the real danger only when it arrives. But in the case of virtual reality i can see one danger. I mentioned that virtual reality helps the user concentrate. That has a downside: that you are more likely to believe what is "happening" in the virtual world. Today there is a lot of misinformation created by Hollywood movies. People walk out of the movie theater thinking that they have learned history from a movie that is actually just... a movie. If the movie is really good at capturing people's attention, many people will believe anything in the story, or at least they will believe that there must be something true about the story. Very few people buy the book of a historian to doublecheck what the movie claims. A realistic movie can convince a lot of people that its story is true. That's one reason why we live in the age of the "conspiracy theory": if a filmmaker distorts the facts to make a successful film, she indirectly creates a new conspiracy theory. If i make a movie about the president of the USA conspiring with the president of Russia to attack China, and the movie becomes a big success, there will be a conspiracy theory according to which that is really happening, and the conspiracy theory will keep spreading for years and years. For example, there are many people in the West who think that Neil Armstrong never went to the Moon because of a Hollywood movie titled Capricorn One". Now imagine the effect of virtual reality, that is much more realistic than a Hollywood movie. The danger is that virtual reality will be used to spread all sorts of false information in a very effective way.

Narnia: What is coming next?

piero:

There will be progress in both the hardware and software of virtual reality, but i also think that virtual reality is "vulnerable" to other technologies, the technologies that use neuroscience. For example, the main industrial application of virtual reality is still simulation. For example, Boeing drops you into a virtual world that simulates a real airplane and you learn to fly the airplane. But Matthew Phillips at HRL, a laboratory owned by Boeing and General Motors, is studying how to transfer the brain waves of an expert pilot into the brain of a novice pilot. This seems to speed up the training of the novice pilot. In other words, the stimulation of your brain with the brain waves of an expert helps you learn from that expert faster. Phillips reported the first results in 2016 (The paper is "Transcranial Direct Current Stimulation Modulates Neuronal Activity and Learning in Pilot Training" in Frontiers of Human Neuroscience, 2016).

Narnia: Virtual Reality is coming of age at the same time that another old discipline, Artificial Intelligence, is coming of age. What do they have in common?

piero:

Artificial Intelligence creates "artificial beings", and Virtual Reality creates an "artificial world". The beings that inhabit a virtual world are avatars of real people, but could also be independent beings that exist only in that world, robots that exist not in hardware but in software. You could create a virtual world and populate it with artificial people that interact with your avatar and with the avatars of your friends. Artificial Intelligence will help define the personality of an artificial person: an accountant who works in a bank, likes hip-hop music and goes to church; or a Buddhist girl who likes long walks in the park and reads Chinese classics; etc. Artificial Intelligence will "power" such an artificial person so that s/he behaves just like a real person. You will not know who are avatars of real people and who are artificial people. Secondly, Artificial Intelligence could be used to "power" your own avatar. My avatar in Second Life disappears when i am not playing. A.I. could keep it awake and playing even when i am not in Second Life. The new generation of avatars could be "autonomous" avatars who learn your personality and then continue living in Second Life even when you "switch off" and return to real life. If you think that this is far-fetched, think again: the Virtual Medical Assistant provided by Sense.ly, a startup founded in 2013 in San Francisco by Adam Odessky and Ivana Schnur, can monitor and simulate your health, New York-based Medical Avatar, founded in 2011 by Virgil Wong and Akshay Kapur of Columbia University, and Anatoscope, founded in 2015 in France, create a 3D digital avatar of your body (also for health-monitoring purposes). The Oakwood Hospital in the Detroit area started offering its patients a personalized anatomical avatar. Wearables will provide continuous data about your life that software can use to learn about you. The technology is evolving rapidly and it will soon be able to create a "character" (an artificial person) based on your habits and data. David Hanson (Hanson Robotics) has been around since 2003 building robots that simulate a real person's personality, most famously the (failed) android/avatar of Russian billionaire Dmitry Istkov (founder of the 2045 Initiative who wants to create immortality).

Your avatar will live in the virtual world while you live in the real world. Every time that you plug into the virtual world again (when you wear the virtual-reality gear again), you regain control over your avatar. The avatar will learn from you how to behave (what kind of person you want it to be), and you will learn from your avatar what that behavior leads to (what that kind of person does). You can have an artificial self besides your real self, and the artificial self operates outside your control. It sound scary, but this happens only inside an artificial world. In the short term this could be the most important psychological experiment since the invention of psychotherapy. In the long term this could be even more important to understand the human mind.

Narnia: What happens to the avatar of me when i die?

piero:

It will continue living forever in the artificial world. In fact, it can live in multiple worlds. Anybody can download your avatar in their virtual world. You will be more than alive: you will multiply yourself, and each avatar may evolve into a different person in each virtual world. You may become a rich entrepreneur in one world and a flight attendant in another one. All of them will share the same personality, but become different "people" because of different circumstances. Many copies of you can exist in many virtual worlds. Forever. Just like we can still read stories about historical characters. But this is not real immortality. A replica in a wax museum also lives forever (well, until it is destroyed by the museum). Living forever in a virtual world is not like living forever in the real world with a real body.

Narnia: Can these virtual characters (these virtual copies of me) be materialized so they can live in the real world?

piero:

Sure. As 3D printing and similar technologies become cheaper, Dmitry Istkov's dream will come true: it will become possible to create a physical body of your avatar as a hologram or a robot.

Narnia: And, as the technology progresses, could the avatar/replica in the virtual world become exactly like me in the way i think? Isn't that "me"?

piero:

Now you understand why i think that virtual reality will become the most important psychological experiment that we have ever tried.
This interview was complemented with another interview: Han Jin, founder of Lucid VR.
Back to the Table of Contents
Piero Scaruffi | Contact