The ‘substitutional reality’ system consists of a video camera, a computer for storing recorded footage and a head-mounted device that displays, and switches between, recorded footage and a live feed captured by an attached camera and microphone.
Fujii and his colleagues recorded participants while giving them instructions about the experiment. Each participant was then asked to wear the head-mounted device, which displayed a sequence of recorded and live scenes designed to surreptitiously substitute the live scenes with recorded ones.
The first scene was a recording of one of the researchers appearing at the door and asking if the participant felt comfortable wearing the device and to test it by looking around. This was followed by a ‘doppelgänger’ scene, in which participants saw the recording of themselves receiving instructions from the researcher and a fake live scene in which the experimenter re-entered the room and explained how the experiment was designed. Finally, the device played a live feed of the researcher returning to reveal that the previous scene was actually a recording.
The participants realized that the doppelgänger scene could not be real but failed to distinguish between the live and recorded scenes during the rest of the experiment, showing that the device can successfully substitute reality with recorded scenes and that the participants subjectively experienced the recorded scenes as real. The researchers determined that head movements and motion parallax—how objects change shape and depth with changes in head position—did not influence the performance of the system and that participants were less likely to notice the switch between live and recorded scenes if it was done while they looked around the room. They also established that some of the participants had noticed a difference in the audio quality of the live and recorded scenes and used these differences to establish when the switch between the two was made.
“We can replicate the delusions of psychiatric patients but the system is not directly usable for diagnosis or treatments,” says Fujii. “We are expanding the quality of the technology and trying to extend the system for people to use as an experience platform. It can be useful not only for scientific experiments but also for entertainment and art.”