SPACE DEMENTIA

_{Interactive Immersive Experience Visualizing the Homonymic Fictional Mental Disorder}

---

PROJECT DESCRIPTION

This project is a site-specific interactive immersive experience visualizing Space Dementia, a hypothetical mental disorder that causes psychological instability to astronauts isolated from the human society in the outer space.

Consisting of an animation and an embedded interaction design, this project breaks the fourth wall in 2D visual art by tracking the motions of the audience and transforming these data into audio feedback. It brings interactivity into convential ways of viewing moving images.

Type
Date
Instructors
Keywords
Tools

Individual Project
December 2019
Sarah Rothberg + Mimi Onuoha
Spatial Interaction + Animation + Motion Tracking
JavaScript (p5.js, ML5.js) + Adobe After Effects + Adobe Audition + Adobe Photoshop

01  KEY CONCEPT

In Space Dementia, the audience is both a viewer and a player:

02  APPROACH AND TOOLS

03  PROJECT FLOWCHART

04  DEVELOPMENT PROCESS I - ANIMATION

︎ VISUAL PART

_Storyboard

There were nine major scenes in this three-part animation:

︎ AUDIO PART

_Script

The script was written as the diary of an astronaut who was trapped inside his space craft and lost connection to the Earth. The 3 parts of the animation refers to 3 pieces of diary from 3 incoherent dates, during which the astronaut becomes more and more mentally unstable. In the Part II of the animation, the astronaut mentions his condition of auditory hallucination and gives instructions that ask the audience to move their arms for motion tracking and spatial interaction.

· Part One

Today is April 2nd, 2050, Eastern Standard Time 2:14 am. This is my third day in galaxy ESO 495-21. My spaceship deviated from its original track and is stuck in this galaxy. Connection to the Earth is lost totally since yesterday. No response at all. Nothing coming from that blue planet, not even some distorted muffled voice telling me they will keep trying to reach me. There is nothing I can do, but staring out of the window and waiting for reconnection. Not much to look at, just the planet M84. I’m keeping this diary because this is the only way that I can talk and think to keep my mind working. I hope the communication equipment could work soon.

· Part Two

Today is, uh, April 8th, 2050. Oh no, April 9th, 2050. Eastern Standard Time, 6:34 pm I think. The 10th day in ESO 495-21 and still there is no connection, but still I’m waiting. Food and energy are enough for one week or two, but I feel so empty like I am frozen inside. Sometimes I have this strange feeling that I am not the abandoned one, but the one who abandons the rest of the world. I also find this subtle harmony with the space, that I can hear the sound of the space coming through my ears, into my mind. If you are listening, I want you to come closer. Come and stand in the center. Stretch your arms like you are about to embrace someone. Do you hear it? The sound of the universe? Stretch your arms so that you can hear different sounds, like you are playing an intangible harp; so mesmerizing. Now raise your arms high. Do you feel the change? The rising amplitude is waking everything in the world. I feel like I am a growing tree, I see the trunk and leaves growing in front of my eyes. I feel like an albatross, flying above the ocean of broken stars. I see images of living creatures are covering up the wall of my spaceship, so surrealistic and so sad, for I miss the touch and the temperature of something alive so much. I miss the feeling of another beating heart.

· Part Three

I lost count of the number of days I have spent in this isolated spaceship. Time starts to seem useless to me; it is just a persistent illusion. I stare at M84 the whole time, and I feel it staring back at me. We star at each other, through the glass, and the jungle in my mind. We are the only two things in my world, and it is getting closer and closer. I am forgetting the days and years I spent on the Earth. My spaceship is the womb that conceives me. My memory is melting away, and my existence is decaying. My spaceship runs out of energy, and it carries me floating away. Soon I will become a dead child in the mother’s womb. The history of mankind is a running river, while I am merely a raindrop of tragedy.

_{Record, Edit, and AV Sync}

The voiceover was recorded using Zoom H6 recorder and edited in Adobe Audition:

︎ ANIMATION OUTCOME

The final outcome for the animation part was a dubbed digial collage animation in surreslitic style and it is 4 minutes and 29 seconds long. Here are some stills:

05  DEVELOPMENT PROCESS II - INTERACTION DESIGN

This interactive part embedded in the Part II of the animation is for re-creating the auditory hallucination experienced by the astronaut. As the audience move their arms as the animation instructs, they will hear music that responds to their movements. I used p5.js and PoseNet in the ML5.js library, a pre-trained machine learning model for Real-time Human Pose Estimation.

︎ MOTION TRACKING

_{Data Detection}

PoseNet processes the image of human into a skeleton of 17 key points and tracks their coordinates. I chose to track the movements of the two wrists of the audience, the result of moving hands and arms, since astronauts move in a weightless environment with this gesture.

_{Data Process}

PoseNet only generates x and y coordinates of tracked parts, so additional code is needed for calculating the distance between the left and right wrists.

︎ AUDIO FEEDBACK

The distance between two wrists ontrols the midi value of the audio:

Calculate the y coordinate of the two wrists controls the amplitute of the audio:

︎ TESTING

---

06  DEVELOPMENT PROCESS III - SPATIAL LAYOUT

This project is installed inside a room with three blank walls:

07  OUTCOME

This project was not set up as large as the ideal spatial layout has planned due to limitations in space. However, it was shown to and tested by more than 40 people in two in-class critiques. It received positive feedback, such as “It is a very creative and engaging way of storytelling.”

︎ IN-CLASS SCREENING

︎ IN-CLASS INTERACTION TESTING