I want to challenge the paradigm of current immersive reality application by improving it to be more universal and inclusive for visually impaired people by engaging senses other than sight such as auditory, tactile, and olfaction into mixed reality platform.
The ultimate goal of my MFA thesis is to speculate whether the multi-modal sensory involved immersive experience is an optimized platform for increasing people's empathy and reducing discrimination through immersive technology.
How do people behave if they cannot see ?
I began with this question in order to learn how people behave if their sights were taken away. In this section, I designed three experiments and one interactive media prototype in order to observe and record how people behave while blindfolded. Will they be able to draw? sculpt? with other remained senses?
Due to pandemics, I was only able to conduct one in-person experiment in the first section. There are four participants who participate in the interactive media prototype in the second session through Zoom.
Experiment I
In this experiment, I asked my participant to blindfold and draw 3 things:
One favorite animal
One favorite food
Two Cubes at different location on paper
Experiment II
In part II, I played 3 soundtracks that the participant never heard before and asked him to react with a gut reaction to draw on the paper while blindfolded. Also, I prepared a bunch of colored pencils without telling him what colors I have. He could ask me for any color he wants for his drawing anytime. If I do not have the color he asked, I just used a sketch pencil to replace it.
Songs I played:
Debussy: Prelude to the Afternoon of a faun
Schoenberg: Verklärte Nacht, Op.4 - Boulez.
Wagner: Ride of the valkyries
“It reminds me a forest in nighttime and there is also moonlight. I feel calm and little bit sorrowful? so I choose blue” —response of Debussy: Prelude to the Afternoon of a faun
“It is a moment you stand in a royal palace’s hallway, where nobody is around you” —response of Schoenberg: Verklärte Nacht, Op.4 - Boulez.
“I fee motivation, domination and provocative so it made me think of a powerful king sitting in the place to make an order” — response of Wagner: Ride of the valkyries
Experiment III
In this, I used three different objects. Instead of drawing, I gave my participant clay to reconstruct what he touched.
The objects:
A mini cup
A box cover
A squishy animal
mini cup
Box
Squishy animals
Drawing With body and sound instead of sight
Ideally, I want to create an installation or space for people to be blindfolded and focused on auditory and tactile senses. However, due to pandemics, I experimentally conducted this through Zoom via webcam and speaker. In this experience, participants are supposed to blindfold themselves and use body gestures to change what they heard and have something in mind to draw with body parts and movements. The webcam detected postures and covert to "virtual brush stroke." The participant's gesture will impact on brush's color, width, and pitch of sound they heard. Participants would not know what is going on the screen or what their drawings look like until they decided to end this experience.
Thank Zoom, I had four participants across the world to test this experimental piece. Also, the piece is available here
Results
Translation between visual and auditoral.
My intuitive response of “how do we behave if we cannot see” is translating all the visual data to sound and possible tactile.
The following section focus on experimenting with the idea of image sonification, data sonification, and physical data visualization
Color To Sound
First, I tranlsated the colors of spinning wheel to midi-keys according to frequencies.
Then, I was exploring the system of converting visual information such as color to sound with Scriabin Correspondence.
Video to Sound
As the numbers of pixels increases, the generative sound will be more “non-sense” and “intense”.
Capstone Research: Previous and Now
In this process, I want to diverge my research on the theory of disability studies, examine previous and current studies on designing for VIP.
Facilitate My Research In Immersive Environment
I tried to facilitate and present my result of capstone research in immersive environments, in this case, an interactive AR platform with physical data visualization.
Gamification To Empower empathy
For the final of first semester, I prototyped and designed a VR escape room that asks participants to solve puzzles while VR Cameras simulates low vision people's eye conditions. with current mixed reality technology such as collaborating and story-based games that require the collaboration of sighted and sightless people, participants to solve the puzzles may help promote empathy and gain the sigless people's self-confidence.