Reminiscence
-
Sponsored by Google
-
This installation showcased at YBCA as a part of the Future Resonance exhibition
-
Our exhibition received recognition in the form of a Bay Area outpost for the prestigious Scholastic Art & Writing Awards
-
Web application based on P5js and interacted via the Playstation controller and proximity sensor
-
Tools:
Figma, p5js, Arduino, AI-generated images by Dall.E, and Azure Cognitive Service for AI-generated sound
Abstract
This project is about exploring a new interactive experience to interact with our past memories in the digital world. We’ve entered the era of big data, and the amount of our personal online data is increasing at accelerated rates. One of the more valuable pieces of data is about our past experiences, which are often encapsulated in pictures and videos. We took a lot of photos and videos, however, the way we access this digital data today is quite different from how we recall our memories. The memory recall process in our brain is often involuntary and unconscious, triggered by the surrounding environment such as visuals, sounds, and smells. On the other hand, our digital photos and videos are stored discretely and disconnected, and we do not scroll through these photos very often as it does not feel natural at all.
In this project, I explore a new interactive experience with our past to replicate how memory naturally wanders in humans. The photo snapshots of our past are connected based on the context such as sounds, emotions, and event types. By doing so, we can explore the past through a chain of events or juxtaposition of multiple events. This digital interface is a refreshing departure from traditional photo galleries, as it allows us to navigate our recorded memories more organically, resulting in a more authentic recollection of our lives.
Eventually, I prototyped the interface via a web-based interactive experience, and the project was then installed and exhibited in YBCA in April 2023. The interaction design is demonstrated by showcasing my personal history and experiences as the content, such as my childhood in China with my grandparents and my immigration to the US. I documented these memories by linking them to specific locations and using sound to create an emotional and aesthetic experience that corresponds to each specific moment. As visitors browse through the interface, certain memories will be triggered automatically, along with associated memories. To the audience, the content of my memory will make no difference from any other stranger’s memory. It is the interactive form where the content lives that matters, as an attempt to replicate the experience of involuntary memories that occur spontaneously and let us re-live the essence of our lives.
Context
Memory is one of the most complex processes of the brain. The ability to remember things is not only important for us to function normally as human beings but also allows us to connect with our past and experience the duration of our lives. There are three major stages when we process the memory, encoding, storing, and retrieving.
During the encoding stage, we perceive the world via our sensing modalities such as visual and sound systems, and the input from the environment is translated into neural signals in our brain.
In the storing stage, the encoded information is stored as short-term memory. Short-term memory is only stored for a very short amount of time from 20 seconds to 30 seconds, and then it either gets lost or transferred to a long-term memory, which is then retained for a longer period of time often until the person dies. Long-term memory is further divided into two types, conscious memory pertaining to facts, and unconscious memory which we are less aware of.
During the retrieval, we access the stored long-term memories.
Empathize
Although I started this project with personal intentions, and the memory-recalling process is a very personal experience, I am also curious about what role involuntary memory plays in other people’s life. On one hand, I want to get inspired by other people’s personal experiences for my design. On the other hand, since I will install my project in YBCA, I would hope to find a way that my installation can engage and interact with the audience.
The questions I had in mind were:
-
What are the most remembered experiences that would often occur in people’s minds as involuntary memories? Are there some common features of such memories?
-
What role does the sound play in the involuntary memory recall process? If any, what are some examples of such sounds?
-
How to repeat someone’s personal memories in a different format that can resonate with them? More specifically, when the audience visits my exhibition at YBCA, would it be possible to collect the involuntary memory experience from the visitor, turn it into some content in my system, and have the visitor interact with their own memory using my interaction design?
Understanding the basis of involuntary memory
Memory has an important aesthetic function. Proust had segregated memory into ”voluntary memory” and ”involuntary memory”. Based on the neurological definition of the memory storage and retrieval process described above, voluntary memory is like the active recall of stored facts, usually controlled by reason and needed for practical uses. This type of memory has eliminated the emotional details, away from the real needs of the mind and emotions. It is not aesthetic memory.
Involuntary memory, on the other hand, is a memory that comes to mind spontaneously, automatically, and with no conscious initiation of the retrieval process. It is highly frequent for us to recall involuntary memories, and the recall is usually associated with the current context we live in. Sound is an effective way to recall such memories that trigger an emotional and aesthetic experience. For example, hearing the sound of a train in the night could bring us back to our childhood when we traveled on an overnight train. Such a memory is about emotional impressions and the atmosphere of the mind and hence it is aesthetic.
Marcel Proust was the first person to coin the term involuntary memory, in his novel A la recherche du temps perdu (In Search of Lost Time or Remembrance of Things Past). Proust viewed involuntary memory as containing the “essence of the past that makes us aware of the eternity of the self”. When the protagonist of Proust’s novel eats a tea-soaked Madeleine, a long-forgotten childhood memory of eating tea-soaked Madeleine with his aunt is restored to him.
“And as soon as I had recognized the taste of the piece of madeleine soaked in her decoction of lime blossom which my aunt used to give me (although I did not yet know and must long postpone the discovery of why this memory made me so happy) immediately the old grey house upon the street, where her room was, rose up like a stage set to attach itself to the little pavilion opening on to the garden which had been built out behind it for my parents (the isolated segment which until that moment had been all that I could see); and with the house the town, from morning to night and in all weathers, the Square where I used to be sent before lunch, the streets along which I used to run errands, the country roads we took when it was fine (Proust, 1928).”
Technology Research
-
Forms of Image Overlay
In my exploration of image overlays, I’ve discovered that they can be a powerful tool in conveying the connection between past and present memories. By layering images on top of each other, I can create a sense of visual depth and show how memories overlap and influence each other.
Throughout my design process, I’ve incorporated these image overlays in various ways to effectively convey this concept to users. In my early concepts, I experimented with different forms of overlays, such as using transparent images to create a ghostly effect or blending images together to show how they relate to each other.
In my final designs, I continued to use image overlays to create a more immersive and engaging experience for users. By layering images and sounds together, I was able to create a sense of continuity between past and present memories and show how they can be triggered by environmental cues.
Overall, my exploration of image overlays has been an important part of my design process, helping me to effectively convey the complex relationships between past and present memories.
Design Concept
-
Image Overlay in Processing
Exploring the visual effects of overlaying memories and photos on top of each other using Processing allowed me to create dynamic and engaging interfaces that effectively conveyed the concept of memory retrieval. Processing is a flexible coding platform that provides me with more control over the visual elements of my design, allowing me to create intricate and compelling visual effects.
The use of Processing as a prototyping tool was particularly useful in the interface demonstration for the final exhibition in the museum, as it allowed me to quickly iterate and refine my design based on user feedback. Additionally, the flexibility of the platform allowed me to easily incorporate additional features and functionalities as needed.
I drew inspiration from the work of other artists who had explored similar concepts, incorporating shattered images and bubble images to represent image overlays and the transition from one memory to another. These techniques effectively conveyed the complex relationships between different memories and provided a visually stunning and immersive experience for users.
Below are some early works I created using Processing code, showcasing the various visual effects and techniques I employed in my design. These examples demonstrate the potential of this platform for creating engaging and effective user interfaces that effectively convey complex concepts.
-
Final Concept and Prototype
For my final prototype, I utilized P5js, a web-based coding technology that allowed me to build digital interfaces and graphics projected onto a web page. I generated bubble images of four major locations from my past life, including my grandma’s room, my childhood room, the airport, and my living room in the US, where I migrated as a first-generation immigrant from China. Each bubble image was programmed to move in a random direction back and forth with a varying speed to simulate the particle-like and hard-to-grasp details when we recall memories.
To create an immersive and dynamic experience, I programmed multiple patterns assigned randomly to the images I generated and listed in the “Design Strategy” section. Each image was linked to specific locations and other images based on common features. Upon opening an image, the sound of that image would play automatically, and the image itself would transform into bubbles that spread into the dark background. Other related images would appear on top with varied dynamic patterns, random floating positions, dimensions, and resolutions that kept changing, just like our memories that are never static but keep changing as we recall them.
I also enabled interaction with the web page using PlayStation controllers to provide an engaging experience for the visitors in the exhibition at YBCA. This feature allowed visitors to explore the memory content and gain a deeper understanding of what the interface is like.
-
Interactive Interface
The eventual exhibition “Reminiscence: Memories of a Life Lived Across Cultures” at YBCA showcases the personal history and experiences of a designer who grew up in China with their grandparents and later moved to the US in 2012. Through a collection of artifacts, photographs, and multimedia displays, visitors are invited to explore the unique memories and emotions of living across divergent cultures and locations.
The exhibition is divided into several sections, each highlighting different aspects of the designer’s memories. The first two sections, “My Grandma’s Room” and “My Childhood Room”, feature family photographs, traditional Chinese artifacts, and personal mementos that capture the sights and sounds of growing up in China.
The third and fourth sections, “Airport” and “My Living Room in US”, showcase my experience of adapting to a new culture and country. Displays include photographs and sounds that document the struggles and rewards of starting a new life in a foreign land.
The multimedia displays are connected by common sight and sound features, bridging the different strands of my experiences and exploring how memories can both bind us together and fade away over time. Visitors are invited to engage with my personal history and to reflect on their own memories and experiences.
-
Design Strategy
Exhibition at YBCA
Video Prototype