super mario spacetime organ (illucia & soundplane)

super mario spacetime organ (illucia & soundplane) from paperkettle on Vimeo.

From the man that brought us the illucia

"Interested in my work? http://chrisnovello.com or http://twitter.com/paperkettle

In this video, I directly manipulate the RAM state of Nintendo's Super Mario Brothers to transform it from a game into a strange instrument. I use two hardware controllers: illucia (a patchbay instrument I designed that lets me connect computer software with cables), and the Soundplane (an amazing multitouch surface by Madrona Labs).

I begin by playing the game as one normally would, just using buttons on illucia.. but I also have access to the game's memory, so I use the Y axis on the Soundplane to alter the value in the memory address that determines Mario's Y position onscreen. This is how I make Mario fly and hover during the playthrough.

Also, before I start playing, notice that I flip a switch on illucia. This triggers recording — not video, but actually recording the entire memory state of the NES for each game frame. Think about it - Mario's universe is held in RAM, which the NES uses to draw the game world for each frame. By saving the entire state of NES memory for every frame, I'm able to go back to any moment in Mario's life. Sort of like a Super Mario time machine.

So then I use the X-axis of the Soundplane to sweep through the timeline of Mario's universe.

Not only that, but the Soundplane is multitouch, so I use a second finger to specify start and endpoints in a playback loop. This is similar to the way samplers and granular synths work, but for recordings of the entire memory state of the NES rather than audio data. Conceptually, it is like Super Mario meets Groundhog Day. Mario's universe computer / time machine gets caught in hellish loops.

Then I use illucia to send alien data into Mario's universe, which makes for all sorts of audiovisual insanity amidst the spacetime loops. I found some memory addresses that produce interesting results, so I use illucia to pump them with unexpected data. This is sort of like circuit bending, but in a protected sandbox - at any point I can revert back to the clean recording of RAM states (aka moments in Mario's universe).

At that point I try to go back to "playing" the game, watching Mario navigate a melting world of glitched-out ephemera. Toward the end of the video I use a pair of rubberband mallets on the Soundplane to jump around in Mario's timeline, all while illucia is left pouring a heavy stream of alien data into Mario's RAM state. I eventually (accidentally/luckily) land at a place that triggers the game over music, and decide to end the take.

=====================

Disclaimer:

I don't condone piracy or the distribution of Nintendo's IP. I'm posting this video as part of a critical-cultural project. Further - although the emulator tools I used to make this performance are fully legal, I'm not distributing any other materials related to making these interventions into NES games, so please don't ask. The video is shown with the intention of stimulating critical thought about the role of software in our lives.

By showing that there are other ways one could interact with this well-known cultural artifact, hopefully I can inspire others to consider: Who decides how and what we see in a computer program? What is hidden? What sorts of strange other things lurk beneath the surface of our trained expectations?

What if we lived in a culture that deeply embraced the expressive possibilities of computer programs? What if we encouraged curiosity about the interior of the systems that compose game worlds? Is that curiosity and critical inquiry itself not a form of play?

Indeed there are platforms that embrace this sort of approach, but many don't. In fact, many close it off. What if we prioritized this sort of inquiry more? Might we find new frontiers for communication or even human knowledge and representation? What is a videogame, and how can it relate to all this?"