Photo Credit: YouTube
We all know what music sounds like…but what does it _look_ like? Or more specifically: What does [Rudy Mancuso](https://www.instagram.com/rudymancuso/?hl=en)’s music look like? That was the question the actor/musician/internet content creator set out to answer. Not because he can’t see his music, it’s actually quite the contrary, but because other’s can’t see the colors and patterns his mind associates with his tunes.
To answer this inquiry, an “experiment” went underway. Advanced machines were created that use midi signals from musical instruments to put paint on a canvas, reading the pitch, note length, dynamics, velocity and range to give it a specific paint stroke. Mancuso performed an original composition four times to create four separate paintings. And so the performance art and NFT series titled _Synesthesia_ was born, named after the neurological condition that Mancuso has and that inspired this entire project.
The four-piece NFT series, including physical offerings, will be available starting Thursday, May 13 on [Foundation](https://foundation.app/).
Read _Flaunt’s_ conversation with the man behind the project below!
**Having synesthesia, what is your relationship with music like?**
I've been trying my best to educate others while simultaneously being educated, because it is quite intricate and complex. There's so many different forms of synesthesia. Today there’s over 70 and growing. Essentially, a lot of people already know what synesthesia is, but it is a condition where a person's brain links senses together in a very interesting rare manner, prompting unusual sensory responses to stimuli. So people with synesthesia, for example, might see a certain color and response to a specific letter of the alphabet. Many people describe hearing colors, feeling sounds, and even as far as tasting shapes.
For me, for as long as I can remember—because I've always thought creatively and through music—music was at the forefront of almost everything that I did. I was playing piano before I could walk. So for as long as I can remember, I would associate certain sounds, ideas, and more specifically musical ideas as colors, shapes, and personifications. And it wasn't until fairly recently that I realized that a lot of people have this condition. For a lot of influential people as well, from Duke Ellington to Pharrell to Geoffrey Rush, they all have forms of synesthesia. Pharrell considers it, and I agree with him, not a condition, but an asset to an artist or to a musician for specific forms. For a lot of other people, it's quite debilitating and anxiety-inducing, and it has been for me, as well, depending on the situation or the symptom.
So I have, from the tests I've done and from what I've been educated on, chromesthesia, an ordinal, linguistic personification and one that's currently unspecified that I consider obsessive rhythm or rhythmic association where I essentially can't help but turn every regular sounds—to you, you may hear footsteps of someone going up the stairs, or a construction worker banging on a hammer, my mind immediately tries to identify the rhythm. And if it's identifiable, interpret it into some kind of musical rhythm. So I'm almost, like, turning things into rhythm or making musical sense out of sounds that have no business being musical, which sometimes enhances and is an asset to my storytelling and to making music and being creative, and other times, like when trying to have something as basic as a conversation with another person, very distracting and anxiety-inducing. It's chromesthesia, which is associating colors with ideas, or sounds, or numbers of letters. Or linguistic personification is exactly what it sounds like, find ordinal numbers, days anything from days to weeks, or letters in the alphabet, you associate with personalities, you start personifying them, I do this more so with music.
All this is to say, this idea of being a synesthete, myself, and being obsessed with synesthesia, the link between more specific sounds and visual music and color notes and personification of those notes have led to this experiment, which you saw, and is a more physical adaptation of the idea behind synesthesia. There's so many different types, and increasingly more and more. So many people have synesthesia and don't really know what it is. It's kind of hard to diagnose so many different kinds. For a lot of people, it's very difficult. For myself, it's been difficult at times and enhances what I do another time.
**It’s debilitating at times, and enhances your abilities at other times: When you first released you had this when you were young, was it just debilitating? When did you realize it could work towards your benefit?**
I remember there are distinct memories I have of, I considered at the time like a learning disability, my brain would pick up on sounds in a classroom or during a meeting and my mind would try to interpret those sounds rhythmically. And other times, I was listening or playing music and would associate certain chords or chord progressions, or scales, as this color. And other times, there's personalities associated with the chords that I hear and play. For instance, a major seven chord, it's specifically the B flat major seven chord played on that instrument. So I'm associating that with this personality or mood or color. And that exact same chord played by a different player on a different instrument, or in a different musical context associated with a completely different personality, or color, or mood. So growing up, I was experiencing this when playing music, which makes a lot of sense, it would help me. _Oh, let me go a little bit to this pink or orange or this blue._ Other times, when not in the context of music, it'd be a bit distracting, it was really hard. I just thought it was a learning disability, which I guess it is. Up until recently, I called it musical OCD, obsessed with pairing this, associating this color or this personality with this chord progression of scale. Now I know it as synesthesia. So growing up, I didn't know how to really articulate what it was, it was just something that was happening. And a lot of times, I thought that most people were experiencing the same thing and they weren't.
**At the beginning of the video introducing this project you said you wanted to ask the question, ‘What does your music look like?’ How long has that been a question that you've had? Is that a question you've had, since you were younger, and you first started making music?**
I have my own version of what my music or experience of listening to music looks like in my head and the images that it projects and how I associate it. And that'll be entirely different from, even with some of the exact same forms of synesthesia, or upbringing or experiences I had, there's no way to replicate that. There's no way to know how someone else experiences something exactly the way you experience it. I have my own versions of that. So I really wanted to see if there was a physical way of music being translated into visual. I've been asking myself that for a long time: _Man, I wonder what my music or this piece or just this melody looks like as a painting or performances_. I mean, you can see right now, a lot of people do this, a lot of people I admire do this for fun. Jacob Collier, for instance, he played an intricate piano scale, melody, and harmony going up and down until it was all being registered in MIDI into this recording software. And you can see the MIDI notes and you turn the MIDI notes into pieces of art. So he made two hearts out of his musical scale. That's a more simpler, less physical translation, essentially doing the same thing. It's what those MIDI notes look like for him, in that moment. It looked like two hearts. It was organized in that way. But I wanted with less parameters to figure out exactly what my music was looking like. I mean, we have sound waves, we have waves where we have technology that allows us to interpret sound as a visual and visual as a sound. It could also go the other way. What does the Mona Lisa sound like? It's complicated because some parameters have to be set. Right? But for at least a decade, I was wondering: _Is there a type of technology that I could just literally input my piece or my voice, and on the other end, see what it looks like on a blank canvas?_ And the truth is, I couldn't find it. So I decided to create it.
**How did you go about creating this technology?**
I've been working with a bunch of different production companies and teams, and my own team, Shots Studios, and independent studios just to produce and shoot content in general for videos and music films. So I've met a lot of people, and I contacted a few of those people just that I've worked with before, have a rapport with and it took a while. I was going through special effects directors and practical effects supervisors, it wasn't until there was this one specific person that I had never worked with before, but was highly recommended by a friend that decided rather than going the filmic route and hiring people that work on film for film, TV content productions, we should look at the tech route and the mechanical design route, people who create robotics and mechanical engineers specializing in mechanical engineering. Marry that with practical effects supervisors and directors and let's have a dialogue on how you can repurpose already created technology, software, and hardware to do what you want, which essentially turns MIDI signals into mechanical signals and those mechanical signals, turning into paint nozzles, spraying on a canvas. This is the 1.0 version of what I would hope to do with the 10.0 version, which is a far more sophisticated, complicated piece of machinery that can translate an entire orchestra's music into visual art. And that's just the beginning, then there's 3D art, 3D printing. The possibilities are endless.
**So you definitely see yourself building upon this project in the future?**
Absolutely. The catalyst for me wanting to explore this was the curiosity I had being a synesthete and the idea of synesthesia as a whole, then this emergence, this craze, this movement, that is NFT's and digital art, which is very new to me. It took me years to understand cryptocurrencies, let alone it being applied to a brand new market. So I educated myself on that, I saw that people were really excited about exploring digital pieces of information. And that's the first time it really all made sense to me as it being done artistically. So people got really excited about this new wave of digital art, including myself, I educated myself a little bit late to the party, but then I took this long last kind of dream I had of: _can I create a proprietary way of turning music into visual art_? And I kind of bridged the gap.
This is a proof of concept. It's just the beginning. I want to partner with creative minds, I already have a few in mind, a few peers of mine, friends of mine, who I admire, also fellow synesthetes to help build the million ten million dollar version of this machine that can translate music into visuals. Hopefully one day I can go the other way around and translate visuals into music. Parameters have to be set, but I would like to see what John Mayer's guitar solo looks like on a blank canvas or as a piece of digital art or as a 3D printed sculpture. I would like to see what Etta James’ voice looks like. It's very interesting to me, and it was why I decided to do this. To stop everything that I was doing and prioritize this experiment. With a lot more time, money resources, I can build a much more sophisticated version, but this piece that I'm going to be releasing in conjunction with the NFT collection is really a big experiment.
**Could you ever see yourself turning this into kind of a live performance?**
Absolutely, we can bridge the gap between live musical performance and an art show. I think it'd be super cool to watch one of my favorite artists, or orchestras perform an entire concert or a piece, while art is being physically, in real-time, created based off of that music. And then at the end of the show, you can bid on buying the art, you can download a digital version of that art. It's something I've never really experienced before. It could be really interesting.
**This entire process only took 16 days. What did the typical day look like or was there no typical day?**
Every day got progressively more stressful. It was 16 days simply because of the nature of this whole movement, in creation in general, is moving increasingly faster and faster, I wanted to be ahead of the curve. As soon as the idea was formulated in the head of a crew I knew it needed to be actualized yesterday. So I put everything in order, and I got everything organized.
Everyday was just a question mark upon a question mark trying to figure out... we know it's possible: Is it possible in the short window? I had only two weeks because I was flying to where I am now, which is London to shoot a movie, and I knew I'd be here for months and this idea two months later is already too old. So the challenge was finding a crew that believed enough to work day and night and not sleep for a couple of weeks designing this machine, actualizing the software and then see turning it into a video, documenting it in real-time—nothing was cheated, or enhanced in post, what you see is what you get, and we weren't quite sure what was going to happen. We built a machine that would move. It was constantly moving on an axis that would cover enough real estate on the canvas. We weren't sure if it was going to look interesting or cool. But what you see is what you get. Music is the artist.
So those 16 days were really a bunch of phone calls, tests, meetings. The team that was building it was on one coast and I was on another. The whole thing was a big risk. The last two days before we were shooting, the engineers had to fly to the west coast to shoot the video with the apparatus in their carrier. At any point, something could have gone wrong. I could have not had the machine, there could have been a tech glitch. The art might have not translated the way I had in my mind. But it was a massive risk. But it ended up somewhat working out. The proof of concept worked.
**Tell me how you felt at the end of those 16 days: Did it translate into what you were anticipating it to?**
It has a very like abstract expressionism feel, if you will, that wasn't the intention. Whatever the instruments and the machine created was what it was. And there were some that we did, we performed it four times in full, and there were times where you know, the color scheme was off. If an art connoisseur were to come in, they would have said, h_uh, this is wrong for this reason and that reason_, but the truth is there was no wrong or right. It was a machine translating when I was playing. I think it exceeded our expectations because we filled the entire canvas with colors and schemes created by music and that was the intention. I could see when I look at each one of the paintings where my notes were. And a lot of it was randomized. At a certain point in the piece, I'm playing drums and they're physically splattering onto the canvas. That's less interpolating MIDI signals and more a physical representation of music affecting art. But if I look at it, I can see exactly where I was playing and what colors that meant and I chose those specific colors because those are the colors that I associate those chords and arpeggios with. We changed the scheme up on every take and I performed it differently on every take, which is why the four pieces are different from each other.
**Is there anything else that you want people to know about this project or the NFT series that's dropping?**
This could be the beginning of something more. Every dollar that the NFT produces will go back into funding the next 2.0 version of this machine. I want to invite other artists, singers, creators, to use this technology, and show us what their art or their artistic language looks like. The possibilities are endless. This is the beginning. And whether people get it or like it or not, will not change the fact that I want to keep going.
.jpg)
