AVFX-1 (wip)
audio visual effect processor
What is this?
The AVFX-1 is a hobby project that explores audio visual effects for real time music. Audio effects can be just about anything you can dream up, and are tightly coupled to visual effects which can also be just about anything. The audio effect can influence the visuals, and/or the visuals can influence the audio effect.
This project has been a lot of fun to work on, and is the 3rd iteration building on some prior experiments see [prior work and special thanks] sections.
I've made a handful of these for friends, but currently don't have any plans beyond that other than using this as a dev kit for my own audio visual explorations
What is this?
The AVFX-1 is a hobby project that explores audio visual effects for real time music. Audio effects can be just about anything you can dream up, and are tightly coupled to visual effects which can also be just about anything. The audio effect can influence the visuals, and/or the visuals can influence the audio effect.
This project has been a lot of fun to work on, and is the 3rd iteration building on some prior experiments see [prior work and special thanks] sections.
I've made a handful of these for friends, but currently don't have any plans beyond that other than using this as a dev kit for my own audio visual explorations
Example Effects
- Electremolo
- Overdrive
-x
-y
-x
Project Goals
- Hot reloading for everything
- Effects completely defined in text and LLM friendly. There are some great node based audio processing frameworks out there, but I wanted something LLM's would be great at
- Demonstrate what's possible if your audio effects are tightly coupled to a stylish, visual representation.
A note about AI
I used AI heavily in the development of this project. My background is in technical art (for video games), and all programming I know has been via osmosis at work and through friends and coworkers. I've found the recent advances in AI fundamentally game changing in allowing me to tackle problems that would be intractable on my own. This project is certainly an example of one of those.
I'm *NOT* using any AI tools in the writing of this project overview. The words, including all typos and grammatical errors are my own. In fact, I quite like having typos because it will help show readers that there is a real human putting thought into these words. One of the aspects of AI I like the last is the erosion of human connection and the skepticism that I know approach all content on the internet with. My errrors here are a bit like my human fingerprint.
What's in the box?
Hardware and OS
The hardware is a Raspberry Pi 4B with a SoundPI audio shield from the excellent folks at Blokas Labs. The SoundPi has a quarter inch stereo in, quarter inch stereo out, preamp and volume knobs. It also runs Blokas Labs Patchbox OS which streamlines audio setup on the pi, including the realtime linux kernel, which achieves ~5ms latency on the audio signal. It feels snappy.
The display is a standard 800x480 LCD touch screen connected via DSI. The box itself design and laser cut by me.
Software
Audio processing is done in Super Collider. The Application is built in Electron with a React front end. Visuals are rendered via a P5.js HTML canvas, which is typically used as a light wrapper for various glsl shader pipelines (Fluid simulation, diffusion (the old school kind, not the llm image gen kind), feedback effects, etc...).
Super collider and the app communiate via OSC messages.
The app was developed on MacOS. While it runs fine on desktop, the latency is better on the raspberry PI thanks to the real time audio kernel.
A note about Wifi
This is a wifi enabled effect processor. It works best on wifi and will auto update as new effects are added to the effect library. It will also download new app releases automatically. I did put extra effort into making sure the device is still usable without wifi, because in practice that's what will happen when your playing with friends.
Making an effect
An effect (in the audio/visual sense) is defined by a json definition file which describes the name of the effect, what faders should be exposed, and inclues paths to the .sc file (super collider sclang) and a .js file (P5.js sketch). I didn't want to limit how the audio and visual systems can interact, so these files are also responsible for defining how OSC messages flow between two. A typical setup involves the audio effect defining some FFT's and other audio effects (reverb, distortion, chorus, vocoder, etc...) and then sends the waveform + FFT + other signals to the p5.js sketch. Meanwhile, the p5.js can send OSC signals back to the audio system.
In practice, this makes the creation of new effects more complicated and tricky to dial in. It's cool, but the majority of effects I've made are typically simpler: the audio effect sends signals to the visual effect, but other than fader values not a lot feeds back into the audio effect. I'd like to explore this more though (for example, you can imagine a fluid sim which changes the guitar tone based on how colors are mixing).
<diagram of an effect here>
Things that worked really well
- hot reloading for everything
- Building the app on mac
- p5.js is actually a great, lightweight wrapper for glsl shader pipeline fun.
- LLMs are incredible at writing and modifying glsl. I used to hate writing shaders for p5.js and other web tech, because you typically just cram your shader inline as a long text string. But that also means you'll get no meaningful syntax highlighting, linting, etc... If you trust fall and let the LLM writing the shader text (of course with clear direction and code review), that problem largely goes away.
- LLMs are great at electron + react + glsl. Pretty good at super collider.
Failed experiments and previous iterations
This is my favorite section. I tried a lot and learned a lot.
The AVFX BFF
In 2022, I worked with my good friends and collaborators Patrick Hacket and Brian Min on a Eurorack visualizer. It had 4 control rate inputs and 4 harware knobs. The application was built with Juice, and visualizaiton was done via OpenGL shaders. This was really neat, but it was difficult to iterate quickly on. I found myself wanting hot reloading of all aspects of the app, which is one my favorite parts of the current iteration.
I also wanted to make these for friends, and building the hardware eighth inch inputs and knob faders was time consuming and fragile. I built an arduino midi controller which connected to the hardware via USB. It worked fine for prototyping, and I'd like to revisit this with a custom PCB built to my own CAD specifications one day.
The Bice Box
in 2023 for my good friend Matthew Bice's birthday, I made the "Bice Box". It's a guitar pedal that you talk to.
The Bice Box was mostly intended for laughs, but I also still really like the idea for education and solo play. It had a separate microphone with a push to talk button on the touch screen. You could talk to virtual Bice and it would pick effects for you and set parameters. There was also a "creative" mode where Bice would generate new sclang (.sc) audio effects for you on request.
As fun as this was, it was impractical for a bunch of reasons - first and foremost that it is super lame to try and talk to your guitar pedal while you're playing music with friends. "Hey guys, can you stop playing for a second I'm trying to talk to my guitar pedal..." It wasn't a good look. I still plan on coming back to some of these ideas though.
The challenge of making good audio effects
I have a lot of experience making visual effects, and it's mostly just creative and fun for me. I found audio effect creation to be a lot harder, largely because I found it so tricky to make something "new". It's easy to chain patches together in different ways and with different parameters, but many of the things I made felt to similar to standard audio effects that have existed for decades. I wanted my audio effects to feel really unique. After all, I now had a super flexible and response development environment, and with Super Collider I was really just limited by my imagination. It turned out that was still a bit of a limitation!
I think I can do better here, but it's hard to give the audio effects the focus they deserve when I found it way too easy to procrastinate by working on the related visual effects.
What's next?
- rethink effects to be a bit more modular for easy swapping/sharing between effectdss
- deeper LLM integration. Effect creation/modification from within the app itself (or from a client web app)
--> use LLM to generate graphics/diagrams for the effects
--> set up an LLM to churn for longer autonomous sessions. I want them to explore the latent space of audio visual effects and then I can come and curate it.
--> I want a better way of sharing community effects. I've been using supabase more recently and it would be neat to integrate user login/effects/vector search.
—> knobs! the old pedal with 1/8” inputs and knobs is so cool. I miss the physical interface. Touchscreens are no fun for music.
Callouts!
Thanks Patrick & Brian for starting down this path with me. Lets go to NAMM next year!
Thanks Matthew for the inspiration and rescuing me from UX paralysis.
I borrowed heavily from ShaderToy shaders for a few visual effects. An LLM is exceptional at taking a shader toy shader and rewriting it in the specific flavors necessary to run in my app.
ShaderToy shaders were used for visual effect inspiration.
•Neon Love by alro: https://www.shadertoy.com/view/WdK3Dz •Oscilloscope by incription: https://www.shadertoy.com/view/slc3DX •Waves Remix by ADOB: https://www.shadertoy.com/view/4ljGD1