About nine years ago I built a modular synthesiser in the browser. It was an experiment built on Angular 4 as a way to play around with the Web Audio API. It didn't get very far, it was just a side project. Three weeks ago I pointed Claude Code at it. It's now far more fun to play with than I'd ever have made it as a side project.
What I didn't expect was finding out how well Claude Code understands music. Give it a text format for sound and it writes things that sound musical, in the right genre, with the right parts. It can't hear what it makes, but it can measure it and look at it, and that gets it further than I expected.
I showed this at the annual Async International Show and Tell meetup in Brighton last night.
A side-project from pre-AI days
Tone Lab is a modular patcher for the Web Audio API. I built it in 2017 when I realised there was a modular synthesiser hiding inside every browser. You add modules to the page, drag cables between them and make noises. I built it with Angular 4 and Tone.js.
The other synth is the Mi-03 Bassline, a C++ audio plugin inspired by the TB-303. It took years of learning digital signal processing, research, and three years of building, and I released it in April 2024.
A toy Mi-03 from one prompt
I started with a throwaway experiment. I opened Claude Code in an empty folder and typed this:
make a cool looking music page using web audio that uses my mi03 bassline synth as inspiration.
in the future I might compile that C++ to wasm but for now just use web audio.
Needs to have a step sequencer bass synth with filter, resonance, mod, and attack controls (inspired but simplified 303) with a simple step sequencer for acid bass line pattern. and then a 909 sounding simple drum machine pattern sequencer.
all on one standalone html page please
The folder was empty, so it went looking. It ran find ~/dev -iname '*mi03*', found the template for the Mi-03's
product page, then looked at a screenshot and read the manual. It never found the C++ source, and it didn't need it. The
look and the sequencer design came from the product page, one screenshot and the manual. (It also read files outside the
project folder without asking, which I hadn't expected.)
The result was surprisingly good. Have a play with it. It was good enough to make me take Web Audio seriously as a platform. What impressed me most was the initiative it took in going off to try and find details of the synth on my machine. What shocked me was how well it rebuilt my UI and design look and feel without having access to the source code, just from reading the manual and looking at a screenshot of the synth plugin.
Tone Lab, back from the dead
If one prompt could do that, what about Tone Lab?
I made this thing a long time ago and it's really out of date: https://github.com/tone-lab/tone-lab
It's on an old version of angular. Can you make it a standalone file like the other synth in this repo?
Use a dynamic workflow to research plan and implement efficiently.
It worked...
It fixed the bugs I had struggled with in the cable routing interaction. And made the cards drag and drop to reorder.
But it didn't look great to my eye, so I asked for a better design and layout for the modules...
Patches are just text. Can we use this to unlock Claude's musical expression?
Tone Lab was now able to export and import patches as JSON. Claude Code is good with JSON. So I wondered whether it could write a patch.
I asked for drum and bass. What came back was stereotypical drum and bass, and that's why it worked. There was no mistaking the genre.
It knows how the genre is put together. Here's the patch it wrote: 174 BPM, kick and snare on a trigger grid, quiet ghost snares in between, hats thinned out by a chance module, and a Reese bass (detuned saws) through drive and a filter with a slow LFO to keep it moving and a sine sub bass layered underneath.
"How did you do that?"
I'd just watched a coding agent write drum and bass. So I asked it how it did that.
The safeguards flagged my question as an attempt to extract its reasoning.
So I asked a different way. Instead of asking it to explain itself, I asked it to write a skill: instructions another session could follow to make a Tone Lab patch. The skill sets out the process:
- Settle the genre, tempo and modules to feature.
- Look up each module's parameters and ports, with no guessing.
- Write the patch as JSON.
- Check that it loads cleanly, and that every part reaches the mixer.
- Render it, and level each channel against similar patches.
The last step is the interesting one. It renders the patch to audio, measures each channel, and compares the levels with existing patches that already sound right. Measuring alone got the levels mostly right. Some patches still needed a tweak by ear.
It can't hear, but it can see
The cymbals show this best. The first crash cymbal it made sounded like static. Claude Code can't listen to a sound, but it can render one to a file, analyse it with tools, and look at a picture of it.
A spectrogram is that picture: pitch up the side, time across, brighter for louder. Here's the crash before the fix, after it, and a recording of a real crash for comparison.
Before, it was noise fading evenly at every pitch. After, it's 200 partials spread evenly in Hz, with the highs dying first. That's the shape of the real thing.
Still not perfect, but a massive improvement, and usable. The first attempt, made without a reference, wasn't.
This is the bit that changed how I think about it. Claude Code doesn't need ears to make a sound better. It needs a way to turn the sound into something it can measure or see, and a reference to aim for.
More features, more glitches
With patches flowing, I kept adding things: Turing Machine, Euclidean, Resonator, Reverb and Trigger Grid modules, more demo patches, frequency quantising, arrangement tools, seeking, and undo and redo.
The more I added, the more the audio glitched. The fix was to rebuild the audio engine around an AudioWorklet, so all the sound is made in one place on the browser's audio thread.
Claude picks JavaScript, I pick Rust
I had Claude Code prototype the new engine twice: once in JavaScript, and once in Rust compiled to WebAssembly. Both sounded good. A parity tool compared their output, and they matched within -144 dB. That's about the smallest difference 24-bit audio can represent, so as far as any ear is concerned they were identical.
Rust was faster in every browser:
| Browser | Rust compared with JavaScript |
|---|---|
| Chrome | 1.4 to 1.9× faster |
| Firefox | 3.2 to 3.9× faster |
| WebKit | 1.5 to 2× faster |
In Firefox, the stress-test patch fell from 11.8% to 3.0% of a CPU core.
Claude Code still recommended JavaScript. Here are its reasons, and my answers:
| Claude's reason | My answer |
|---|---|
| 294 KB of wasm against 53 KB of JS | Small for an audio app, and 111 KB gzipped |
| A panic stops the whole engine | Good: a failure you can hear gets fixed |
| Glue code between JS and Rust | Written once, and the messages are stable |
| Harder to debug inside the worklet | So test the engine outside it, with cargo test |
| Seeking is no faster in Chrome | No slower either, and over twice as fast in Firefox |
None of them were hard to answer, and speed wasn't even my main reason. I love Rust, and a Rust engine isn't stuck in
the browser. It runs on the command line, under cargo test, and it can render a patch offline to a file. The speed
gives me headroom for more ambitious DSP later.
Spikes before the big build
Before rebuilding everything, I ran a few more spikes.
For the design, Claude Code mocked up 35 directions. I picked one, refined it a bit, and then had it built a design system.
I also got it to prototype some solutions to the biggest UX problem that made large patches hard to read. For this "cable soup" (the tangle you get once a patch grows past a handful of modules) it tried a dozen approaches: quieter cables, routed cables, net labels, groups, folded modules and an overview map.
For the backend, I spec'ed out a Rust service to save and load patches and render them offline.
All of that went into markdown docs, reference files, and a list of todos I had been maintaining.
Twelve hours unattended
Then I handed it over with one prompt:
use a dynamic workflow to progress the work described in task 2 of @docs/todo.md
launch multiple dynamic workflows in series to complete the work.
commit after each batch of work to keep a log but make as much progress as possible without getting stuck in endless testing and verification.
we can do more intense testing at the end, for now progress using multiple workflows as much as possible.
toward getting a working v2 rust Tone Lab that exceeds where we got to with v1 (but backward compatibility is not required).
| Time | Workflows | Agents | Tokens written | |
|---|---|---|---|---|
| Session 1 | 7h 08m | 4 | 48 | 3.7M |
| Session 2 | 4h 45m | 1 | 25 | 1.8M |
| Total | 11h 53m | 5 | 73 | 5.5M |
There are two sessions because the first one crashed. I think it was a macOS kernel panic when the machine ran out of memory. I restarted it and let it carry on.
When it finished, the new version worked and sounded great. A bit of clicking around turned up some issues to fix. I browsed the code to get an overview, mostly out of curiosity about how it worked. This is a side project, and I don't feel any urge to read and understand every line.
Where it's at
The new Tone Lab sounds much better, and it's a much better foundation to build on. There are bugs, and there's plenty more to do. I'll keep working on it.