Check out my
new book!
HTML5 Games book
Music visualization with Canvas and SoundManager2 I'm not much of a Flash person and I guess I just hadn't been paying attention since I only found about the new audio features when Scott Schiller added support for them in his SoundManager2 JavaScript/Flash library and later posted a (very cool) favicon VU-meter that would just dance all night to the sound of your music. Since then I've been wanting to do something with those abilities since I figured canvas and live audio data would be a perfect match for some groovy audio visualization.

This is the first of hopefully many such experiments. It's a fun little music video of sorts, where I've just thrown in all sorts of things to the tune of some Radiohead. If you don't care about all the details, scroll down to the end for the link.

So, apparently Flash has a function called computeSpectrum which returns the current audio state, either as frequency data or as a waveform. In SM2 this has been split into two separate properties of the SMSound object (you have to specifically enable both with a setting and it's only available in Flash 9 mode). The data is available as 256-element arrays of values between 0 and 1.

For the frequency bars at the bottom, I've simply summed the values in 9 broader groups rather than paint the full spectrum. The same goes for the binary values on the left sides, they're simply averaged in 16 groups and then converted to binary. At the top of the screen you'll see the waveform, drawn at intervals of 8 for performance reasons.



The fun part is the flock of boids that is spawned at the beginning of the song. They are controlled by simple rules similar to those described by Craig Reynolds' original 1987 model for emergent flock behaviour (cohesion, separation, alignment) but will also start to react to the beat of the music. Every frame, the audio is analysed and if certain areas of the spectrum reaches a threshold limit a "pulse" is fired, making the boids attract each other and then repel each other again. When this happens, they also form grids and shapes that are drawn on the canvas. As the song progresses, the drawing takes the shape of Thom Yorke's lovely face (see the image at the beginning), but each playback is random and thus will produce a new drawing.

You can pause the whole thing by clicking on the main area. Clicking on the waveform at the top controls the playback position.

Naturally, this is only for canvas enabled browsers (no Internet Explorer). It runs a lot better for me in Chrome than any other browser. I have not tested it in Firefox 2 but Firefox 3 runs ok but slow (might be better on more powerful machines than mine) as does Opera 10. Safari 4 is fine as well although it stutters a bit at times. I'm sure there's plenty room for more performance improvements but that will have to wait for another time.

Another puzzling issue I came across: I get a security error from Flash's computeSpectrum when trying to run this if I also have a tab open with a YouTube video (probably some other Flash sites as well). Exact error from SM2: "(Flash): computeSpectrum() (waveform data) SecurityError: Error #2122". Happens in at least Firefox and Chrome and a bit of googling tells me it's a Flash issue.

I chose Radiohead because I'm a big fan and it just seemed like a nice place to start, given the Flash based visualization of their House of Cards "video" last year. And in case you're wondering, the track is Idioteque off of Kid A, taken from a 2001 performance in Paris that can be seen on YouTube here (just remember the tab issue I just mentioned).

I'm not aware of any other music visualization projects using JavaScript and canvas as its output medium and Flash as the "backend" (besides simple dancing bars and such), so I'd be very interested if you know of any.

Now have fun with the demo!

If you like this, you should definitely check out JuicyDrop, another music visualization project of mine (think Canvas meets Winamp).
⇓ 25 comments Paul Irish

Very impressive, Jacob. Especially great to see some of the flocking behavior that I've only seen thus far inside the Processing domain.

March 9, 2009 at 3:07 PM
raistooza

Rebirth of demoscene!
Awesome mashup :)

March 9, 2009 at 3:58 PM
Jacob Seidelin

Thanks!

@Steven Wittens: The thought had crossed my mind, actually. I've also been looking at MilkDrop, the files for that one would be pretty easy to parse.

March 9, 2009 at 4:04 PM
Dark This comment has been removed by the author. March 9, 2009 at 6:30 PM Dark

http://imgkk.com/

Click JS Demo in the footer. Watch. :P

(Yes, it is partly based on code from that Dynamic Hypnoglow thing)

And yes, I'll admit my algorithm is slightly less complex, but it gets the job done :P

March 9, 2009 at 6:34 PM
Anonymous

Artistic! One of the niftier uses of SM2's spectrum data I've seen, someone else made a visualizer similar to those seen in WinAmp a number of months back. Unfortunately 20 minutes of searching my email didn't dig it up, but it was pretty good. Maybe they read your blog too and will chime in. ;)

Re: YouTube videos, security errors etc.: With computeSpectrum(), it attempts to read data from all "source" SWFs outputting audio (presumably) at the time, and it can't read "low-level" data by default because of cross-domain security restrictions. Put another way, soundmanager2_flash9.swf on A.com may not access sound data coming from (or originating from?) a .SWF on youtube.com. This may seem a bit odd, but consider that you could potentially "steal" audio from other .SWFs, etc. As far as I know, there is no way to isolate and read only the local .SWF's audio output as the soundMixer object is a Flash system global.

March 9, 2009 at 7:20 PM
Jacob Seidelin

@Darkimmortal: Nice!

@Scott: Thanks for clearing up the security error issue a bit. It had me completely baffled until I somehow figured out that an open YT tab was causing the problem. I guess it makes sense, though, if the sound output isn't tied to the local SWF.

And do let me know if you find that visualizer :-)

March 10, 2009 at 1:53 AM
Anonymous

OK, after a bunch of email archive and Get Satisfaction searching, I eventually found it. It was a DJ/music project of sorts someone was working on, it came out of a SM2 discussion on frequency vs. waveform. Unfortunately it seems the site is down, at the time of writing at least. The developer is probably still around though and seemed like a cool guy, FWIW.

March 11, 2009 at 7:17 PM
Anonymous

@Jacob: Neat. I should really look into SoundManager2

@Darkimmortal: Saying " Yes, it is partly based on code from that Dynamic Hypnoglow thing " is kind of an understatement. I don't mind people using part or whole of of my code, especially when it's credited properly unlike here. Interesting use of the Dynamic Hypnoglow thing though.

March 12, 2009 at 5:44 AM
Jacob Seidelin

@Scott: Thanks for digging, too bad it's down. Maybe I'll see if I can contact him.

@Mathieu: Yes you should. Scott has taken all (well, most of) the 'argh' out of sound in the browser, even if you do have to swallow the fact that it's Flash.

March 12, 2009 at 8:36 AM
Anonymous

Jacob: I met P01 some time in 2002 (via the Ozone Asylum JS/DHTML forum) and while none of us are really big Flash fans, I've come to realize that there are a few areas where it has advantages and is widely available due to the 90-something-percentile install base. That plus ExternalInterface (allowing JS/Flash communication) make doing stuff from JS-land possible, which means it's an attractive potential option to DHTML fans like ourselves.

In the meantime I'd love to see HTML 5 audio and video grow to a point where they can replace the reliance on Flash, and ultimately can provide a similarly full-featured API (eg., allowing access to waveform data and so on.)

March 12, 2009 at 9:48 AM
Jacob Seidelin

Absolutely, I agree. And there is no Flash hate here - the right tool for the right job and all that - there are just some areas where it isn't/shouldn't be necessary to resort to plugin based technology.. Anyway, I suppose its widespread use has both upsides and downsides since it means that we do have that option for things like audio but it also makes it that much harder to replace. So, yea, I love your library but I'm also looking forward to the day that I won't need it!

March 12, 2009 at 10:24 AM
ooblogger

Hello,

please add your site at http://www.sweebs.com where other people can find you among the best sites on the internet!

Regards
Kris

March 13, 2009 at 2:15 AM
Anonymous

wow. it's really awesome.. wow~!

April 17, 2009 at 4:55 AM
Anonymous

Nice. I bet you can't get close to this though! http://www.noisecradle.com

September 9, 2009 at 8:35 AM
Prince_Porter

I've been messing with dong this sort of work in AS3 using FlashDevelop and Flex, it's a lot of fun. I've always been an audio guy, but never actually stuck with it, I've always stuck to programming and game design; working with audio in game design has definitely given me a place to be proud of both game design and audio. I definitely love this kind of stuff though, I'm eager to throw it into game design in a new way. Thanks for sharing, awesome stuff.

October 26, 2009 at 8:01 PM
Anonymous

Hi,
the links to Scott Schiller and SoundManager2 seem not to work?
Stefan

February 27, 2010 at 1:34 AM
Anonymous

Links work now. Something has been fixed.
Stefan

February 27, 2010 at 1:29 PM
Robert D. Thompson

Sorry Thom, very nice work and performs okay.

But JuicyDrop made things quite obvious...nothing near what flash ActionScript 3.0 can do, and if you haven't seen FlashPlayer 10.1 in action, you'll be amazed.

I've worked with JavaScript for years but most browsers don't have the time granularity.

For example, click on "The Royal We" at my site www.ActiveCommunity.com and you'll find that on most occasions it will end abruptly at the just right time,

I tried to put enough timing in there, first tried 400, 300, then all the way down to 50 milliseconds to check the exact time on a YouTube Chromeless player show I could show a layer that accentuated the Guitar-rough (you'll notice) but it never got quite the right timing.

I've been working on a YouTube Chromless player get better timing and Flash in General for Music and various Graph Theory (as HMM, Petri-Nets and Random Boolean Networks as well as Bayesian Probability) but it will take a little while as I have to do projects for clients to earn a living, but it'll get done.

~ Always a fan [Fake Plastic Trees came out at a time when I was particularly hurting and it really helped calm me. I made 3 sided triangles out of tooth-picks and pasted plasticine plastic onto each side like you get in binders for schools. Then I made a larger model made up of them and put a message on it but only if you put the puzzle together ... I felt a lot of love while making it.

May 7, 2010 at 9:14 AM
Anonymous

Hi, Jacob. I had emailed you about 14 months ago, interested in creating a music-less version of a 'boids drawing' canvas demonstration for classroom use. You replied, stating that you would consider re-releasing such if you found time & invited me to attempt modifying the code (to remove the music aspect) in the meantime.

Although I did try, earnestly (I understood your explanation of how the grayscale data is passed in, via an array) I failed to accomplish the task of removing the music 'signal' drawing dependency. I'm back begging you to consider creating a musicless demo and publishing it to your gallery.

May 11, 2010 at 10:06 PM
Anonymous

(same 'anonymous' as the comment immediately above this one)(I guess you can see IP address for each comment)

Finally. I have the app working without sound! The sticking point was my failure to recognize the significance of the settimeout call within init()

Now I need to figure out a workflow for extracting grayscale color data from additional images and populating those values into a js array

May 18, 2010 at 3:25 PM
Anonymous

fine tutorial

July 17, 2010 at 3:56 AM
mydigitalself

Something appears broken? Get: "No waveData - Check for open YouTube tabs!". Can hear the music, but the visualiser appears borked.

May 9, 2011 at 2:00 AM
Lucius

Splendid, but how did you calculate the data in data.js? I guess that is the controller of the boids that draw the picture?

September 19, 2011 at 6:33 AM
JNestudi

Again, How do you calculate the data in data.js? I love this work and I want to use something similar for a web but I want to change the image. I already changed the "img" but how can I give the data for canvas?

Thanks!

March 12, 2012 at 1:29 AM
Post a Comment