WebGL Music Visualizer
I'm not going to comment on the number of months since my last post here. I know I'm a slacker.
For most of December and some of January, I was travelling in East Africa. It's been about 5 years since I last visited there and it was fantastic to be back. Yay for sunburns and lions. Anyway, just before I left, I finished a neat little WebGL project that I never got around to posting here.
Google asked me to do a demo for when WebGL moved from Chrome dev channel to beta. The project I did was a music visualizer - yes, I'm that predictable. It's more like a visualizer building tool, actually, with a few pre-made visualizers. Well, WebGL landed in the beta channel on December 16th, the day after I left. Fortunately, I didn't have to do any remote debugging.
Have a look-see: WebGL Music Visualizer
To get started, click "Load preset", pick a visualizer and start the music. Elements in the scene can be modified by double clicking in the object list. Most parameters can take both constant values and expressions using math functions and/or pre-defined variables (time, beat, bass, mid, etc..).
What's also cool is that the visualizer was used at a Google-hosted party at the Game Developers Conference. I just had to modify it a bit to allow music input from the DJ's setup. Sadly, I wasn't there myself but I believe everything worked out great. I like to close my eyes and imagine that the Android was my representative there, so I was almost (almost!) at GDC.
Here are a few pics from the event:
Sweet!
Read more...
For most of December and some of January, I was travelling in East Africa. It's been about 5 years since I last visited there and it was fantastic to be back. Yay for sunburns and lions. Anyway, just before I left, I finished a neat little WebGL project that I never got around to posting here.
Google asked me to do a demo for when WebGL moved from Chrome dev channel to beta. The project I did was a music visualizer - yes, I'm that predictable. It's more like a visualizer building tool, actually, with a few pre-made visualizers. Well, WebGL landed in the beta channel on December 16th, the day after I left. Fortunately, I didn't have to do any remote debugging.
Have a look-see: WebGL Music Visualizer
To get started, click "Load preset", pick a visualizer and start the music. Elements in the scene can be modified by double clicking in the object list. Most parameters can take both constant values and expressions using math functions and/or pre-defined variables (time, beat, bass, mid, etc..).
What's also cool is that the visualizer was used at a Google-hosted party at the Game Developers Conference. I just had to modify it a bit to allow music input from the DJ's setup. Sadly, I wasn't there myself but I believe everything worked out great. I like to close my eyes and imagine that the Android was my representative there, so I was almost (almost!) at GDC.
Here are a few pics from the event:
Sweet!
Read more...
WebGL and HTML5 audio fixes
A couple of updates and fixes to some of my earlier WebGL and HTML5 audio experiments which have stopped working since I first put them online.
First up is the musical solar system which went and got horribly broken due to changes in the WebGL spec. I published it when the WebGL implementations were still very much evolving and changes were happening almost faster than you adapt to them.Now with the spec having reached a more mature level, things have slowed down and the implementations are almost in agreement so browser-specific workarounds aren't so necessary. So anyway, after a few tweaks here and there, the planets should once again be dancing. Check it out.
Next up, Worlds of WebGL which had suffered pretty much the same fate. I'd already updated it once to accommodate some of the spec changes and now it should be fully working again.
However... In both WebGL projects, I had to reintroduce some code I had previously taken out because it was causing problems for some Mac users (iirc). So if you experience any problems with either experiment, please let me know. In particular, the affected parts would be the sun in the solar system and all the particles/planets in Worlds of WebGL.
Finally, my HTML5 audio project Pocket Full of HTML5 had lost its support of the new HTML5 audio data api, so it used the SoundManager2 fallback even when using a audio-data-enabled Firefox build. Again, the problem was due to changes in the API, which is still very early in its development, but fortunately the necessary changes were few and simple and Yury Delendik even provided me with a patch (thanks Yuri!). So check that out too.
Read more...
A couple of updates and fixes to some of my earlier WebGL and HTML5 audio experiments which have stopped working since I first put them online.First up is the musical solar system which went and got horribly broken due to changes in the WebGL spec. I published it when the WebGL implementations were still very much evolving and changes were happening almost faster than you adapt to them.Now with the spec having reached a more mature level, things have slowed down and the implementations are almost in agreement so browser-specific workarounds aren't so necessary. So anyway, after a few tweaks here and there, the planets should once again be dancing. Check it out.
Next up, Worlds of WebGL which had suffered pretty much the same fate. I'd already updated it once to accommodate some of the spec changes and now it should be fully working again.
However... In both WebGL projects, I had to reintroduce some code I had previously taken out because it was causing problems for some Mac users (iirc). So if you experience any problems with either experiment, please let me know. In particular, the affected parts would be the sun in the solar system and all the particles/planets in Worlds of WebGL.
Finally, my HTML5 audio project Pocket Full of HTML5 had lost its support of the new HTML5 audio data api, so it used the SoundManager2 fallback even when using a audio-data-enabled Firefox build. Again, the problem was due to changes in the API, which is still very early in its development, but fortunately the necessary changes were few and simple and Yury Delendik even provided me with a patch (thanks Yuri!). So check that out too.
Read more...
HTML5 audio visualizations
For a while now I've been playing around with music visualization, you can see some of my previous endeavours here. The canvas element gives us everything we need to draw all sorts of cool stuff in the browser but the audio part still needs Flash to be any fun. While the new HTML5 audio element does allow for audio playback, it doesn't give us much in terms of audio data information - which is crucial if the visualizations are to react to the music. In another experiment of mine I tried to solve it by doing a pre-analysis of the MP3 file and saving the data as JavaScript. The result was decent, I think, but having to pre-bake the data is a pretty severe limitation.
However, a few people have been hacking away at Firefox, attempting to provide just that functionality for the audio element. Some documentation of the new API is available here and if you're interested, you can follow the development in this bugzilla thread.
Anyway, I've been wanting to mess around with this and I figured that retro-fitting my "Pocket Full of Canvas" experiment to use HTML5 audio would be a fairly easy task. You can check out the result here.
At the moment you'll need a patched version of Firefox to get access to these extras but if you're not comfortable with building your own version, there are builds available here. It will still run in other (canvas-enabled) browsers but it will fall back to using SoundManager2 for the audio. While Flash audio is not that exciting anymore, it lets it run in eg. Chromium and Opera - and man, that's something else. Even if Firefox/Minefield has the upper hand with the audio stuff, it is feeling reaaally sluggish compared to those two (and you should definitely be using a nightly Minefield build). Chromium has always been speedy and Opera has gotten really crazy fast as well, running the visualizations in full-screen is just great (click the viz area to toggle).
And if you feel like playing around with the demos yourself, just click the "Show code" button and start hacking. I made a quick write-up of the very simple API here. The emphasis has been on making it easy to get from zero to something and not so much on making a lot of fancy functionality, so it is not very advanced.
Try it out: http://www.nihilogic.dk/labs/pocket_full_of_html5/
More examples:
Read more...
For a while now I've been playing around with music visualization, you can see some of my previous endeavours here. The canvas element gives us everything we need to draw all sorts of cool stuff in the browser but the audio part still needs Flash to be any fun. While the new HTML5 audio element does allow for audio playback, it doesn't give us much in terms of audio data information - which is crucial if the visualizations are to react to the music. In another experiment of mine I tried to solve it by doing a pre-analysis of the MP3 file and saving the data as JavaScript. The result was decent, I think, but having to pre-bake the data is a pretty severe limitation.
However, a few people have been hacking away at Firefox, attempting to provide just that functionality for the audio element. Some documentation of the new API is available here and if you're interested, you can follow the development in this bugzilla thread.
Anyway, I've been wanting to mess around with this and I figured that retro-fitting my "Pocket Full of Canvas" experiment to use HTML5 audio would be a fairly easy task. You can check out the result here.
At the moment you'll need a patched version of Firefox to get access to these extras but if you're not comfortable with building your own version, there are builds available here. It will still run in other (canvas-enabled) browsers but it will fall back to using SoundManager2 for the audio. While Flash audio is not that exciting anymore, it lets it run in eg. Chromium and Opera - and man, that's something else. Even if Firefox/Minefield has the upper hand with the audio stuff, it is feeling reaaally sluggish compared to those two (and you should definitely be using a nightly Minefield build). Chromium has always been speedy and Opera has gotten really crazy fast as well, running the visualizations in full-screen is just great (click the viz area to toggle).
And if you feel like playing around with the demos yourself, just click the "Show code" button and start hacking. I made a quick write-up of the very simple API here. The emphasis has been on making it easy to get from zero to something and not so much on making a lot of fancy functionality, so it is not very advanced.
Try it out: http://www.nihilogic.dk/labs/pocket_full_of_html5/
More examples:
Pocket Full of Canvas
One thing I found interesting when I did the JuicyDrop music visualization was MilkDrop's deformation effects. Rather than processing deformations for each and every pixel it works on a grid of points and then just interpolates the results for the actual pixels. I sort of mimicked that in JuicyDrop but in a simpler way. The grids used in JuicyDrop are something like 5x5 to 9x9 where MilkDrop uses much higher resolution grids and instead of doing per-pixel interpolation, the grid points are used to cut out triangles from the previous frame and paint them on the new frame, slightly transformed.
Since the deformations are usually very small when seen on a frame-by-frame basis, you can get some pretty good results even with fairly low resolution grids, and most recent browsers are more than capable of rendering 100 or even more triangles on a canvas. In the end, I was pleasantly surprised at how well everything turned out since I wasn't even sure I was going to get anything remotely close to what the original MilkDrop plugin produced.
While I still plan on doing some more work on JuicydDrop eventually, I decided to rip out just the grid deformation part and built something new around it. You see, every now and then I get the urge to just throw something quick together and make some flashing light or dancing balls or whatever but usually that urge comes when I only have 30 minutes to spare. So I figured I'd try to build a mini framework for making stupid demo effects and stuff like that.
I've ended up with a small application that loads simple scripts, exposes a bunch of functions to these scripts and then takes care of rendering and processing whatever the script tells it to. It's probably best explained by just taking a look at it. The whole idea was to make it as easy as possible for me to just throw some silly effect together real quick and hopefully not write too much code in the process, so it might not be the most thought through design but it gets the job done. The functions available range from basic drawing and image processing (via Pixastic) to audio data and 3D (via Pre3D).
There's no larger goal with this and there are already more robust and more elaborate frameworks out there for programming and animating graphics with JavaScript/Canvas(Processing.js for instance) so this is just my own little time-sink. You're of course welcome to play around with it, modify the existing scripts or even make your own.
To wrap things up, I made a little a demo comprised of scripts I cooked up while testing and developing this thing (as well as a few adaptations of other people's work). I totally recommend using Chrome and if possible, the dev channel as it's given me by far the best performance and visual appearance.
Watch the demo here
Play around with the application here
Read more...
One thing I found interesting when I did the JuicyDrop music visualization was MilkDrop's deformation effects. Rather than processing deformations for each and every pixel it works on a grid of points and then just interpolates the results for the actual pixels. I sort of mimicked that in JuicyDrop but in a simpler way. The grids used in JuicyDrop are something like 5x5 to 9x9 where MilkDrop uses much higher resolution grids and instead of doing per-pixel interpolation, the grid points are used to cut out triangles from the previous frame and paint them on the new frame, slightly transformed.
Since the deformations are usually very small when seen on a frame-by-frame basis, you can get some pretty good results even with fairly low resolution grids, and most recent browsers are more than capable of rendering 100 or even more triangles on a canvas. In the end, I was pleasantly surprised at how well everything turned out since I wasn't even sure I was going to get anything remotely close to what the original MilkDrop plugin produced.
While I still plan on doing some more work on JuicydDrop eventually, I decided to rip out just the grid deformation part and built something new around it. You see, every now and then I get the urge to just throw something quick together and make some flashing light or dancing balls or whatever but usually that urge comes when I only have 30 minutes to spare. So I figured I'd try to build a mini framework for making stupid demo effects and stuff like that.
I've ended up with a small application that loads simple scripts, exposes a bunch of functions to these scripts and then takes care of rendering and processing whatever the script tells it to. It's probably best explained by just taking a look at it. The whole idea was to make it as easy as possible for me to just throw some silly effect together real quick and hopefully not write too much code in the process, so it might not be the most thought through design but it gets the job done. The functions available range from basic drawing and image processing (via Pixastic) to audio data and 3D (via Pre3D).
There's no larger goal with this and there are already more robust and more elaborate frameworks out there for programming and animating graphics with JavaScript/Canvas(Processing.js for instance) so this is just my own little time-sink. You're of course welcome to play around with it, modify the existing scripts or even make your own.
To wrap things up, I made a little a demo comprised of scripts I cooked up while testing and developing this thing (as well as a few adaptations of other people's work). I totally recommend using Chrome and if possible, the dev channel as it's given me by far the best performance and visual appearance.
Watch the demo here
Play around with the application here
The Dancing Scrollbars!
I had an hour to kill and got inspired by this entry on the Chrome Experiments site. Andrey has made a nice wave simulator using scrollbars and some JavaScript magic. Taking that idea and mixing it up with what I've been doing lately (see here and here), here is the magnificent troupe of Dancing Scrollbars (ok, they're more like the Jumping Scrollbar handles, really).
The music is Tom Cruise Crazy by Jonathan Coulton.
Have fun!
And now I'm totally done with this. For now. Read more...
I had an hour to kill and got inspired by this entry on the Chrome Experiments site. Andrey has made a nice wave simulator using scrollbars and some JavaScript magic. Taking that idea and mixing it up with what I've been doing lately (see here and here), here is the magnificent troupe of Dancing Scrollbars (ok, they're more like the Jumping Scrollbar handles, really).
The music is Tom Cruise Crazy by Jonathan Coulton.
Have fun!
And now I'm totally done with this. For now. Read more...
JavaScript + Canvas + SM2 + MilkDrop = JuicyDrop
More canvas music visualization - now with 100% more Winamp-iness.
A couple of weeks ago I played around with music visualization using JavaScript/canvas and SoundManager2. Well, I couldn't leave it at that and as I mentioned in the comments, I had an eye on the MilkDrop plugin for Winamp. The result so far is a little Winamp lookalike called JuicyAmp with its own music visualizer JuicyDrop that feeds on Milkdrop preset files.
If you just want to see the pretty colors -> CLICKY. (But please use Chrome or Firefox 3+)
MilkDrop is nice because, although there's a built-in editor in the plugin, the presets are in plain text. They are basically just lists of variables and equations that, with a bit of mangling, can be evaluated as JavaScript. There are also extensive guides that explain how to author presets and how variables are passed around between the different equations. And, even better, the source code for the plugin was released a couple of years ago.
MilkDrop presets consist of a number of different elements (waveforms, shapes, per-pixel effects, etc). Some of them I haven't touched at all, but JuicyDrop supports enough at the moment that a good handful of presets run just fine. That said, there are a whole bunch of problems to work out and the presets included were selected because they looked alright and didn't make it blow up.
I strongly recommend using Chrome for this. Firefox 3 can play too but is probably somewhat slower. There's something screwy going on with Safari, it's like it's refusing to update the display (try holding down a key on your keyboard) (Edit: at least a few people have reported that Safari is working fine for them, YMMV), I'm not sure what exactly is causing it. Opera is a mixed experience for me and it seems to have a problem with playing the music that I haven't found the reason for yet.
The issue with open Flash/sound-using tabs, etc. is of course still here as well (read the comments here for more).
A couple of keyboard controls:
So without further ado, go have yourself a canvas trip.
I'm not sure where to go from here (besides a lot of optimizing). I'm not done with this, but I'm not sure I want to try to get full MilkDrop support. I believe the presets of today use a lot of pixel shaders anyway, which obviously is no good here. What I might do, though, is add another preset format (probably JSON?) that's a bit easier to work with and then shape that into something more suited for canvas and JavaScript. But for now I have a date with Pixastic..
Read more...
More canvas music visualization - now with 100% more Winamp-iness.
A couple of weeks ago I played around with music visualization using JavaScript/canvas and SoundManager2. Well, I couldn't leave it at that and as I mentioned in the comments, I had an eye on the MilkDrop plugin for Winamp. The result so far is a little Winamp lookalike called JuicyAmp with its own music visualizer JuicyDrop that feeds on Milkdrop preset files.
If you just want to see the pretty colors -> CLICKY. (But please use Chrome or Firefox 3+)
MilkDrop is nice because, although there's a built-in editor in the plugin, the presets are in plain text. They are basically just lists of variables and equations that, with a bit of mangling, can be evaluated as JavaScript. There are also extensive guides that explain how to author presets and how variables are passed around between the different equations. And, even better, the source code for the plugin was released a couple of years ago.
MilkDrop presets consist of a number of different elements (waveforms, shapes, per-pixel effects, etc). Some of them I haven't touched at all, but JuicyDrop supports enough at the moment that a good handful of presets run just fine. That said, there are a whole bunch of problems to work out and the presets included were selected because they looked alright and didn't make it blow up.
I strongly recommend using Chrome for this. Firefox 3 can play too but is probably somewhat slower. There's something screwy going on with Safari, it's like it's refusing to update the display (try holding down a key on your keyboard) (Edit: at least a few people have reported that Safari is working fine for them, YMMV), I'm not sure what exactly is causing it. Opera is a mixed experience for me and it seems to have a problem with playing the music that I haven't found the reason for yet.
The issue with open Flash/sound-using tabs, etc. is of course still here as well (read the comments here for more).
A couple of keyboard controls:
- Z : switch to smaller (128x128) visualization view (in case of low framerate)
- X : switch back to normal (256x256) view
- D : Toggle rendering of deformation mesh points
- 1 : Toggle basic waveform
- 2 : Toggle custom waves
- 3 : Toggle custom shapes
- 4 : Toggle borders
- 5 : Toggle per-pixel effects
- 6 : Toggle video echo
So without further ado, go have yourself a canvas trip.
I'm not sure where to go from here (besides a lot of optimizing). I'm not done with this, but I'm not sure I want to try to get full MilkDrop support. I believe the presets of today use a lot of pixel shaders anyway, which obviously is no good here. What I might do, though, is add another preset format (probably JSON?) that's a bit easier to work with and then shape that into something more suited for canvas and JavaScript. But for now I have a date with Pixastic..
Read more...
Music visualization with Canvas and SoundManager2
I'm not much of a Flash person and I guess I just hadn't been paying attention since I only found about the new audio features when Scott Schiller added support for them in his SoundManager2 JavaScript/Flash library and later posted a (very cool) favicon VU-meter that would just dance all night to the sound of your music. Since then I've been wanting to do something with those abilities since I figured
This is the first of hopefully many such experiments. It's a fun little music video of sorts, where I've just thrown in all sorts of things to the tune of some Radiohead. If you don't care about all the details, scroll down to the end for the link.
So, apparently Flash has a function called
For the frequency bars at the bottom, I've simply summed the values in 9 broader groups rather than paint the full spectrum. The same goes for the binary values on the left sides, they're simply averaged in 16 groups and then converted to binary. At the top of the screen you'll see the waveform, drawn at intervals of 8 for performance reasons.
The fun part is the flock of boids that is spawned at the beginning of the song. They are controlled by simple rules similar to those described by Craig Reynolds' original 1987 model for emergent flock behaviour (cohesion, separation, alignment) but will also start to react to the beat of the music. Every frame, the audio is analysed and if certain areas of the spectrum reaches a threshold limit a "pulse" is fired, making the boids attract each other and then repel each other again. When this happens, they also form grids and shapes that are drawn on the canvas. As the song progresses, the drawing takes the shape of Thom Yorke's lovely face (see the image at the beginning), but each playback is random and thus will produce a new drawing.
You can pause the whole thing by clicking on the main area. Clicking on the waveform at the top controls the playback position.
Naturally, this is only for
Another puzzling issue I came across: I get a security error from Flash's
I chose Radiohead because I'm a big fan and it just seemed like a nice place to start, given the Flash based visualization of their House of Cards "video" last year. And in case you're wondering, the track is Idioteque off of Kid A, taken from a 2001 performance in Paris that can be seen on YouTube here (just remember the tab issue I just mentioned).
I'm not aware of any other music visualization projects using JavaScript and
Now have fun with the demo!
If you like this, you should definitely check out JuicyDrop, another music visualization project of mine (think Canvas meets Winamp). Read more...
I'm not much of a Flash person and I guess I just hadn't been paying attention since I only found about the new audio features when Scott Schiller added support for them in his SoundManager2 JavaScript/Flash library and later posted a (very cool) favicon VU-meter that would just dance all night to the sound of your music. Since then I've been wanting to do something with those abilities since I figured canvas and live audio data would be a perfect match for some groovy audio visualization.
This is the first of hopefully many such experiments. It's a fun little music video of sorts, where I've just thrown in all sorts of things to the tune of some Radiohead. If you don't care about all the details, scroll down to the end for the link.
So, apparently Flash has a function called
computeSpectrum which returns the current audio state, either as frequency data or as a waveform. In SM2 this has been split into two separate properties of the SMSound object (you have to specifically enable both with a setting and it's only available in Flash 9 mode). The data is available as 256-element arrays of values between 0 and 1.
For the frequency bars at the bottom, I've simply summed the values in 9 broader groups rather than paint the full spectrum. The same goes for the binary values on the left sides, they're simply averaged in 16 groups and then converted to binary. At the top of the screen you'll see the waveform, drawn at intervals of 8 for performance reasons.
You can pause the whole thing by clicking on the main area. Clicking on the waveform at the top controls the playback position.
Naturally, this is only for
canvas enabled browsers (no Internet Explorer). It runs a lot better for me in Chrome than any other browser. I have not tested it in Firefox 2 but Firefox 3 runs ok but slow (might be better on more powerful machines than mine) as does Opera 10. Safari 4 is fine as well although it stutters a bit at times. I'm sure there's plenty room for more performance improvements but that will have to wait for another time.Another puzzling issue I came across: I get a security error from Flash's
computeSpectrum when trying to run this if I also have a tab open with a YouTube video (probably some other Flash sites as well). Exact error from SM2: "(Flash): computeSpectrum() (waveform data) SecurityError: Error #2122". Happens in at least Firefox and Chrome and a bit of googling tells me it's a Flash issue.I chose Radiohead because I'm a big fan and it just seemed like a nice place to start, given the Flash based visualization of their House of Cards "video" last year. And in case you're wondering, the track is Idioteque off of Kid A, taken from a 2001 performance in Paris that can be seen on YouTube here (just remember the tab issue I just mentioned).
I'm not aware of any other music visualization projects using JavaScript and
canvas as its output medium and Flash as the "backend" (besides simple dancing bars and such), so I'd be very interested if you know of any.Now have fun with the demo!
If you like this, you should definitely check out JuicyDrop, another music visualization project of mine (think Canvas meets Winamp). Read more...
