House of Canvas - Radiohead meets JavaScript
You may know that Radiohead has gone and done the awesome thing of making a music video using not cameras but LASERS and DATA! Rock! Using neat tricks and technology, they made a visualization of 3D data captured by their magic laser thing and turned it into a music video for their House of Cards song. And to make it even awesomer, they released some of the data that went into making this video on Google Code!.
I figured it would be a nice little experiment to try visualizing this data using JavaScript and Canvas so I went and did just that. The data is simply point clouds, meaning a whole bunch of points with x,y,z values (and intensity) for each frame. The data on Google Code is about 800 MB, so obviously a bit of trimming had to be done. You can't expect 30 fps with Javascript and this kind of data, so I've only used every 5 frames giving us a framerate of 6 fps, not great but acceptable. Then the actual points, each frame has more than 6,000 points.No way this will render with 6 fps in any browserYou need a good deal of power and/or Opera to get that rendering at 6+ fps, so again I've taken only 10% of the points. Additionally, I've tried to filter away the noise around Thom Yorke's head since that took up a good deal of points. The interesting bit is him singing, anyway. In the end, we have a dataset of about 4 MB (converted to a JS array) for the one minute clip they released.
Now the data is in a more manageable state and the visualization can begin. It's not as good as the real thing, obviously, but I think it's ok (it's best when you look at Thom in profile). The audio clip is as usual played via SoundManager 2 which also gives us free timing information to sync the rendering to. I've played around and made a few different effects that you can toggle on and off (by pressing keys 1-9). While it is playing you can also rotate around the vertical axis by moving the mouse horizontally over the video. Also try clicking/doubleclicking.
If you're interested in the JavaScript code, it's right here. Note that since it uses the Canvas element, it won't work in Internet Explorer!
Edit: I removed effect 3 (xor) since it wasn't working so well anymore and replaced it with some intensity coloring that looks more like what is used in the real video.
Edit 2: A new, high-res version has been posted. It's only one frame, so no animation (also no music), but it uses all the points from the dataset and now you can actually see that it's Thom Yorke. Since we're dealing with so many points, don't expect it to rotate smoothly. I'm only getting something like 2 fps with FF2.
Edit 3: You can now press w and s to zoom in and out, respectively.
Check it out here
You may know that Radiohead has gone and done the awesome thing of making a music video using not cameras but LASERS and DATA! Rock! Using neat tricks and technology, they made a visualization of 3D data captured by their magic laser thing and turned it into a music video for their House of Cards song. And to make it even awesomer, they released some of the data that went into making this video on Google Code!.
I figured it would be a nice little experiment to try visualizing this data using JavaScript and Canvas so I went and did just that. The data is simply point clouds, meaning a whole bunch of points with x,y,z values (and intensity) for each frame. The data on Google Code is about 800 MB, so obviously a bit of trimming had to be done. You can't expect 30 fps with Javascript and this kind of data, so I've only used every 5 frames giving us a framerate of 6 fps, not great but acceptable. Then the actual points, each frame has more than 6,000 points.
Now the data is in a more manageable state and the visualization can begin. It's not as good as the real thing, obviously, but I think it's ok (it's best when you look at Thom in profile). The audio clip is as usual played via SoundManager 2 which also gives us free timing information to sync the rendering to. I've played around and made a few different effects that you can toggle on and off (by pressing keys 1-9). While it is playing you can also rotate around the vertical axis by moving the mouse horizontally over the video. Also try clicking/doubleclicking.
If you're interested in the JavaScript code, it's right here. Note that since it uses the Canvas element, it won't work in Internet Explorer!
Edit: I removed effect 3 (xor) since it wasn't working so well anymore and replaced it with some intensity coloring that looks more like what is used in the real video.
Edit 2: A new, high-res version has been posted. It's only one frame, so no animation (also no music), but it uses all the points from the dataset and now you can actually see that it's Thom Yorke. Since we're dealing with so many points, don't expect it to rotate smoothly. I'm only getting something like 2 fps with FF2. Edit 3: You can now press w and s to zoom in and out, respectively.
Check it out here
No. Freaking. Way.
July 16, 2008 at 12:01 PM RyanNothing comes up for me... There is no "Load".
July 16, 2008 at 1:33 PM Aaron HeckmannAwesome and Inspiring, once again.
July 16, 2008 at 1:36 PM AnonymousGreat work!
July 16, 2008 at 1:45 PM AnonymousAwesome. U rock
July 16, 2008 at 8:04 PM AnonymousThis is too much!
July 16, 2008 at 11:12 PM Jacob SeidelinThanks for the nice comments, all.
July 17, 2008 at 3:11 AM Anonymous@ryan: If you're using IE, then that's the problem. I forgot the usual "Canvas = No IE support" disclaimer.
Groovy!!
July 17, 2008 at 7:12 AM Jacob SeidelinJust looked at the code and saw room for optimizations in the render() method e.g. caching variables, using LUTs, preprocessing/modifying the point cloud, reduced branching.
I'd like to give a shot at these to see if my intuitions are correct.
Thanks. Sure, go ahead. I just updated the script and also added a few optimizations but I'm sure you can tweak it further!
July 17, 2008 at 7:46 AM Clarityin FF3 on windows I see nothing? no Load...
July 17, 2008 at 7:49 AM Jacob Seidelin@clarity: Hmm. Works fine for me.. Try reloading the page?
July 17, 2008 at 8:17 AM AnonymousAnyone else having problems with FF3?
Works perfectly un the current Firefox trunk build.
July 17, 2008 at 10:18 AM AnonymousYou said you were delivering the data as a (probably gzipped) JS Array... have you ever thought about embedding it in an image instead and then reading the individual bytes via canvas? If you'd encode it as a 128x128x8bpp image (in other words 7bit in every direction if we use value as Z and 1 bit for visibility) you'd come in at about 15 meg uncompressed for a three minute clip, no matter how unclean the source material is. Nice side effect would be that it's server-agnostic.
Ok, I hacked together an optimized version of House of Canvas based on your HiRes data set of 6485 dots. It runs at ~15 fps in Opera 9.51.
July 17, 2008 at 4:38 PM Jacob SeidelinOrdering the dots by angle would eliminate the occasional front to back overdraws at little to no cost.
@mathieu: Cool! I'm afraid I won't have any fun time until after the weekend (am out of town) but I'll take a look at what you did then. Looks like Opera likes this type of thing :)
July 18, 2008 at 1:50 AM AnonymousThis is just awesome. Thumbs up.
July 18, 2008 at 2:46 AM AnonymousOpera 9.51. Nothing.
July 18, 2008 at 11:56 AM AnonymousTurns out that pre-sorting the point cloud by angle, optimizes the rendering a little and cut down the size of the data by ~25%
July 19, 2008 at 12:49 PM AnonymousSee the URL I pasted above.
Can't you cut your big data string into N chunks and reuse your PNG compression technique to convert chunks to PNGs. Then hopefully your browser downloads images in parallel - if necessary from different (virtual) hosts to overcome the parallel-connections limit of the browser - and you stitch the chunks back together using Array.join("") or plain string concatenation. Worth trying out, or not? I'd be interested whether it works on Apple's Safari for Windows, in particular.
July 22, 2008 at 6:52 AM Jacob Seidelin@anon (and the other anon): Nice ideas. Yes, you could use any number of tricks to get the file size down, but stuffing data of this size in images would also add some considerable overhead when reading the data. Anyway, the goal here was the data visualization, so I'll leave it at that.
July 22, 2008 at 9:29 AM Anonymous@p01: Nice job with the optimizing.
The CanvasPixelArray [1] of the ImageData object does not inherit from the Array object, so it does not have a join() method. Beside only Firefox 3 and Opera 9.5 support ImageData so such technique will not work at all in Safari 3.1.2
July 22, 2008 at 9:31 AM Anonymous[1] http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas.html#canvaspixelarray
i like you work in javascript. especially the canvas tag haven't found many sites working this much with it. well i tested in the latest nightly buiilds of each browser (opera 9.52 beta safari latest webkit and firefox minefield a2) and the speed is in the order listed but all should load it on the default settings.
August 21, 2008 at 2:47 PM Anonymousi get amazing speed in Google chrome nightly (the 5th sept)
September 5, 2008 at 7:04 PM AnonymousDidn't load for me either on FF 3.0.4, but it does on Gnome Epiphany 2.22.2 with gecko-1.9
December 10, 2008 at 12:52 PM Gavin DoughtieWhy not use WebGL for the next rev? It's being actively dropped in all the browsers that currently support Canvas.
February 8, 2010 at 8:49 AM Jacob SeidelinAgreed, a WebGL version would be interesting.
February 8, 2010 at 9:18 AM baby teethingI would like to say thanks for letting me post here,your topic about Radiohead,it's very interesting I want to find out more about it,I also want to invite you to visit my site in order to keep you informed about baby teething cheers!!!
May 4, 2010 at 2:02 PM AnonymousUpload your photo or browse our image gallery to print photos on canvas, photo prints or fine art prints. We have exclusivity on many of the images displayed in our gallery and these cannot be purchased anywhere else. nice post.
June 15, 2010 at 4:53 AM Anonymousthanks
personalised wallpaper
Great. Another idea already implemented ;-) Thanks for sharing your experiences and knowledge.
July 29, 2010 at 4:22 PM Anonymous1I used only one in 20 frames, but using 100% of the points after filtering noise. It totals over 8MB. The points are sorted by color, because parsing the fillStyle property for every pixel was too slow. No sound though. I hacked together an optimized version of House of Canvas based on Jacob’s HiRes data set of 6485 dots. It runs at ~15 fps in Opera 9.51.
March 6, 2012 at 5:37 PMOrdering the dots by angle should eliminate the occasional front to back overdraws at little to no cost.