This is a post I've been looking forward to write for quite some time, but I've just been to busy with work, CodeCampOz and REMIX to get around to do it… At the MVP Summit in Seattle some weeks back I had dinner with a group of friends, including fellow RD and co-worker Sondre Bjellås, and long time IRC-friend Wilco Bauwer (now working at Microsoft). Like when ever a group of programmers get together the talk soon was about Silverlight, C# and programming an general. We started discussing Silverlight and what features we would love to se in future versions. One of the things Sondre felt was missing is support for microphone and webcam. I don't know why Sondre think this is a big deal (but he DO read Love and Sex with robots… btw, Sondre, you should write a book review!). Wilco have worked on the HTML-bridge in Silverlight, so we started talking if it might be possible to "cheat" basic webcam support by bridging Silverlight and Flash. Without knowing the details we all agreed that it was probably possible to do.

When I got back from Seattle I started doing some initial research, like learning how to program against the webcam in Flash, what kind of HTML-bridging  capabilities you got and so on. A video tutorial titled "Webcams, PNG and AIR", combined with a blog post titled "Use JavaScript to Take a Screenshot of a Flash movie", gave me all the pieces to build webcam support for Silverlight (using Flash). So I asked my self: "What would MacGyver do", and decided to glue everything together into a proof-of-concept of webcam support in Silverlight 2.

The video tutorial covers the basics of how to create a camera object, attach it to a video and grab a video frame as a BitmapData object. It also covers how to use the as3core library, a set of open source extensions to ActionScript3, which includes a PNG-encoder class. The PNGEncoder encodes a BitmapData object into a ByteArray. Once I had the ByteArray I needed a way to pass it back to the browser. Since the set of types you can marshal between ActionScript and the browser (at least to my knowledge) is limited, I figured a Base64 encoded string would be the way to go. The blog post about taking screenshots used this technique to pass an image from Flash to JavaScript. It also covered how to expose ActionScript functions to JavaScript using the ExternalInterface APIs.

Once I had the Base64 encoded PNG image in JavaScript I could simply return it back to Silverlight, which then would use the Convert.FromBase64String method to create a byte array. With the image stored in a byte array I could simply load it into a MemoryStream, and then set the MemoryStream as the source of a BitmapImage object.  I'll let MacGyver show you how it all comes together:

It literally didn't take more than 20 lines of ActionScript, 5 lines of JavaScript and 5 lines of C# to get a single frame from my webcam onto a Silverlight image control.  This first piece of code is the ActionScript needed to start the webcam, capture a frame, set the mode and expose two functions to the browser:

The next part is the JavaScript needed to grab the object/elem tag containing the Flash player, and calling the ActionScript functions to grab a snapshot from the webcam. The Base64 string is just returned from the function, as this function is being called from C#:

The final part is the C# code using the HTML bridge to invoke the JavaScript function, grab the Base64 string, decode it, and then load the PNG image:

Using the basic webcam integration I created two sample applications. The first one is a "performance test" to see how many frames per second I'm able to transmit to Silverlight. Since I'm not able to pas the raw stream from the camera, and have to PNG encode each frame, it's all CPU bound. The application uses a timer, which will grab a frame and measure the time it takes. Once it got the frame it will adjust the interval of the timer to the amount of time spent on the previous frame + 50 milliseconds (to give the UI thread some room to handle input events etc). I was able to get roughly 12fps 180x140, but on higher resolutions it starts to get sluggish. Also note that when using all the CPU power to capture frames you don't have CPU left to do Silverlight animations etc, so doing "live" video is really something you don't want to do. Click the image to check out the sample:

The second sample is a Silverlight "Photo Booth" application, with out any of the cool features of the real Apple Photo Booth app. The interesting part about this sample is how I overlay the Flash container on top of Silverlight. This way my Silverlight application don't need to use CPU power to pre-view the video, and I only grab photos when the user clicks the button. Each image is also added to the thumbnail control, which I re-used from my YouCard demo. Click the image to check out the sample:

I've made the code available for everyone to play with, including the Flash webcam wrapper. In order to build the Flash app you need Adobe Flash CS3. If you don't have Adobe Flash CS3 you can still build the app in Visual Studio and use the prebuilt WebcamWrapper.swf file. The code is not optimized in any way, and probably contains some bugs. But, this initial post is meant as a proof-of-concept, and I'll probably revisit this topic later.

And to summarize: THIS is how MacGyver would have built webcam support for Silverlight 2!