xsvdo’s blog

Nothing like a good vacation to clear the mind and the schedule.  Not happy with the current state of computer vision apps for multi-touch and doing a bit of research, I've decided to embark on building TactTool, another multi-touch framework.  There are quite a few out there, the most notable is Touchlib (soon to be renamed CommunityCore), BBTouch, and the soon to be released Lux.

So why you writing one instead of contributing to one of the above?

I've thought long and hard about this and decided to write my own because I want to do it in Processing.  (I wanted to use Openframeworks but after some evaluation, I don't think it is quite there yet). I also was evaluating Python because of the built-in OpenCV support. I am actually torn about Processing v. Python because I'd like to use PyAMF to get output directly into Flash.  We'll see how well I can integrate Merapi into Processing.  I don't know what I don't know yet so hopefully we'll see if this was a good bet.

Once I get everything in good shape, I'll open source it.  Watch this blog for up to the week progress! I have written semi-functional blob tracking in as3 but was never quite happy with the results because it was too slow. I know about Pixel Bender and have created some filters and even a basis of an Air app. to read pbj files and apply it to a webcam feed. However Flash10 (according to this blog post)  is _not_ going to leverage the GPU for now (I understand the reasoning for Flash but hopefully that won't be the case for Air).

Using OpenCV, I won't have to do the blob tracking math. I can focus on creating cool filters and integrating with light sensors from Phidgets for dynamic filter adjustments.  For now, these are the requirements :

- load and save configurations
- multiple camera support
- define filter region
- output options : TUIO//XML via AMF/socket server

I'm excited that I was able to get an account at Shapeways.  It is a 3D printing service that will allow me to create some funky cold fiducials (yes, cheesy 80's reference) for my DI table.

Below is a screeny of an OpenCV port to Processing.  Blob detection in under 100 LoC. The top two screens are raw camera output, the bottom left is the background subtraction image and the bottom right is the processed blob detection with x/y coordinates.  Looks like it won't take much to port output via TUIO and send directly to Flash.  I have some light sensors laying around and may try to hook 'em up and adjust brightness/contrast/tolerance based on outside light conditions.

OpenCV is very cool and Processing makes it easily accessible and joy to code.

There goes the AIR app. I was building that was going to use Pixel Bender to filter FTIR setup for blob detection  . . . don't even get me started on face recognition (eye tracking anyone)? Download for yourself and check it out.

I'm testing out Pacifica, Adobe's Flash VOIP library, and I have to say, "it works as advertised", meaning the demos shown at Max and other events are true to form (I'm under NDA so I can't say much more).  With RibbitPhone, Red5, Smart Fox Server, ElectroServer, FMS and Adobe Connect, there are now quite a bit real-time possibilities with Flash serving different needs.

Add to that the new hardware acceleration additions in Flash 10 and my real-time, multi-touch dreams are coming true!  The only thing left is a built-in Flash screen codec to capture screen recordings.

Ribbit Phone, the VOIP library for Flex/Flash, is awesome and simply amazing.  In less then 5 minutes, using the sample code, I was able to create a Flex project that made and received phone calls!  Wait, it gets better.  Not only can I make and receive calls, I have access to voicemail.  As a developer, I can programatically manipulate my voicemail.  Hold on, it doesn't stop there . . . voicemail is automagically transcribed to text!!

So, I'm in the throws of putting together Spyke.  Spyke = Amazon S3/EC2 + IM functionality + webcam chat + Ribbitphone.  Technically not so challenging, more a user experience nut that needs to be cracked.  I might break down and get on the Facebook application bandwagon for exposure (once the Ribbit Phone forums open up, I'm going to find out the account setup process).  Anyway, should be fun.

I know I know, I have been away too long.  My startup was acquired and now I'm working for the MAN.  I'm still working on multi-touch software (using diffused illumination now instead of FITR) but what brings me back to blogging is the anticipation of being able to develop a Flex/Air application based on the Ribbit Phone VOIP component.  Here's a snippet of an e-mail I got today:

"You have been chosen as a finalist from among the hundreds of applicants to Ribbit's Phone Component pre-release program!  We've whittled the number of applicants to a select pool of highly qualified folks with really cool ideas.  Our next step is to listen to the details of your ideas and make our final selection."

My fingers are crossed.  If I'm chosen I will blog about my experience developing Spyke, a real-time, ad supported social networking/collaborating video/phone/text chat application in Flex/Air using the Ribbit Phone Component for the front end and Amazon S3/EC2 on the backend.  I plan on it being compatible with AOL/MSN/GTalk/YIM and OpenID/SAML/(whatever else is easy to port social graphs) and last but not least . . . open source.  Yes, I will also create a "widget" for other networks using something like Clearspring/Gigya.

If I have enough time, I'm planning on adding a photo/Powerpoint slideshow and mp3 broadcaster.  I am going to try and make this will work with FlashLite 3 too.

In other news, ffmpeg plans to decode NellyMoser!!! 

When something essential is purposely flawed.  Derived from the standards body that regulates javascript/Jscript and Actionscript3, the most widely used client side programming languages on the Internet today. 

I've been asked, "What programming language should I master"?  Naive question for sure but it got me thinking about the browser as the new desktop and more specifically about client side RIAs.  A new religious war is bubbling . . . Flash or Silverlight.  Some frame it as an open source v. Microsoft battle.  However, that does not take into account the client side operating system (including Symbian here) and the server side technology.

This opens a very interesting door for the likes of Laszlo.  If they pretty themselves for a date with Google, I think they can come out on top.

Back to server side technologies.  Besides Livescript which ran on Netscape web servers, I don't know of any server based language that is ECMA compliant.  FMS doesn't count because it doesn't serve http.  What language am I missing?  Haxe really doesn't count . . . at least not yet. 

So if all the client side technology is ECMA based, is there an equivalent server side language?  If so, then ECMA would be good programming "language" to master because learning the different dialects would cover you as a newly minted CS graduate.  It would also get you a running start with Python and Java.

Having one standard to rule them all is great but if ECMA is that standard, it is inherently flawed.  A beautiful dilemma.

I'm a little late to the game but Microsoft announced the release of their CLI.  With the release of Tamarin, I sense an Adobe v. Microsoft fight which hopefully will result in the ability to compile AS3 code to a .NET compatible binary?  I'm sure the Haxe developers are all over it already.

Nothing major but I am too excited!  I have simple blob detection working in AS3.  The swf is less than 5k and it works better than I expected (I guessed at values and got pretty much spot on).  I know each slate is going to have different values due to infrared intensity, ftir, camera sensitivity/distance/other adjustments, etc.  I thought that I would waste a lot of time adjusting sensitivity levels but detection happens right when I touch the acrylic when using different fingers and hands. Once I get to refactoring I'll create easy to use adjustments slider knobs or something.

Now on to the interesting and fun part . . . creating event handlers multi input based event handlers.