A Different Tune

Multitouch is cool, but its also becoming cheaper and more accessible thanks to the work of folks like Jeff Han, and communities like the NUI Group. Of course, devices like the iPhone have made many of these ideas mainstream, but not everyone wants to do multitouch on a tiny display. Multitouch surfaces have great potential as intuitive, multi-user, social computing environments.

I wanted to build a small, portable touch table that can be used by one or two people. The display surface is the horizontal table top, and people "play" at opposite sides. Its a bit like those old-style sit-down arcade games, though for the life of me I can't remember any games that were simultaneous two-player. Both users can touch the surface, but also maintain face-to-face interactions across the table. The device is designed particularly with kids in mind, for play between two children, or between a child and parent.

One of the main motivations for building this was to come up with an aid for educating kids as young as 2 years old. Some children struggle to learn stuff that most of us pick up effortlessly, like how to talk and interact socially. Very early intervention can make a huge difference for kids with conditions like autism and other language disabilities. While therapies like ABA can be very effective, they can also be gruelling and intense for both the child and the therapist (usually, the parent). If we can simplify this process using engaging interactive games, and by sharing and building resources collaboratively on-line, this has to be a good thing. More on this later...

Design

There are many ways to do multitouch sensing, but the basic idea used here is to measure pressure optically with a camera. A great place to get some background is this thread at NUI group. Since I wanted to do fiducial tracking, I chose an approach based on diffuse illumination. This means that the display does not rely on a very tight coupling with the surface to illuminate objects.

I wanted my table to be small, cheap and robust, so I chose to use an LCD rather than a projector. LCDs don't transmit diffuse IR very well, so the illumination is provided by the surface itself (Diffuse Surface Illumination, or "DSI"). A special type of acrylic (called "Endlighten" or "DLux", depending on the manufacturer) is used as the top surface of the display. This has embedded particles which reflect light out the surface when lit from the side. This material is often used in display signs.

Parts List

• LG W2252TQ-PFT 22" LCD Monitor
• Unibrain Fire-i B/W Camera
• 2.5mm focal length lens
• Mac Mini
• Firewire 9-pin to 6-pin cable
• IR pass filter 850nm
• Acrylic panel, 10mm "Endlighten"
• IR LED strip illuminator
• Power supply, 12V for illuminator
• Misc hardware: MDF, wood screws, nut/bolts, square dowell, 1" aluminium angle, steel strip, T-nuts, hinge, construction adhesive, silicone sealant, magnetic catches, gaffa tape

I found a lot of useful information on-line, but much of it was fragmented. So here is an illustrated guide to how I built my system. Click on the images for higher resolution versions.
Continue reading ‘LCD Multitouch Table Mark 1’ »

Autostitch is an automatic 2D image stitcher. Give it a collection of partly overlapping images, and it will position and blend them into a seamless panorama. A demo version of Autostitch is available for free download. It has a simple GUI which can be used to select input images, and adjust parameters such as output resolution, cropping, and image rotation. However, the GUI interface is a little inconvenient when you have a large number of repetitive stitching tasks to perform, for example : panoramic video.

A panoramic video can be made by stitching each frame of an input video against a background panorama, as described in this earlier post. To do this we need to run thousands of separate stitching operations, each with different sets of images. Its too much to do by hand via the GUI, which is why I wrote this little batch controller for Autostitch.

The batch controller is a simple program that pretends to be a user, sending sequences of events to the GUI to simulate the normal operation of the program:

1. In the Autostitch window, select menu File -> Open.
2. When the open dialog appears, enter a set of image names into the edit control, and then press the "Open" button
3. Wait for Autostitch to finish processing
4. When Autostitch opens the preview window, close it. This makes Autostitch ready to perform the next stitching operation

Fortunately, this is not too hard to do using the Win32 API. FindWindow(NULL, name) gets us a top-level window with a given name (eg. "autostitch" or "Open"). The controls for a window can be recursively enumerated using EnumChildWindows(window, callback, data). We can use our own callback function to search based on the name or class of a control. Windows uses messages to communicate events to controls. We can generate the same messages using SendMessage. For example, to click a button:

  SendMessage(button, BM_CLICK, 0, 0);

or to enter text into an Edit control:

  SendMessage(edit, WM_SETTEXT, 0, (LPARAM) text);

If you want more details, check out the source from the download links below.

Usage

To use the batch controller, you need to first start Autostitch in the usual way. Autostitch must be "pointing" at the directory containing your source images. You can set this using the "File->Open" menu. Just select one or more images and press "Open" - it doesn't matter whether the result is successful or not. From the command-line, you can then stitch any set of images from that directory like this:

  stitch file1.jpg file2.jpg file3.jpg

This will stitch the images as usual, opening the result "pano.jpg" in the default image previewer. To run multiple stitches in sequence, the preview window must be closed before the next stitch can start. Request this by specifying -wait -close. For example:

  stitch -wait -close file1.jpg file2.jpg file3.jpg

To stitch a sequence of images, use the -template option, like this:

  stitch -template frame%03d.jpg 20 50 file1.jpg file2.jpg file3.jpg

This specifies a file name template (using sprintf syntax), and a range of file numbers. The example generates a sequence of names "frame020.jpg", "frame021.jpg", ..., "frame050.jpg". Each stitch operation combines one of the sequence names with the remaining arguments, and renames the output panorama (by default, to "out0020.jpg" ... "out0050.jpg"). Template mode automatically implies -wait -close.

Here are some simple examples using the Autostitch demo images:

  stitch -wait -close 100-0023_img.jpg 100-0024_img.jpg 100-0025_img.jpg

This stitches the bottom 3 images of the "mountain" panorama.

  stitch -path s:/stewart/programs/autostitch/images/test/ 
             -template 100-%04d_img.jpg 38 40 
             100-0023_img.jpg 100-0024_img.jpg 100-0025_img.jpg

This stitches each of the top three images (38..40) against the bottom three images, producing three outputs, "out0038.jpg" .. "out0040.jpg". (change the path to reflect your Autostitch installation).

There are a bunch of settings that you might need to change. For example, the system looks for the preview window using one of the standard applications ("pano.jpg - Windows Picture and Fax Viewer" or "Microsoft Photo Editor - [pano.jpg]"). If you have a different preview app, you'll need to change the title of the preview window using the -output option.

In "template" mode, the system renames each "pano.jpg" to a different file name using the sequence number, but to find the resulting file you need to specify the result directory using -path. Alternatively, run the stitch command from the directory containing the source images.

Options:
-help      Show help message and exit
-version   Show version and exit
-wait      Wait for Autostitch to complete (single stitch only)
-close     Close preview opened by Autostitch (single stitch only)
-template  temp start end Specify template for batch operation
           A range of images is processed for start <= id <= end. Use sprintf
           formatting (eg. id=23, temp='file%04d.jpg' -> 'file0023.jpg')
-dryrun    Just output batch image lists without calling Autostitch
-path      Path to image directory (for batch stitch). INCLUDE TRAILING '/'

Stuff you probably don't need to change:
-window    Name of Autostitch window (default: autostitch)
-status    Name of Autostitch status window (default: Status)
-output    Name of preview window (default: 'pano.jpg - Windows Picture and Fax Viewer')
-fileOpen  Menu name for File->Open dialog (default: 'File:Open')
-openTime  Delay (in ms) for open dialog to populate controls (default: 500)
-closeTime Delay (in ms) to wait before closing preview window (default: 2000)
-outjpg    Template (dir,id) for output files (default: '%sout%04d.jpg')
-outtxt    Template (dir,id) for output files (default: '%sout%04d.txt')

Download

• Source code and Windows binary (gzipped tar file, 10K)
• C source on-line

This app is fairly minimalist, but it should not be hard to modify if required. I used mingw + gcc which are available for free from mingw.org.

Have fun! And if you manage to create something interesting, please post a comment.

Check here for more fun with Autostitch.

Perhaps you too have an unused exercise bike languishing in a dark corner. Lets face it, its pretty boring exercising on one of these things. How to get motivated? Make it fun using WiiFit!

Here's what to do:

1. Stick your bike in front of the TV.
2. Fire up WiiFit, and select Jogging (under Training -> Aerobic Exercises -> Jogging).
3. Attach the Wiimote to the lower part of your leg. Its normally meant to go in your hand or pocket so its pretty insensitive to orientation but does need a good amount of movement. If you're wearing socks, you can just slide the Wiimote in - remember to face the buttons away from your leg so you can press "A" through your sock. It also works to put it the cuff of track pants, though it tends to flop about too much for my liking.
4. Press "A" and start riding. It can take a couple of seconds for the Wii to learn the motion pattern.

Wiifit gives you some nice visual feedback that helps you keep a constant cadence. As you ride, your avatar runs along a path, following your training guide. The landscape is quite pleasant, and you can see other runners and dogs exercising too. If you slow down, your avatar starts to lag behind your guide and it will prompt you to keep a steady pace. You can select your desired activity level as you start the game. If your bike has a variable load, you can adjust this if you find the jogging pace too easy.

Have fun!

Youtube is the undisputed leader of the video-sharing game with world-wide market share of 66%. Everyone wants a piece of that market, and there are lots of new players entering the business. To attract users they have to offer something different, something compelling.

This post shares my experience with two providers: Youtube and Viddler. There are significant differences between the two, and some of these are not obvious. Check out the feature-by-feature comparison. Read on for the details, or cut to the chase.
Continue reading ‘Video Sharing Showdown : Viddler vs Youtube’ »

The cassette tape is rapidly becoming a "vintage" audio technology, but some of us who have a reasonable amount of material in this format. And part of my collection is a box of damaged and disembowelled cassettes waiting to be repaired. For at least a few of these patients, the wait is over.
Continue reading ‘Cassette Surgery’ »

Autostitch is "the world's first fully-automatic 2D image stitcher". Developed by Matthew Brown at the University of British Columbia, Autostitch can automatically generate panoramas from a collection of overlapping images.

Autostitch uses a key-point detector known as SIFT (Scale Invariant Feature Transform) to locate interesting features in images. By finding the correspondence between key-points in different images, Autostitch can tell how the images are arranged with respect to each other, allowing them to be rendered into a seamless panorama. Scale invariance means that the features still look the same under scaling (rotation, translation, zoom, etc) which makes them robust to typical camera motion.

Click on the following images for a detailed version:

Swan River shore, Bicton, Western Australia

Playground, Palm Beach, Queensland

Lookout near Byron Bay

View of Tweed Valley from Montville

Sugar mill at Tully, view from atop the Golden Gumboot

Autostitch is clever, but its also fun! After playing with it a bit I got thinking. If you can build a panorama automatically, then you can "embed" any image taken from the same position into the panorama. This could be quite interesting as a creative tool, but also has applications in tools like media browsers where the relationships between images could be useful for organising and navigating media.

Check these posts for some ideas about how to use this:

• Embed images into a panorama
• Panorama Montage
• Panoramic Video
• A Batch Controller for Autostitch

In a previous experiment, we found that images can be embedded into a background panorama. Now its time to see if we can use this to make a stitched panoramic movie.

Step 1. Record a background panorama

I made a movie of my lounge room using a Panasonic NS-DV50A digital video (DV) camera. I captured this to DV-AVI format, then used VirtualDub to de-interlace and convert frames to JPEG images.

I took a couple of overlapping pans around the room. Nothing fancy here. Just think "painting with camera". Next I selected every 10th frame and made the background panorama using Autostitch:

Step 2. Record some action

Next, I took some video of the kids playing in the room. Again, I de-interlaced and converted the frames to JPEG images. You can see this video in the output below.

Step 3. Stitch frames against panorama

Here's the tricky part. We want to take each frame of the video, and stitch it together with all of the frames for the background panorama. This will generate one panorama per input frame, which we will then combine to produce the output.

The catch is that Autostitch is a GUI application. Doing this by hand using the GUI is way too time consuming, so I wrote some black-magic code to control Autostitch via a batch process.

This produces images with a fixed width but variable height. I resized these to a consistent size.

Step 4. Convert panoramas to movie

Next, I loaded the panorama image sequence into VirtualDub and converted it to a movie file. Additionally, I combined the panoramic image with the original movie, and exported these frames as a movie.

Results

Here's the stitched panoramic movie along side the input movie:

I tried to stretch the capabilities of the matching by doing some zooms and rotations with the camera. Towards the end of the movie is a section where the matching breaks down because there is probably too much transformation of the image. Although the SIFT should be able to handle this, there may be orientation-dependent differences in the characteristics of the features that cause it problems. This is due to the de-interlacing of the original image, which means that scan lines are duplicated so the detail in the horizontal and vertical directions are not the same.

Overall, I'm pretty happy with the result. Its not perfect, and could probably be improved with some specific support for this sort of thing. We are asking Autostitch to do a few things it was not designed to do. Basically, we don't want to re-stitch the entire panorama for each new foreground frame. We want the background to be stable, but can be more lenient when stitching the foreground frame. Also, with specific knowledge about which frame should go on top, it should be possible to smoothly blend the frames without losing the foreground.

Being able to embed video into a panorama suggests interesting possibilities for immersive media browsing environments based on 3D panoramas. See: "Browsing personal media archives with spatial context using panoramas".

Previously, we saw that subjects can be stitched into a background panorama. An extension of this idea is to stitch multiple subjects, or subjects in different positions into a single panorama.

Basically, the steps for doing this are the same except that we are now stitching more than one frame against the background panorama. The only real catch here is to make sure that the stitched frames don't overlap too much or they will obscure each other.

With some care, you can get a result like this. The panorama is generated automatically by Autostitch. I added a small white border around each image. Then I added the strip at the bottom to show the source images and the links to the embedded frames.

The ability to do this sort of thing offers interesting possibilities for media browsing. If we can automatically relate photos to the spaces they are taken in, this might be a useful way of organising collections. See: "Browsing personal media archives with spatial context using panoramas".

We know that Autostitch is fun for building panoramas. After playing with it a bit I got thinking. If you can build a panorama automatically, then you can "embed" any image taken from the same position into the panorama. This could be used creatively to build montages, or even panoramic video.

Does it work? Lets find out...
Continue reading ‘Fun with Autostitch : Panorama Embedding’ »