Autostitch uses a key-point detector known as SIFT (Scale Invariant Feature Transform) to locate interesting features in images. By finding the correspondence between key-points in different images, Autostitch can tell how the images are arranged with respect to each other, allowing them to be rendered into a seamless panorama. Scale invariance means that the features still look the same under scaling (rotation, translation, zoom, etc) which makes them robust to typical camera motion.
Click on the following images for a detailed version:
Autostitch is clever, but its also fun! After playing with it a bit I got thinking. If you can build a panorama automatically, then you can "embed" any image taken from the same position into the panorama. This could be quite interesting as a creative tool, but also has applications in tools like media browsers where the relationships between images could be useful for organising and navigating media.
Check these posts for some ideas about how to use this:
In a previous experiment, we found that images can be embedded into a background panorama. Now its time to see if we can use this to make a stitched panoramic movie.
Step 1. Record a background panorama
I made a movie of my lounge room using a Panasonic NS-DV50A digital video (DV) camera. I captured this to DV-AVI format, then used VirtualDub to de-interlace and convert frames to JPEG images.
I took a couple of overlapping pans around the room. Nothing fancy here. Just think "painting with camera". Next I selected every 10th frame and made the background panorama using Autostitch:
Step 2. Record some action
Next, I took some video of the kids playing in the room. Again, I de-interlaced and converted the frames to JPEG images. You can see this video in the output below.
Step 3. Stitch frames against panorama
Here's the tricky part. We want to take each frame of the video, and stitch it together with all of the frames for the background panorama. This will generate one panorama per input frame, which we will then combine to produce the output.
The catch is that Autostitch is a GUI application. Doing this by hand using the GUI is way too time consuming, so I wrote some black-magic code to control Autostitch via a batch process.
This produces images with a fixed width but variable height. I resized these to a consistent size.
Step 4. Convert panoramas to movie
Next, I loaded the panorama image sequence into VirtualDub and converted it to a movie file. Additionally, I combined the panoramic image with the original movie, and exported these frames as a movie.
Here's the stitched panoramic movie along side the input movie:
I tried to stretch the capabilities of the matching by doing some zooms and rotations with the camera. Towards the end of the movie is a section where the matching breaks down because there is probably too much transformation of the image. Although the SIFT should be able to handle this, there may be orientation-dependent differences in the characteristics of the features that cause it problems. This is due to the de-interlacing of the original image, which means that scan lines are duplicated so the detail in the horizontal and vertical directions are not the same.
Overall, I'm pretty happy with the result. Its not perfect, and could probably be improved with some specific support for this sort of thing. We are asking Autostitch to do a few things it was not designed to do. Basically, we don't want to re-stitch the entire panorama for each new foreground frame. We want the background to be stable, but can be more lenient when stitching the foreground frame. Also, with specific knowledge about which frame should go on top, it should be possible to smoothly blend the frames without losing the foreground.
Previously, we saw that subjects can be stitched into a background panorama. An extension of this idea is to stitch multiple subjects, or subjects in different positions into a single panorama.
Basically, the steps for doing this are the same except that we are now stitching more than one frame against the background panorama. The only real catch here is to make sure that the stitched frames don't overlap too much or they will obscure each other.
With some care, you can get a result like this. The panorama is generated automatically by Autostitch. I added a small white border around each image. Then I added the strip at the bottom to show the source images and the links to the embedded frames.
We know that Autostitch is fun for building panoramas. After playing with it a bit I got thinking. If you can build a panorama automatically, then you can "embed" any image taken from the same position into the panorama. This could be used creatively to build montages, or even panoramic video.