Process stereoscopic image to convert from 3D to 2D

Question

I am wondering if it is possible to take a stereoscopic image and sum them in such a way that makes it a monoscopic image, with the resulting viewpoint in the center of where the two viewpoints actually where.

For example, if I had two cameras about 1 ft. apart, each takes an image of a subject 3 ft. away at the same time, could I take the resulting images, process them, and make the resulting image appear as if it were taken from a 3rd camera placed in the center of the outer two?

Are there tools for this sort of interpolation? I remember a couple scenes in the making of The Matrix where they talked about filling in frames of rotating, and it only took about 4 or 5 cameras to handle the shooting. (To be fair, I also remember the part where they had a whole array of cameras, each shooting one frame.)

Is this possible? If so, what tools would you use to do it?

Answer 1

You could probably in most cases reconstruct an image that looks convincing, but you can't recreate what's actually seen from a point between the viewpoints.

Take for example an image that is taken right in front of a wooden fence, where the two viewpoints are each in the middle of a panel so that they both only see the panel. A viewpoint in between would see between the panels:

    ^
    |
--- | ---
 o  x  o

It's of course not possible to recreate what's seen between the panels from the two pictures of panel.

In most pictures there would be certain angles that weren't seen from either viewpoint, but would have been seen from a viewpoint in the middle. A tool that would create a monoscopic image would have to guess what might have been seen.

Answer 2

Check out Piku Piku.

As far as I can tell, they use something like optical flow to find matching pixels in the two images, then use the depth information to make an interpolated view, with a simple nearest-neighbor guess for the missing pixels. It works pretty well on stereoscopic images I've tried it with, and gives an improved effect over wiggle stereoscopy. You can see the filled-in pixels and some errors in this example.

In general this is called view interpolation.

Answer 3

In many such cases, such as Guffa's wooden fence, it is impossible. In some other cases, such as an infinitely distant focus, it is trivial. For everything in between, it is just really really hard. There are pieces of software to do it, including the firmware on the Kinect. The trick is that you have to take the two views and use them to produce a 3d model of the world in front of the camera, then produce a view of that world from the desired point.

With some of the current Kinect projects, you can move the virtual viewpoint arbitrarily, looking at the scene from above or from the side. The farther from the cameras you move, the more unavailable information there is, and thus the more gaps in the resulting picture there are.