Convert video into long horizontal image

Question

I have recorded a video driving with my car, with my phone stuck to the side window. Basically it's like a 2d view, but with depth. The video has 2 minutes in length and the distance traveled by the car is around 2-3km (not that relevant I think).

I want to transform that video (more correctly would be the distance) into a background for a 2d game (more like an entire level where the hero travels from the start of the video to the end), do you happen to know any software that would extract the frames and put them back into a long horizontal long image compiled from the video? If I snapshot frames one by one and put them one near another in Photoshop it's not what I want, it looks somehow distorted or something, I tried with 5-10 meters but a house turned squeezed.

Hope it's clear, let me know if now and I will try to explain in another way, thx!

Answer 1

I have not done this myself, but here is how I would approach it roughly.

Assuming that:

• the video has a constant framerate of 30 frames per second (f/s)
• there are 30 [f] * 120 [s] = 3600 frames in total
• the vehicle travelled between 2000 [m] / 120 [s] = 17 m/s and 3000 [m] / 120 [s] = 25 meters per second
• each frame is shifted at most by 25 [m/s] / 30 [f/s] = 0.8 meters each frame

For a (mosaic¹) panorama you don't need a photo every 0.8 meters. I think a photo every 16 meter should be sufficient, i.e. every 20th frame.

ffmpeg is the tool I would use to extract from a video, this Stackoverflow question deals with that.

After I would have the frames (about 180) I would start stitching batches of +/-10 images in Hugin. I guess Photoshop might work, but Hugin provides far more control over the stitching process. Stitching all the 180 frames in one go sounds like something my computer (and sanity) would not be able to handle.

Once the batches are stitched, I would stitch them into the final image.

Instead of only taking every 20th frame, another clever way to reduce the number of overlapping pixels (suggested by xenoid in the comments) is to take a narrow column of pixels from each frame (e.g. 10% of the width of the whole frame) and stitch that.

Not sure how I would display the result though, as the final image would have an approximate aspect ratio of 1:240 (height:width).

Good luck, and don't forget to keep us updated on the outcome.

^{1) I think that's what it's called when you move the camera and take photos to stitch together.}

Answer 2

What you want to do is not particularly difficult, nor uncommon. The method Saaru suggested will work, as will the method you tried (manually in photoshop).

This article on Amnesty International covers the topic in some detail.

But your real issue is probably going to be getting images suitable to stitch together in the first place; primarily due to the rolling shutter characteristic of the phone's sensor readout.

If I snapshot frames one by one and put them one near another in Photoshop it's not what I want, it looks somehow distorted or something, I tried with 5-10 meters but a house turned squeezed.

As you have found, if you try to stitch images with rolling shutter artifacts the result will be distorted. You may be able to improve the results if you first edit the video with iMovie to fix the rolling shutter effect (there may be other programs that offer similar).

Even if you avoid/correct the rolling shutter issue, you still have the issue of recorded motion blur per frame. The standard SS for 30fps video is 1/60; that's not sufficient to freeze much motion/speed, particularly with motion parallax in a moving vehicle. You would likely need to use high speed video like 120fps w/ 1/240 SS (or higher), or have the ability to set a higher SS separately.

In the end, I think you would be much better off just taking multiple still images using a camera with a mechanical shutter (or very fast readout speed, or stationary).

Answer 3

The effect could be directly achieved using a slit-scan camera. That is more or less how Google makes StreetView.

Disregarding the one edge of the first few frames and the other edge of the last few frames, use the column of pixels at the center of each and every frame as the slit.

So the bitmap would start with a “stack” of pixels from the center of the first frame.

A stack of pixels from the center of frame two would be next to it. A stack from frame three next to it and so on.

Depending on travel speed and video frame rate the slit might need to be wider (more than one pixel) than a single pixel or the sample rate lower than every frame.

Sharpness might be enhanced by combining pixels from several frames. This might also help reduce compression artifacts.

For a serious endeavor, it is perhaps worth considering building dedicated slit scan hardware…that’s what Google does.