Convert video into long horizontal image

I have recorded a video driving with my car, with my phone stuck to the side window. Basically it's like a 2d view, but with depth. The video has 2 minutes in length and the distance traveled by the car is around 2-3km (not that relevant I think).

I want to transform that video (more correctly would be the distance) into a background for a 2d game (more like an entire level where the hero travels from the start of the video to the end), do you happen to know any software that would extract the frames and put them back into a long horizontal long image compiled from the video? If I snapshot frames one by one and put them one near another in Photoshop it's not what I want, it looks somehow distorted or something, I tried with 5-10 meters but a house turned squeezed.

Hope it's clear, let me know if now and I will try to explain in another way, thx!

• Relevant: photo.stackexchange.com/q/95348/9161 Feb 3, 2022 at 22:40
• Why do you have to convert the video to a still image? Why can't you play the video in the background at the appropriate speed while matching the movement of the 'hero'? Feb 4, 2022 at 9:24
• Good point by @jarnbjo. If you're interested in the game development side you might be better of asking at gamedev.stackexchange.com how to reach your goal. Feb 4, 2022 at 11:47

I have not done this myself, but here is how I would approach it roughly.

Assuming that:

• the video has a constant framerate of 30 frames per second (f/s)
• there are `30 [f] * 120 [s]` = 3600 frames in total
• the vehicle travelled between `2000 [m] / 120 [s] ` = 17 m/s and `3000 [m] / 120 [s]` = 25 meters per second
• each frame is shifted at most by `25 [m/s] / 30 [f/s]` = 0.8 meters each frame

For a (mosaic1) panorama you don't need a photo every 0.8 meters. I think a photo every 16 meter should be sufficient, i.e. every 20th frame.

`ffmpeg` is the tool I would use to extract from a video, this Stackoverflow question deals with that.

After I would have the frames (about 180) I would start stitching batches of +/-10 images in Hugin. I guess Photoshop might work, but Hugin provides far more control over the stitching process. Stitching all the 180 frames in one go sounds like something my computer (and sanity) would not be able to handle.

Once the batches are stitched, I would stitch them into the final image.

Instead of only taking every 20th frame, another clever way to reduce the number of overlapping pixels (suggested by xenoid in the comments) is to take a narrow column of pixels from each frame (e.g. 10% of the width of the whole frame) and stitch that.

Not sure how I would display the result though, as the final image would have an approximate aspect ratio of 1:240 (height:width).

Good luck, and don't forget to keep us updated on the outcome.

1) I think that's what it's called when you move the camera and take photos to stitch together.

• With a powerful enough machine, instead of throwing frames, I would keep them all but crop them to a thin narrow strip around the center. Feb 3, 2022 at 23:31
• @xenoid, that's clever, feel free to edit that in or add it as another answer. Feb 3, 2022 at 23:33
• I have not tested it, so I won't make an answer from it. Feb 3, 2022 at 23:35
• Are you sure that this will work? Panorama stitiching software like Hugin is made to merge multiple frames taken from the same location, where the direction of the camera varies from frame to frame. OP needs to stich frames where the camera moved from frame to frame, but essentially pointing in the same direction. That will, mathematically speaking, be a quite different problem to solve. Feb 4, 2022 at 9:22
• As mentioned in my first sentence I have not tried this, so I'm not sure. However, the links in my answer (2nd link 5th paragraph and 2nd link footnote) seem to suggest that Hugin can handle panorama's taken from a moving point, it calls those mosaic panorama's. Feb 4, 2022 at 10:51

What you want to do is not particularly difficult, nor uncommon. The method Saaru suggested will work, as will the method you tried (manually in photoshop).

But your real issue is probably going to be getting images suitable to stitch together in the first place; primarily due to the rolling shutter characteristic of the phone's sensor readout.

If I snapshot frames one by one and put them one near another in Photoshop it's not what I want, it looks somehow distorted or something, I tried with 5-10 meters but a house turned squeezed.

As you have found, if you try to stitch images with rolling shutter artifacts the result will be distorted. You may be able to improve the results if you first edit the video with iMovie to fix the rolling shutter effect (there may be other programs that offer similar).

Even if you avoid/correct the rolling shutter issue, you still have the issue of recorded motion blur per frame. The standard SS for 30fps video is 1/60; that's not sufficient to freeze much motion/speed, particularly with motion parallax in a moving vehicle. You would likely need to use high speed video like 120fps w/ 1/240 SS (or higher), or have the ability to set a higher SS separately.

In the end, I think you would be much better off just taking multiple still images using a camera with a mechanical shutter (or very fast readout speed, or stationary).

• Thanks, guys, very useful information, I know the direction that I need to take now. I can upload this topic if needed with the results but I have no idea when I will actually get them. 👋 Feb 5, 2022 at 20:49

The effect could be directly achieved using a slit-scan camera. That is more or less how Google makes StreetView.

Disregarding the one edge of the first few frames and the other edge of the last few frames, use the column of pixels at the center of each and every frame as the slit.

So the bitmap would start with a “stack” of pixels from the center of the first frame.

A stack of pixels from the center of frame two would be next to it. A stack from frame three next to it and so on.

Depending on travel speed and video frame rate the slit might need to be wider (more than one pixel) than a single pixel or the sample rate lower than every frame.

Sharpness might be enhanced by combining pixels from several frames. This might also help reduce compression artifacts.

For a serious endeavor, it is perhaps worth considering building dedicated slit scan hardware…that’s what Google does.