46

I recently thought of a time lapse technique and want to know if there is there a name for it or if it has been widely used before.

I want to take a time lapse of my garden at 1 frame per day for a long time, say one year (similar to this post) : Doing a time lapse of my backyard for 1 year

Obviously the large problem with this method is that there are many things that will cause flicker or jitter in the image like differing brightness levels from clouds, sunrise and sunset patterns, and wind blowing the trees.

My idea was to use a raspberry pi to take thousands of pictures per day, say one every few seconds, and delete all of them in any given day apart from one. The one which is kept is the one which is closest to the previous day's frame.

I wrote a simple script to select the best frame for the day by comparing pixel by pixel and selecting the frame with the minimum sum of squared differences.

Obviously the first frame needs to be chosen manually.

Initial results after a week look promising, it's been able to get frames for each day with minimal difference to each other.

I want to know if this is a novel technique, or if it's been investigated before and there are any extra hints or tips other people have discovered about this particular method.

Edit: This received a lot more attention than I expected so I'll make a few notes here about things frequently raised in the coments.

  • Shadows: If there are strong shadows in the scene then this will mean that images need to be selected at almost the exact same time each day to avoid jerky shadows caused by the sun at different angles. I found that I can easily avoid this by picking the first frame when the scene isn't being lit by direct sunlight, such as at dawn or dusk or when there is light cloud cover. There are generally a few hours each day where there is adequate light for a good photo but no direct shadows, and the lighting is mid-level so it's fairly easy to find a frame with matching light levels the next day.
  • Snowfall, or other objects that are not in the scene for very long. It's quite clear that only comparing against the previous day's frame in the cost function could cause some pretty big problems. So to counteract this it would definitely help to use more than one previous frame in the cost function. What's not clear is exactly how to do this, since I can think of a few different techniques, but I'll experiment with a few. These could include making the cost function the sum of the difference between the previous frame and also the first frame, or constructing an artificial frame for today's cost function using the modal, or average pixels from the last 5 day's frames (amount not certain).
  • "A one-year timelapse of most-similar frames might look a bit dull and boring, since there'd be barely any "change" in the sequence". Of course, the algorithm will try to hide changes as far as possible, such as changes in brightness or changes in the position of the trees when blown by the wind, but there are things that will change in the sequence, that the algorithm can't hide, such as plants growing gradually over time, which is what I'm interested to see.
  • "keep in mind that 365 pictures give a very short video at 30 frames per second" : That's very true, I'll probably go for something like 10fps.

For those interested in seeing the results of my project, I intend to put the code on github and link it back here once I've tidied it and fixed a few minor issues. I'll also link back to the generated videos, but obviously it may take a long time to get those results.

The responses here give me enough ideas to implement at least 3 different algorithms, and I'll make the code put the outputs in different folders to keep them entirely separate. It will be interesting to see how the resulting videos differ.

  • 2
    @xenoid: Having shadows in opposite directions will incur a cost, and with enough images taken per day that cost should be able to be optimised away. In other words, it should be easy enough to find a picture the next day with a shadow in the same direction, and having the shadow in the same direction will minimise the difference, resulting in that picture being highly likely to be selected. – Karl Mar 30 at 11:19
  • 1
    Cost = diff(frame n, frame n-1) + diff(frame n, frame 0) – Karl Mar 30 at 19:08
  • 11
    As a dev who likes photography, I think your project is really cool. I'd like to point out, though, that a one-year timelapse of most-similar frames might look a bit dull and boring, since there'd be barely any "change" in the sequence. E.g.: winter → grey and cloudy, summer → bright and sunny; if your first picture is taken in winter, your algorithm might select summer pictures where the light wasn't as bright, maybe those where a lone cloud just happened to tone down the colors, so they match the winter ones more closely. I'd expect a 1y timelapse to portray seasonal change better than that. – walen Mar 31 at 6:18
  • 3
    Hello, I can't answer your question, but you might want to check that video by Sam Morrison. It's a video composed with more than a thousands of pictures from Instagram. It's "a bit" similar to your project. – Amaury Hanser Mar 31 at 7:06
  • 1
    I'm not sure what you want to do with it at the end but keep in mind that 365 pictures give a very short video at 30 frames per second. – steros Mar 31 at 12:40
8

Many time lapse cameras already do this, but on a shorter time scale. For example, GoPro mentions in TimeWarp documentation:

For the best results, try speeds 10x and up when shooting footage that may get bumpy. Higher speed will often provide better stabilization as there are more frames for TimeWarp to choose from

Which sounds like it is selecting the frames with least movement from the previous frame. Similarly in this Microsoft publication Real-Time Hyperlapse Creation Via Optimal Frame Selection:

We optimally select frames from the input video that best match a desired target speed-up while also resulting in the smoothest possible camera motion.

In addition to the similarity criterion, they are selecting for most stable movement speed between frames. When the camera is moving, the technique is usually called "Hyperlapse", but I couldn't find examples of it being applied to stationary camera and 24 hour intervals.

| improve this answer | |
  • Accepted because of the link to the Microsoft research, which most closely matches the algorithm I'm trying to develop, they've just used a moving camera instead of a stationary one. – Karl Apr 2 at 11:29
39

Well, that is a so-called greedy algorithm that may lead into dead ends unnecessarily that are expensive to get out again. If you instead want to minimize the total change, you'd keep all images and record for each image a) the best preceding image for an optimal sequence arriving here b) the cost of arriving here via the optimal sequence.

You can prune comparisons by noting that any sequence that is already more expensive than the current optimum candidate without even adding the newest distance can be dropped without doing an actual comparison. Images that end up in "dead sequences" that can never be part of an optimum sequence regardless of what comes up next can be pruned altogether. That way, the active set of images that needs to be kept is confined, though with the close similarity of your images, there may be "close races" for a long time keeping numerous candidates active.

| improve this answer | |
  • 4
    +1 This makes perfect sense and is a great addition to the technique that I didn't think of. Without any constraint like this, there may be a drift over a long period of time into an undesired state, for example drift into the night. Although it doesn't exactly answer the question: are you aware of this technique being known or studied by others before? – Karl Mar 30 at 10:54
  • 4
    This approach is a good candidate for so-called dynamic programming. To find the minimum-difference path to get to Day 10, just add the step cost to the minimum-difference path cost for each choice of Day 9. To find those minimum-difference paths to Day 9, add the step cost to the minimum-difference paths costs for Day 8, and so on. With N images at each of T time points, this is O(TN^2) rather than the O(N^T) naive method of building each sequence from scratch. – Nuclear Wang Mar 30 at 19:20
  • 3
    An equivalent way (to @NuclearWang's dynamic programming solution) to think about it is that you're building a graph of O(TN) nodes containing O(TN^2) edges, then finding the shortest path. The advantage of doing it this way is that you only need to calculate the neighbors of nodes that are actually expanded during the search. – BlueRaja - Danny Pflughoeft Mar 31 at 3:03
  • 4
    In other words, the Dijkstra algorithm – Roman Odaisky Mar 31 at 13:45
13

What you describe is similar to what astrophotographers call lucky imaging (or speckle imaging). The idea behind lucky imaging is to take several (hundreds, even thousands) of very short exposures and only keep the few images that appear to be the least disturbed by atmospheric distortions.

In a general sense, taking more images than needed in order to select the best ones later is (sometimes derisively) called spray & pray. =)

Regarding novelty to your specific use-case, I’ve seen videographers create time lapses by recording segments of high-resolution video for each time lapse frame period, and either average the frames of each video segment, or select the best frame from each segment.

| improve this answer | |
  • 4
    Pray & Spray: The goto- photography method of every camera raised in the air to get a shot of the action. Got some great shots of that in college during football games, but remember- you only had 36 chances. – J.Hirsch Mar 30 at 20:25
  • 3
    @J.Hirsch back in the film day, you could tell a pro from the hobbyists because he took a lot of shots. But in digital, you can tell a pro by how few shots he needs to take. I’m ashamed to say that sometimes I just hold down the shutter and machine-gun until the buffer lag gets too annoying... – scottbb Mar 30 at 22:51
  • ...or take the best pixels from each image, for some definition of "best". Choosing the most common value (mode) is one approach to removing foreground obstructions and creating a composite image of a background (like removing tourists from burst photos of a famous landmark). In OP's case, "best" would be the one that best matches whatever similarity function they're using. – A C Mar 30 at 23:00
  • @AC: Alternatively, take the average value of a pixel in all the shots, identify all pixels in all shots that are substantially different, and then take the average of all pixels from all shots excluding those that deviated from the average, and reducing the weight of those which are near the excluded ones. – supercat Mar 31 at 16:01
  • @scottbb I totally agree with you. It was always 3:1 for film, and now it's 1:3 for digital. I still hate editing in digital far more than I did in film. There's just SO much more. – J.Hirsch Apr 1 at 22:53
2

tl;dr Sounds like you want a feature-vectorized video recording where minor/transient features are filtered out, producing a video that shows the conceptual evolution of the scene over time.


You probably want a video of the major features' evolution.

If you just wanted a time-lapse video of images, you could just do that.

However, you're asking about a time-lapse sequence with minimal changes between the frames. I'm guessing what you truly want is a video that focuses on the overarching narrative captured by the camera. This is:

  • You do want to capture how the major features in the scene change over time.

  • You don't want the video to look jittery due, say, it being cloudy one day but not the next, or random, uninteresting objects getting in the shot one day but not any other.

For example, say there's a clock in the scene:

  • You do want to capture how that clock's appearance changes over time if, say, its paint gets lighter in the sunlight, or if starts to get scratches/cracks over time.

  • You don't want to see the clock's time randomly skipping around from frame-to-frame.

Going a step further:

  • You do want to be able to play the time-lapse video with a higher frame rate, showing the clock's time smoothly transitioning as though this were a real-time video, even though the camera never actually captured images of the clock at all of those times.

In other words, you probably don't want the computer to record random pixels, like in a bitmap, but rather you want a video composed of conceptual components that the computer can display.


Generate a vectorized video by extracting features from the raw bitmaps.

Feature extraction isolates features from a scene.

For example, have you seen one of those "Law & Order"-type TV shows where someone who witnesses a crime would describe a suspect to a police sketch artist, who will then try to draw the suspect?

You want an algorithm that witnesses the raw data from the camera, then produces a video file that describes what it witnessed rather than literally copy/pasting the pixels. Then you want software that animates a scene based on that video-file's description, much like a police-sketch-artist drawing each frame based on a witness's recollection, rather than a traditional video-player just displaying colored dots.


Filter out the noisy features to get the desired video.

If you fully vectorize the raw video, you'd just end up with a new video file that shows the exact same thing, just stored/processed in a different format.

The magic comes from being able to display the video without minor features. For example, say that a random person walks in front of the camera in one shot; that person's presence in that frame is a feature which you'd presumably want to exclude. So, you'd just need to tell the video-player which features to display – presumably you'd want to show features that smoothly stretch over a non-trivial number of frames.


Technical implementation: What software to use?

I'm gonna dodge recommending any specific software, libraries, etc.. Part to be lazy, and part because this is an emerging practice where things are likely to change quickly.


Summary

Sounds like you want to generate feature-vectorized videos from a time-lapse of still images, enabling you to view videos of the conceptual time-evolution of the scene without the noisy distractions.

A lot of research is going into this area since it's useful in a lot of things; for example, vectorizing a bunch of frames from a self-driving car's cameras could generate a model of the world around the car, enabling it to make correct driving decisions. Much like you, self-driving cars don't care about the noise; they just want to understand the important features of the scene, e.g. where objects are, rather than exact pixel-level data from raw images.

It'd be neat to see someone apply this to time-lapse videography!

| improve this answer | |
  • 1
    +1 for the eloquent description of the things that I do and don't want. Although I'm not familiar enough with the solution technique you describe to make an implementation with the limited technical details described here. – Karl Apr 2 at 11:21
1

Since you are using a raspberry pi, have you thought about connecting it up to some fill lights. You could set it up so the lights only turn on for some of the pictures if you still want to do a lot of shots during the day. At least that way, your plant is guaranteed to have some kind of standard amount of light on it regardless of what is going on in the background of the image.

| improve this answer | |
  • Good idea, but I wanted to create an algorithm that doesn't have to rely on external support so that it can be applied to a wider range of scenarios. For example, what if I live in a city apartment with a good view of some construction projects going on, and want a time lapse of skyscrapers rising? – Karl Apr 2 at 11:26
0

In terms of enduring timelaps, my hero in the field is Joe DiGiovanna. From the shore of Weehawken, New Jersey, he takes each 30 s a photo while aligned to NYC (2,880 images each day). In fact, it is a 30 yr project he started more than 4 years ago. Using an ardunio-based intervalometer, dedicated laptop, battery backup power, etc. See his website or a report by CNN style about him, including examples like July 13, 2019 (a larger blackout in the city) or the rise of sky scrapers.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.