Has my idea for a time lapse technique been used before?

Question

I recently thought of a time lapse technique and want to know if there is there a name for it or if it has been widely used before.

I want to take a time lapse of my garden at 1 frame per day for a long time, say one year (similar to this post) : Doing a time lapse of my backyard for 1 year

Obviously the large problem with this method is that there are many things that will cause flicker or jitter in the image like differing brightness levels from clouds, sunrise and sunset patterns, and wind blowing the trees.

My idea was to use a raspberry pi to take thousands of pictures per day, say one every few seconds, and delete all of them in any given day apart from one. The one which is kept is the one which is closest to the previous day's frame.

I wrote a simple script to select the best frame for the day by comparing pixel by pixel and selecting the frame with the minimum sum of squared differences.

Obviously the first frame needs to be chosen manually.

Initial results after a week look promising, it's been able to get frames for each day with minimal difference to each other.

I want to know if this is a novel technique, or if it's been investigated before and there are any extra hints or tips other people have discovered about this particular method.

Edit: This received a lot more attention than I expected so I'll make a few notes here about things frequently raised in the coments.

• Shadows: If there are strong shadows in the scene then this will mean that images need to be selected at almost the exact same time each day to avoid jerky shadows caused by the sun at different angles. I found that I can easily avoid this by picking the first frame when the scene isn't being lit by direct sunlight, such as at dawn or dusk or when there is light cloud cover. There are generally a few hours each day where there is adequate light for a good photo but no direct shadows, and the lighting is mid-level so it's fairly easy to find a frame with matching light levels the next day.
• Snowfall, or other objects that are not in the scene for very long. It's quite clear that only comparing against the previous day's frame in the cost function could cause some pretty big problems. So to counteract this it would definitely help to use more than one previous frame in the cost function. What's not clear is exactly how to do this, since I can think of a few different techniques, but I'll experiment with a few. These could include making the cost function the sum of the difference between the previous frame and also the first frame, or constructing an artificial frame for today's cost function using the modal, or average pixels from the last 5 day's frames (amount not certain).
• "A one-year timelapse of most-similar frames might look a bit dull and boring, since there'd be barely any "change" in the sequence". Of course, the algorithm will try to hide changes as far as possible, such as changes in brightness or changes in the position of the trees when blown by the wind, but there are things that will change in the sequence, that the algorithm can't hide, such as plants growing gradually over time, which is what I'm interested to see.
• "keep in mind that 365 pictures give a very short video at 30 frames per second" : That's very true, I'll probably go for something like 10fps. Also, there's no reason why the same algorithm can't be applied to any time span. For example: for every hour, keep the picture that's most similar to the picture kept on the hour before, for a 1 frame per hour video.

Answer 1

Many time lapse cameras already do this, but on a shorter time scale. For example, GoPro mentions in TimeWarp documentation:

For the best results, try speeds 10x and up when shooting footage that may get bumpy. Higher speed will often provide better stabilization as there are more frames for TimeWarp to choose from

Which sounds like it is selecting the frames with least movement from the previous frame. Similarly in this Microsoft publication Real-Time Hyperlapse Creation Via Optimal Frame Selection:

We optimally select frames from the input video that best match a desired target speed-up while also resulting in the smoothest possible camera motion.

In addition to the similarity criterion, they are selecting for most stable movement speed between frames. When the camera is moving, the technique is usually called "Hyperlapse", but I couldn't find examples of it being applied to stationary camera and 24 hour intervals.

Answer 2

Well, that is a so-called greedy algorithm that may lead into dead ends unnecessarily that are expensive to get out again. If you instead want to minimize the total change, you'd keep all images and record for each image a) the best preceding image for an optimal sequence arriving here b) the cost of arriving here via the optimal sequence.

You can prune comparisons by noting that any sequence that is already more expensive than the current optimum candidate without even adding the newest distance can be dropped without doing an actual comparison. Images that end up in "dead sequences" that can never be part of an optimum sequence regardless of what comes up next can be pruned altogether. That way, the active set of images that needs to be kept is confined, though with the close similarity of your images, there may be "close races" for a long time keeping numerous candidates active.

Answer 3

What you describe is similar to what astrophotographers call lucky imaging (or speckle imaging). The idea behind lucky imaging is to take several (hundreds, even thousands) of very short exposures and only keep the few images that appear to be the least disturbed by atmospheric distortions.

In a general sense, taking more images than needed in order to select the best ones later is (sometimes derisively) called spray & pray. =)

Regarding novelty to your specific use-case, I’ve seen videographers create time lapses by recording segments of high-resolution video for each time lapse frame period, and either average the frames of each video segment, or select the best frame from each segment.

Answer 4

tl;dr– Sounds like you want a feature-vectorized video recording where minor/transient features are filtered out, producing a video that shows the conceptual evolution of the scene over time.

---

You probably want a video of the major features' evolution.

If you just wanted a time-lapse video of images, you could just do that.

However, you're asking about a time-lapse sequence with minimal changes between the frames. I'm guessing what you truly want is a video that focuses on the overarching narrative captured by the camera. This is:

• You do want to capture how the major features in the scene change over time.

• You don't want the video to look jittery due, say, it being cloudy one day but not the next, or random, uninteresting objects getting in the shot one day but not any other.

For example, say there's a clock in the scene:

• You do want to capture how that clock's appearance changes over time if, say, its paint gets lighter in the sunlight, or if starts to get scratches/cracks over time.

• You don't want to see the clock's time randomly skipping around from frame-to-frame.

Going a step further:

• You do want to be able to play the time-lapse video with a higher frame rate, showing the clock's time smoothly transitioning as though this were a real-time video, even though the camera never actually captured images of the clock at all of those times.

In other words, you probably don't want the computer to record random pixels, like in a bitmap, but rather you want a video composed of conceptual components that the computer can display.

---

Generate a vectorized video by extracting features from the raw bitmaps.

Feature extraction isolates features from a scene.

For example, have you seen one of those "Law & Order"-type TV shows where someone who witnesses a crime would describe a suspect to a police sketch artist, who will then try to draw the suspect?

You want an algorithm that witnesses the raw data from the camera, then produces a video file that describes what it witnessed rather than literally copy/pasting the pixels. Then you want software that animates a scene based on that video-file's description, much like a police-sketch-artist drawing each frame based on a witness's recollection, rather than a traditional video-player just displaying colored dots.

---

Filter out the noisy features to get the desired video.

If you fully vectorize the raw video, you'd just end up with a new video file that shows the exact same thing, just stored/processed in a different format.

The magic comes from being able to display the video without minor features. For example, say that a random person walks in front of the camera in one shot; that person's presence in that frame is a feature which you'd presumably want to exclude. So, you'd just need to tell the video-player which features to display – presumably you'd want to show features that smoothly stretch over a non-trivial number of frames.

---

Technical implementation: What software to use?

I'm gonna dodge recommending any specific software, libraries, etc.. Part to be lazy, and part because this is an emerging practice where things are likely to change quickly.

---

Summary

Sounds like you want to generate feature-vectorized videos from a time-lapse of still images, enabling you to view videos of the conceptual time-evolution of the scene without the noisy distractions.

A lot of research is going into this area since it's useful in a lot of things; for example, vectorizing a bunch of frames from a self-driving car's cameras could generate a model of the world around the car, enabling it to make correct driving decisions. Much like you, self-driving cars don't care about the noise; they just want to understand the important features of the scene, e.g. where objects are, rather than exact pixel-level data from raw images.

It'd be neat to see someone apply this to time-lapse videography!

Answer 5

Since you are using a raspberry pi, have you thought about connecting it up to some fill lights. You could set it up so the lights only turn on for some of the pictures if you still want to do a lot of shots during the day. At least that way, your plant is guaranteed to have some kind of standard amount of light on it regardless of what is going on in the background of the image.

Answer 6

In terms of enduring timelaps, my hero in the field is Joe DiGiovanna. From the shore of Weehawken, New Jersey, he takes each 30 s a photo while aligned to NYC (2,880 images each day). In fact, it is a 30 yr project he started more than 4 years ago. Using an ardunio-based intervalometer, dedicated laptop, battery backup power, etc. See his website or a report by CNN style about him, including examples like July 13, 2019 (a larger blackout in the city) or the rise of sky scrapers.