Automatic identification of lucky images for landscape photography

Question

I recently did a trial run for a project to photograph a certain rock climbing area from a variety of directions. I took shots at a couple of different times of day from the same spot, tried different exposures, and took 16 shots for the setups that seemed the best. The best shot from the best session was this one (thumbnail below). I had originally thought that I would try stacking images, but the result of stacking the 16 shots looked worse (less detail) than the best individual shot. Atmospheric turbulence was clearly visible through the lens, as the hot afternoon sun hit the west side of the rock. Although I got some shots in the morning with indirect light, they actually showed less detail because of the lack of shadows and contrast, and the lower light levels. There is no location from which I can get the right perspective on my subject from nearby, so all my images are going to have to be shot with a 135 mm or 300 mm lens from far away.

Is there some way to automate the process of looking for the lucky images from a set of landscape photos like this? Since people doing lucky imaging with astronomical photography sometimes do hundreds of shots, I assume they automate this somehow, but I don't know if the algorithms they use are appropriate for landscape photography. This project will involve lots of driving and hiking, so I would be happy to take hundreds of exposures in a row if it would be likely to be helpful. I'm looking for an open-source solution that works on linux, preferably something I can do from the command line.

Or would I be likely to get better results by using a video technique? Impressive results are described here. But it seems like to do this you need to be able to shoot about 10 frames a second, and I haven't figured out if my camera can do this. (I have a Fuji x-e1.)

[EDIT] Having looked around on the web a little more, I think I have some partial answers to my own question. (1) Amateur astronomers using lucky imaging all seem to be using Windows and distributing "shareware" Windows software. (2) AFAICT the astronomical techniques all involve picking a "reference star," which is assumed to be a point source. If you have such a reference star, then it's fairly trivial to get a measure of image quality. A common one seems to be the Strehl ratio, which is basically the peak intensity of the image of your reference star. So if these impressions are correct, then it may make sense for me to try to roll my own image selection algorithm for landscapes.

Answer 1

This is not a definitive answer to my own question, but it's too long for a comment.

I implemented the idea of using the RMS Laplacian. The idea is that if the brightness of the image is represented by an array of pixels a[i,j], then at any point (i,j), we have the discrete approximation to the Laplacian L=a[i-1,j]+a[i+1,j]+a[i,j-1]+a[i,j+1]-4a[i,j]. This measures the sharpness of features in the image. For example, if the image was out of focus, L would be lower. The RMS value of the Laplacian, R, is the square root of the mean of the square of the Laplacian.

Here is my code that calculates R for an input PNG image:

#!/usr/bin/ruby

# To batch convert a bunch of JPGs to png:
# perl -e '$i=0; foreach $f(<*.JPG>) {$s=sprintf("%03d",$i); $c="convert $f $s.png"; print "$c\n"; system($c); $i=$i+1;}'

require 'oily_png'

# require 'hsluv'
  # http://www.hsluv.org
  # https://github.com/hsluv/hsluv-ruby
  # sudo gem install hsluv

# Sloppy and probably not physiologically valid, but fast.
# Returns an integer from 0 to 255*3.
def color_to_brightness(c)
  return ChunkyPNG::Color::r(c)+ChunkyPNG::Color::g(c)+ChunkyPNG::Color::b(c)
end


def rms_laplacian_from_file(input_file)
  image = ChunkyPNG::Image.from_file(input_file)
  n = 0
  sum = 0
  sum_sq = 0
  w = image.width
  h = image.height
  1.upto(w-2) { |i|
    ### if i%1000==0 then print "i=#{i}\n" end # show progress
    next unless i>w/3 && i<(2*w)/3 ## for efficiency, only use center of frame
    1.upto(h-2) { |j|
      next unless j>h/3 && j<(2*h)/3 ## for efficiency, only use center of frame
      next unless rand(10)==0 # for efficiency
      a = Hash.new
      (-1).upto(1) { |k|
        (-1).upto(1) { |l|
          c = image[i+k,j+l] # color, represented as a 4-byte rgba value
          a[[k,l]] = color_to_brightness(c)
        }
      }
      laplacian = a[[1,0]]+a[[-1,0]]+a[[0,1]]+a[[0,-1]]-4*a[[0,0]]
      n = n+1
      sum = sum + laplacian
      sum_sq = sum_sq + laplacian*laplacian
    }
  }
  sum = sum.to_f/n
  sum_sq = sum_sq.to_f/n
  rms = Math::sqrt(sum_sq-sum*sum)
  return rms
end

ARGV.each { |input_file|
  rms = rms_laplacian_from_file(input_file)
  print "#{input_file} -- rms=#{rms}\n"
}

This is implemented in Ruby and runs on Linux using the open-source oily_png library. If anyone's interested in trying it, it should require almost no modification to run on other platforms, if you have Ruby and oily_png installed.

To test that it measures sharpness, I took the first image of my set of 16, measured R, and then added a 5-pixel gaussian blur using GIMP and re-measured R. The result was R=30.8 before the blur, and R=7.8 after. So this does seem to confirm that it measures sharpness.

My 16 images are numbered 000 to 015. Looking at the images by eye, I had previously picked out image 003 as the best. That was the image that I posted a link to in the question.

I ran my code on the 16 shots I had taken, and got the following output:

000.png -- rms=30.809465960392004
001.png -- rms=31.215359700578606
002.png -- rms=31.909926250066476
003.png -- rms=31.83243374839454
004.png -- rms=31.310612756003305
005.png -- rms=30.353258897447564
006.png -- rms=30.61244684985801
007.png -- rms=30.882745734215135
008.png -- rms=28.667104210689384
009.png -- rms=29.862966602367973
010.png -- rms=29.72001987743495
011.png -- rms=30.51274847773823
012.png -- rms=30.84316910530572
013.png -- rms=29.21751498027252
014.png -- rms=29.067434969521976
015.png -- rms=30.831305018709617

Of the 16 images, my choice had the second-highest R value. This would seem to confirm that this statistic could be a useful as an alternative to inspecting images and judging them subjectively by eye.

My implementation is pretty slow, and to make up for that I did some things to improve its performance. I only inspect the middle of the field, and I only sample the Laplacian at 1/10 of the points. In a more optimized implementation, these shortcuts could be eliminated if desired.

It later occurred to me that there might be a much simpler way to do this. An image with more detail should not compress as well, so the largest JPG file might simply be the best one. Sure enough, doing an ls -lS to list the files in order of decreasing size gave a list that was very nearly in the same order as the files sorted by decreasing R:

-rw-rw-r-- 1 bcrowell bcrowell 16970354 Oct 25 15:48 003.png
-rw-rw-r-- 1 bcrowell bcrowell 16927174 Oct 25 15:48 002.png
-rw-rw-r-- 1 bcrowell bcrowell 16903104 Oct 25 15:48 004.png
-rw-rw-r-- 1 bcrowell bcrowell 16882373 Oct 25 15:47 000.png
-rw-rw-r-- 1 bcrowell bcrowell 16861082 Oct 25 15:47 001.png
-rw-rw-r-- 1 bcrowell bcrowell 16817527 Oct 25 15:48 006.png
-rw-rw-r-- 1 bcrowell bcrowell 16816529 Oct 25 15:49 011.png
-rw-rw-r-- 1 bcrowell bcrowell 16793982 Oct 25 15:49 012.png
-rw-rw-r-- 1 bcrowell bcrowell 16786443 Oct 25 15:48 009.png
-rw-rw-r-- 1 bcrowell bcrowell 16773575 Oct 25 15:48 005.png
-rw-rw-r-- 1 bcrowell bcrowell 16771759 Oct 25 15:49 010.png
-rw-rw-r-- 1 bcrowell bcrowell 16765674 Oct 25 15:48 007.png
-rw-rw-r-- 1 bcrowell bcrowell 16764562 Oct 25 15:49 015.png
-rw-rw-r-- 1 bcrowell bcrowell 16750179 Oct 25 15:48 008.png
-rw-rw-r-- 1 bcrowell bcrowell 16732854 Oct 25 15:49 013.png
-rw-rw-r-- 1 bcrowell bcrowell 16684073 Oct 25 15:49 014.png

Answer 2

Is there some way to automate the process of looking for the lucky images from a set of landscape photos like this?

As I understand it, your definition of "lucky image" is one which is sharper than the average. Since many cameras use measurement of sharpness of (a region of) an image in their autofocus mechanisms¹, it's clear that there is some way to automate its measurement. However, the advantages of different approaches and the possibility of combining them is a subject of active research, so you can't expect a definitive answer. E.g. Robust Automatic Focus Algorithm for Low Contrast Images Using a New Contrast Measure, Jinshan Tang et. al., Sensors (2011) says that

Many contrast measures have been used for passive AF ... Results indicate that 2D spatial measurement methods such as Tenengrad, Prewitt Edge detection, and Laplacian yield best performance in terms of accuracy and unimodality. However, they are very sensitive to noise, and not robust to different scene conditions such as low light conditions.

On the contrary, variance based methods are fast and robust. The basic idea is to calculate the variance of image intensity. The image is best focused when the variance reaches a maximum. A typical method in discrete cosine transform (DCT) domain is to compute the AC coefficients of images, which can also be used to represent information about variance function of the luminance.

Since you're not constrained to realtime performance on a low-powered processor in camera, you could probably implement multiple approaches and try to combine their scores. You talk about stacking, so I assume you have multiple shots taken from the same position in about the same light conditions: it might be worthwhile thinking about stitching rather than stacking, so that you select the photo which has the best contrast in one area, and the photo which has the best contrast in another, and stitch them together. I'm not sure to what extent existing stitching software supports this kind of contrast selection stitch.

---

¹ To be precise, those which use contrast detection autofocus as opposed to phase detection autofocus. And yes, I know that DSLRs with live view use PDAF with the mirror down and CDAF in live view.

Answer 3

This is probably too far out to be of much interest, but: the Laplacian in one dimension boosts high-frequency content by multiplying the Fourier spectrum by the square of frequency. (Hope I got that right.) So it amplifies noise as well as signal. It might be interesting to see if an edge-detector could extract features of interest. Given a particular edge-detector, perhaps the photo with the longest total edge length would be of interest. This might just be tail-chasing, but perhaps not?