Emulate Shutter Duration on RAW Photos using Exposure Stacking

Question

So I've been on this problem for about a year now and haven't been able to solve it. I gave up on it a while back, but now I'm researched up and ready to get an answer.

I'm trying to fake a long exposure by taking multiple RAW photos and doing something with the Bayer filter values in order to get the effect of having left the shutter open for that long.

The idea I have in my mind is that, intuitively, long exposure works by leaving the shutter open for longer, and so the light sensor values (that RAW pictures store) are simply an aggregation of multiple shorter photos' values.

For example, if I have three 1/3 second-long photos taken without moving the camera, they should store the same information as a 1 second-long photo.

The problem I'm getting is with how I can quantify that aggregation. I found that many different methods fail, and I'm not sure why. If I simply average out the multiple photos, I get a heck of a lot of noise reduction. So that part is nice to have. I can later use Camera Raw to brighten up the photo and I'll get a close approximation to what I would have gotten with a 1 sec long exposure. The problem is, I want to do that on-device. I'm currently using an iPhone to test, but I don't think that should matter.

Current Setup & Formula

I'm using three 1/3 sec exposures to try and fake a 1 sec photo. My current formula is simple:

out_picture[x][y] = picture1[x][y] + picture2[x][y] + picture3[x][y]

The problem is that that gives me really pink pictures like so:

I tried multiple different variations of the above formula. One that I thought was promising was the derivative of the sigmoid function. The reason why I used the derivative of the sigmoid is because I did a case-study where I studied the effect of final RGB outputs as a function of the "EV" slider value on Photoshop Camera Raw, and found a sigmoid-like pattern. Here is what that looks like when I plot it for some pixels:

So I thought maybe the derivative of the sigmoid (high changes for middle values, low changes for low and high filter values) would fix things. I normalized the filter values to a 0-1 range by using the camera's black level and max bit value. I made some modifications to the derivative so that it would fit within the range of values 0-1, this is the function I ended up with. I then multiplied by number of pictures taken, to emulate the EV slider in Camera Raw.

Here is the formula for that (in python):

def exposureAdjust(x):
   black_level = 528    ## iPhone RAW Photos have a black_level of 528
   max_pixel_value = 2**14 - 1    ## 14-bit depth
   normalized = (x - black_level)/max_pixel_value    ## now x is between [0,1]
   EV = num_pics    ## faking EV value
   adjustment = EV * (np.exp(-10 *(x - 0.25)))/(np.exp(-10*(x - 0.25)) + 1)^2

   return min(x * (1+adjustment), max_pixel_value)    ## Cap at 2**14 


num_pics = 3
out_picture[x][y] = picture1[x][y] + picture2[x][y] + picture3[x][y]
out_picture[x][y] = exposureAdjust(out_picture[x][y]/3)

Running this formula, I get a similar result:

---

Conclusion & Observations

As you can see, it's still pinkish. The pics above are JPEG's yes, but I'm actually working with the RAW's. I just had to upload them as JPEG's so they can be attached inline.

Pink happens when there is a lack of green. The camera has a bayer filter layout of RGGB, but I really don't see how I could be getting this effect. Anyone have any idea what is to blame here and how I can adjust it programmatically. I have the entire pipeline set up. I only need the formula, but somehow it's not working. I've coded this up in Python and Swift, and the results are the same. There's definitely an issue with the formula using the RAW bayer filter values. I think this is a cool challenge, but I've reached the end of my wits on this.

Any Thoughts?

---

UPDATE: Here are the histogram results for the picture before and after the exposureAdjust function is applied. It seems like the green channel needs to be adjusted differently?

Answer 1

Observations:

1. You are summing the values of each pixel, but your exposureAdjust() function assumes that after conditioning, each pixel will be in the range [0, 1). This is not correct. Assume a pixel's value in each of the 3 input images is, say, 50% full-scale (thus, 2¹³ = 8192). Summing that three times yields 3 * 8192 = 24,576. Then the result after normalized = (x - black_level)/max_pixel_value is about 1.47, definitely not normalized to less than 1.

2. Remove your sigmoid adjustment = line until you get the rest of the logic sorted out (such as above). Here is a graph your sigmoid function:

   

   Notice that for values of x above about 0.4, increasing x yields quickly smaller values of y (your "adjusted" value. For comparison, a non-adjusted value should correspond to the y = x linear plot on the graph.

3. For normalization (assuming correctly-bounded inputs of [0, max_pixel_value - 1]), you should probably reduce the denominator by black_level as well (i.e., normalized = (x - black_level)/(max_pixel_value - black_level)). If you don't then your normalized values are in the range of [0, ∼0.969) (precisely, (2¹³ - 511) / (2¹⁴ - 1)).

Answer 2

Your results are decidedly magenta and that is the opposite of where they should be. They should be green for an uncorrected bayer RGB.

I suspect that your observed sigmoid correction was heavily influenced by the raw data already having a white balance correction applied in ACR. You may even be getting "doubled correction" at this point, but I don't know.

This article on Uni-WB has a lot of information on the raw data corrections/methods and WB correction factors. http://www.guillermoluijk.com/tutorial/uniwb/index_en.htm