1
\$\begingroup\$

So I've been on this problem for about a year now and haven't been able to solve it. I gave up on it a while back, but now I'm researched up and ready to get an answer.

I'm trying to fake a long exposure by taking multiple RAW photos and doing something with the Bayer filter values in order to get the effect of having left the shutter open for that long.

The idea I have in my mind is that, intuitively, long exposure works by leaving the shutter open for longer, and so the light sensor values (that RAW pictures store) are simply an aggregation of multiple shorter photos' values.

For example, if I have three 1/3 second-long photos taken without moving the camera, they should store the same information as a 1 second-long photo.

The problem I'm getting is with how I can quantify that aggregation. I found that many different methods fail, and I'm not sure why. If I simply average out the multiple photos, I get a heck of a lot of noise reduction. So that part is nice to have. I can later use Camera Raw to brighten up the photo and I'll get a close approximation to what I would have gotten with a 1 sec long exposure. The problem is, I want to do that on-device. I'm currently using an iPhone to test, but I don't think that should matter.

Current Setup & Formula

I'm using three 1/3 sec exposures to try and fake a 1 sec photo. My current formula is simple:

out_picture[x][y] = picture1[x][y] + picture2[x][y] + picture3[x][y]

The problem is that that gives me really pink pictures like so:

JPEG version of RAW file

I tried multiple different variations of the above formula. One that I thought was promising was the derivative of the sigmoid function. The reason why I used the derivative of the sigmoid is because I did a case-study where I studied the effect of final RGB outputs as a function of the "EV" slider value on Photoshop Camera Raw, and found a sigmoid-like pattern. Here is what that looks like when I plot it for some pixels:

Sigmoid maybe?

So I thought maybe the derivative of the sigmoid (high changes for middle values, low changes for low and high filter values) would fix things. I normalized the filter values to a 0-1 range by using the camera's black level and max bit value. I made some modifications to the derivative so that it would fit within the range of values 0-1, this is the function I ended up with. I then multiplied by number of pictures taken, to emulate the EV slider in Camera Raw.

Here is the formula for that (in python):

def exposureAdjust(x):
   black_level = 528    ## iPhone RAW Photos have a black_level of 528
   max_pixel_value = 2**14 - 1    ## 14-bit depth
   normalized = (x - black_level)/max_pixel_value    ## now x is between [0,1]
   EV = num_pics    ## faking EV value
   adjustment = EV * (np.exp(-10 *(x - 0.25)))/(np.exp(-10*(x - 0.25)) + 1)^2

   return min(x * (1+adjustment), max_pixel_value)    ## Cap at 2**14 


num_pics = 3
out_picture[x][y] = picture1[x][y] + picture2[x][y] + picture3[x][y]
out_picture[x][y] = exposureAdjust(out_picture[x][y]/3)

Running this formula, I get a similar result:

Still broken


Conclusion & Observations

As you can see, it's still pinkish. The pics above are JPEG's yes, but I'm actually working with the RAW's. I just had to upload them as JPEG's so they can be attached inline.

Pink happens when there is a lack of green. The camera has a bayer filter layout of RGGB, but I really don't see how I could be getting this effect. Anyone have any idea what is to blame here and how I can adjust it programmatically. I have the entire pipeline set up. I only need the formula, but somehow it's not working. I've coded this up in Python and Swift, and the results are the same. There's definitely an issue with the formula using the RAW bayer filter values. I think this is a cool challenge, but I've reached the end of my wits on this.

Any Thoughts?


UPDATE: Here are the histogram results for the picture before and after the exposureAdjust function is applied. It seems like the green channel needs to be adjusted differently?

\$\endgroup\$
9
  • 1
    \$\begingroup\$ In your example, what are you taking a picture of? It seems very dark and underexposed. \$\endgroup\$
    – scottbb
    Nov 26, 2019 at 7:01
  • 1
    \$\begingroup\$ Wait a minute, are you working directly with raw values, or with debayerized data? \$\endgroup\$
    – scottbb
    Nov 26, 2019 at 7:48
  • 1
    \$\begingroup\$ FWIW this is exactly what "Night Mode" on recent Android does. \$\endgroup\$
    – mattdm
    Nov 26, 2019 at 13:19
  • 1
    \$\begingroup\$ Do you have working code to correctly display a single image taken with a middle ("correct") exposure? \$\endgroup\$
    – mattdm
    Nov 26, 2019 at 13:23
  • 2
    \$\begingroup\$ I think you have a misunderstanding about stacking: don't underexpose each individual shot in the stack, expecting to substantially boost the exposure in post. Use the full range of your camera for each shot, and just average the shots together. \$\endgroup\$
    – scottbb
    Nov 26, 2019 at 16:57

2 Answers 2

1
\$\begingroup\$

Observations:

  1. You are summing the values of each pixel, but your exposureAdjust() function assumes that after conditioning, each pixel will be in the range [0, 1). This is not correct. Assume a pixel's value in each of the 3 input images is, say, 50% full-scale (thus, 213 = 8192). Summing that three times yields 3 * 8192 = 24,576. Then the result after normalized = (x - black_level)/max_pixel_value is about 1.47, definitely not normalized to less than 1.

  2. Remove your sigmoid adjustment = line until you get the rest of the logic sorted out (such as above). Here is a graph your sigmoid function:

    enter image description here

    Notice that for values of x above about 0.4, increasing x yields quickly smaller values of y (your "adjusted" value. For comparison, a non-adjusted value should correspond to the y = x linear plot on the graph.

  3. For normalization (assuming correctly-bounded inputs of [0, max_pixel_value - 1]), you should probably reduce the denominator by black_level as well (i.e., normalized = (x - black_level)/(max_pixel_value - black_level)). If you don't then your normalized values are in the range of [0, ∼0.969) (precisely, (213 - 511) / (214 - 1)).

\$\endgroup\$
3
  • \$\begingroup\$ 1. First block of code was just my initial trial. I 100% agree that that one was not normalized properly, I just wanted to show how wrongly normalized input yielded the same output. \$\endgroup\$
    – JoeVictor
    Nov 26, 2019 at 13:07
  • \$\begingroup\$ 2. So the reason why I wanted to have that function was because I thought there was some non-linearity with the way the bayer values were being scaled up. y=x would be ideal, but I thought maybe the cause of the issue was that dark pixels (0-0.5) weren't being scaled up enough. \$\endgroup\$
    – JoeVictor
    Nov 26, 2019 at 13:09
  • \$\begingroup\$ 3. Sure, this is a fair point, that's a bug in my calculations for sure. Unfortunately I don't think this will fix the issue :/ \$\endgroup\$
    – JoeVictor
    Nov 26, 2019 at 13:10
0
\$\begingroup\$

Your results are decidedly magenta and that is the opposite of where they should be. They should be green for an uncorrected bayer RGB.

I suspect that your observed sigmoid correction was heavily influenced by the raw data already having a white balance correction applied in ACR. You may even be getting "doubled correction" at this point, but I don't know.

This article on Uni-WB has a lot of information on the raw data corrections/methods and WB correction factors. http://www.guillermoluijk.com/tutorial/uniwb/index_en.htm

\$\endgroup\$
2
  • \$\begingroup\$ The RAW data is applied directly from the photos taken from the camera, never passes by ACR. You are right that there is definitely something wrong going on with the channels as here are the results before and after the exposureAdjustment function. Clearly the red and blue channels are affected completely differently. \$\endgroup\$
    – JoeVictor
    Nov 26, 2019 at 14:49
  • \$\begingroup\$ You stated that you came up with a sigmoid correction based upon what you observed using Photoshop Camera Raw... that observation had white balance corrections applied. What program are you using to render the manipulated raw data now? AFAIK, the only program that can render a raw file without applying WB corrections is DCRAW; maybe RawDigger. \$\endgroup\$ Nov 26, 2019 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.