1
\$\begingroup\$

All cameras capture their data off the sensor, then they might save the image in RAW, but can convert to JPG. A RAW file can be 12 or more bits, but a JPG file is only 8 bits.

What process does a camera use to reduce the bit size to 8? Does it just chop off the RAW histogram tails, or does it do some form of gentle tone mapping?

It might vary between cameras, so information relating to the Olympus OM-D W-M1 mk2 and Canon M50 mk2 would be most useful to me.

Thanks!

\$\endgroup\$
6
  • 3
    \$\begingroup\$ Could you clarify the photographic problem you're trying to solve here? Remember that a RAW file is not an image; you can't just "chop off the RAW histogram tails" because you won't have any colour information. \$\endgroup\$
    – Philip Kendall
    Jun 11, 2022 at 10:41
  • \$\begingroup\$ A JPEG file is actually 24-bits per pixel. 8-bits for the Red channel, 8-bits for the Green channel, and 8-bits for the Blue channel. Raw files only have a single 12-bit or 14-bit luminance value per photosite (a/k/a sensel or pixel well). \$\endgroup\$
    – Michael C
    Jun 12, 2022 at 4:27
  • \$\begingroup\$ Thanks, Philip. The "problem" is making simple sense of observations and advice. For example, I have a friend who uses Sony cameras, only uses jpg, and his images frequently are technically "better" (in my perception) than my Olympus and Canon ones, whether I go via RAW or directly to jpg. I suspect that the Sony camera processes his jpgs rather better than my Olympus or Canon manage for the type of photos he takes. I often use HDR to "improve" my images, and I am struggling to understand this difference. It would be useful to know and would help in the talk I'm preparing (see below). \$\endgroup\$
    – Stuart444
    Jun 12, 2022 at 6:58
  • \$\begingroup\$ ....... I'm starting to understand that my knowledge of digital imaging theory is too weak to understand this "problem". I had imagined that a RAW file in some state of development must pass a stage where a histogram spanning up to 12+ bits must occur (since sensors have this latitude at low ISO) which then is processed to a jpg by rejecting bright or dark information or by tone mapping. I'm trying to reconcile information from lots of sources aimed at users. \$\endgroup\$
    – Stuart444
    Jun 12, 2022 at 8:11
  • \$\begingroup\$ @Stuart444 Is it possible that his Sony cameras have full frame sensors while your Olympus cameras are probably m4/3 with sensors one-quarter the area of a FF and your Canon cameras have APS-C sensors that are roughly 40% the area of a FF sensor? Even when sensors are the same size, some are just better than others with regard to things like signal-to-noise ratio, resolution, etc. Some have stronger or weaker anti-aliasing filters which intentionally blur on a microscopic level to prevent false color moire due to fixed patterns in the subjects being photographed, etc. \$\endgroup\$
    – Michael C
    Jun 12, 2022 at 22:10

3 Answers 3

1
\$\begingroup\$

While the RAW-to-JPEG processing is a huge subject and can only be summarized as "camera can do anything", the question seem to stem from the basic assumption that there is a huge excess of data in the 12-bit RAW form, which can only barely be squeezed into 8-bit JPEG.

No, the sensor manufacturers do not unnecessarily waste resources by producing those 12 bits images only to immediately throw away 1/3 of the bits.

The missing detail is the encoding. Raw sensor data is linear while JPEG (and PNG and display screens in general) are gamma corrected. The 12-bit linear range is "accidentally" similar to the 8-bit sRGB range (only slightly bigger, but not 16 times bigger like it would be in the linear case).

Of course it still means the amount of data is reduced, but it naturally follows from the nonlinearity and all the input bits actually take part in the process. The excess provides some headroom for color and exposure correction and allows for increased precision.

Now, the question can be rephrased for the 14- or 16-bit raws... this time there is no excuse, those cameras really produce "too much" data and allow for greater freedom in postprocessing corrections, tonemapping, shadow/hightlight recovery, etc.

\$\endgroup\$
3
  • \$\begingroup\$ Thanks. I'm preparing a talk for our photographic club on HDR. The process of tone mapping is well described on the internet. However in the course of my work I realised that when a camera processes its data to a jpg, it must do something with the darker tones and lighter tones that are outside the 8-bit range of the jpg. It seems to me that it can either throw away those out-of-range tones, or it must tone map . My Olympus takes HDR brackets and will merge them in camera, and the result is quite different to what it will produce when set to taking a single frame to jpg. \$\endgroup\$
    – Stuart444
    Jun 11, 2022 at 14:59
  • \$\begingroup\$ @Stuart444, nothing is necessarily "outside the 8-bit range"... but the more you try to encode with 8-bit the more compression (rounding) is required, and it eventually results in banding. E.g. you have 256 numbers available to describe 256 tonal values; those tonal values can be very close together (smooth transitions) or they can be far apart (banding), it doesn't matter. BUT, most of the jpeg banding results are due to less accurate 8-bit post processing math (which is why HDR post processing is typically done in 32-bit). \$\endgroup\$ Jun 11, 2022 at 17:48
  • \$\begingroup\$ @Stuart444 You're making the incorrect assumption that the number of bits must equal the number of steps of dynamic range in the scene, or even the number of steps of the DR of the sensor. That is no more the case than that Ansel Adams' Zone System which had 11 zones had to equate to 11 stops of DR in the scene captured. Nothing could be further from the truth. The entire point of the Zone System was to capture scenes with more DR and squeeze details from all of it into the more limited DR of the photo papers Adams had to work with. \$\endgroup\$
    – Michael C
    Jun 12, 2022 at 4:35
1
\$\begingroup\$

Most human senses are logarithmic in that "twice as much" is the same difference to us. That applies to force (touch), sound (hearing), and light (vision). That means that perceptually the difference between 2 and 4 is the same as the difference between 16 and 32; whatever those numbers represent.

And in 12 bit exposure the difference between 2 and 4 is the same as the difference between 2048 and 4096 is... the values are linear, but our perception of them is logarithmic. That means there are a lot of values between those last two stops that we cannot perceive. So you can drop a lot of those values and rewrite the exposures using fewer numbers (compression); and you can make the numbers non-linear by applying a gamma curve (jpeg). In doing so, **an 8 bit non-linear jpeg can display about the same exposure range as can be encoded by 12 bit linear raw with no perceptual loss.


** there's usually other changes made to the data in the conversion to jpeg which do result in perceptual changes/losses... and the bigger issue with jpegs is editing in 8 bit.

Edit to add: Even greater dynamic range can be encoded by an 8 bit jpeg; just not with the standard curve (e.g. HDR composite images). And technically, applying (and removing) a gamma curve isn't required for non-linear encoding.

\$\endgroup\$
2
  • \$\begingroup\$ Thanks, Stephen and others. It seems I've been reading too many simplified descriptions of the digital process, as I've heard of many of these terms and thought I understood them, but the replies by you all show I've just skimmed the surface. I suppose that a number of optimisations and playoffs are going on to get the "best possible" jpg, which we can modify to a degree. I've been trying to use software like FastRawViewer, RAW processors in Affinity Photo, RawDigger, RawTherapee etc, and HDR programs like Aurora to understand HDR processing. Seems like I need more basic knowledge to do that. \$\endgroup\$
    – Stuart444
    Jun 12, 2022 at 8:01
  • \$\begingroup\$ @Stuart444, no worries; there's also a lot of bad information available on the web; like the last stop of exposure occupying half of the total available bits is a big deal... it is true in a way, but pretty much irrelevant; it's just a voltage converted into a number, and it must be 2x higer to represent the next stop (in linear raw). \$\endgroup\$ Jun 12, 2022 at 13:57
1
\$\begingroup\$

First things first:

All "bits" do not store information in the same way

Even if the 12 or 14 bit monochrome luminance values in a raw file were 16-bit monochrome luminance values they would not be the same as the 16-bit per color channel values in an uncompressed 16-bit TIFF, PSD, or PNG. Raw image files do not store information in the same way as color raster image files do. Obviously color image files derived from raw files can not contain information that was not derived from the information in the original raw file, but color raster files typically contain much less actual information, even though the way they store that information can make them much larger than the raw file from which they were derived.

Raw files, whether 12-bit, 14-bit, or any other bit-depth include only a single brightness value per photosite (a/k/a sensel for sensor+pixel a/k/a pixel well). All light that makes it into that sensel gets counted as light energy. Some light from all wavelengths of the visible spectrum will make it past each of the three color filters used by color digital cameras. More blue light than red or green light will make it past the blue filter, but some of all three make it through the blue filter. The same is true of the other two colors used in Bayer masks. Some of all of the visible spectrum will make it through each of the three differently colored filters. This imitates the way our retinal cones are sensitive to various wavelengths of light in an overlapping manner. These overlapping responses of our retinal cones are what allows our brains to create the perception of color. There are no colors intrinsic to a particular wavelength of light, there are only the colors our eye-brain system perceives when stimulated by specific wavelengths or specific combinations of wavelengths of light.

It's no different than when we used red filters with B&W film to make the sky look darker and more dramatic. Our red filters did not make the blue sky totally black, as would have been the case if the red filter blocked all blue light. Instead, the red filter made the bright blue sky a darker shade of gray in our B&W photo than it otherwise would have been so that the darker green and yellow forest or field, or the red brick buildings beneath the bright blue sky could be a lighter shade of gray in our B&W photo, relative to the brightness of the blue sky.

The 14-bit values in a raw file are monochromatic luminance values for each photosite on the sensor. These values that describe only the total brightness of all wavelengths of light detected by the sensel are not equivalent to a 14-bit color channel value that would be directly comparable to an 8-bit or 16-bit value for each of three color channels per pixel. When converted to RGB via demosaicing each pixel is assigned an 8-bit or 16-bit value for each of the three color channels. This means that each pixel requires 24-bits (for an "8-bit TIFF, PNG, or JPEG) or 48-bits (for a 16-bit TIFF, PNG, etc.) to express the combined color of that pixel.

RGB color image files with 16-bits per color channel contain three values per pixel. One 16-bit value for red, one 16-bit value for green, and one 16-bit value for blue are included for every pixel. These color values are interpolated via the process of demosaicing the monochrome luminance values of the raw image file and comparing the relative brightness of adjacent photosites filtered by differently colored filters. Creating these R,G, and B values for each pixel in the image also involves setting a black point (defining what is the highest luminance value in the raw file that will still be depicted as pure black with no brightness value?), a white point (defining what is the lowest luminance value in the raw file that will be depicted as full brightness?), and white balance (defining what color multipliers will be used when converting linear monochrome luminance values to logarithmic color values to account for the spectral content of the light illuminating the scene?).

In a typical color camera, the photosites are covered by a Bayer mask, which is a filter array of three differently colored filters. We often call the colors of the respective filters over each photosite "red", "green", and "blue", but they are really closer to "blue-violet", "slightly yellow green", and "yellow-orange". They are not the same colors as our emissive displays that do emit colors very close to what we mean when we say "red", "green", or "blue". For "red", it's not even remotely close. The colors of the Bayer mask emulate, to a degree, the three colors to which the three types of retinal cones in the human vision system are each most sensitive. Using one set of colors for sensing, and another set of colors for displaying is perfectly fine. It is the trichromatic nature of human vision that makes it work.

enter image description here

The three types of cones in the human retina are most sensitive to the the color of the three dots in the above diagram. We started calling or "L" cones "red" cones almost a century before we actually nailed down to what wavelengths of light each of the three types of cones are most sensitive.

Bayer masks, to one degree or another, emulate the response of humans' retinal cones. Different camera makers may use slightly different colors for each of the three filter colors in their Bayer masks, but none of them remotely resemble the three colors that RGB color space is based on and which our RGB emissive displays aim to emit.

enter image description hereenter image description here

All of the cute little RGB checkerboard drawings all over the internet like the one on the left above notwithstanding, the actual colors of Bayer masks are like the magnified view of a sensor partially covered by a Bayer mask on the right. Part of the Bayer mask has been removed.

When the data set in a color raster image file like a lossless TIFF or PNG is then converted to JPEG, that data is compressed by grouping pixels with identical (lossless) or very similar (lossy) RGB values into groups and then only describing the locations of each pixel within each group, rather than repetitively describing each pixel within that group with the same individual RGB values for each individual pixel.

What am I looking at on my screen when I open a "raw" file?

When you view a raw file on the back of your camera or open it using an application or viewer on your computer, what you are seeing on the screen is not "THE raw file." What you are seeing is one of a near countless number of possible interpretations of the raw information that has been processed into a very JPEG like form. You may even be looking at an actual JPEG. When cameras save raw files, they also produce an in-camera JPEG preview image based on the in-camera settings at the time the photo was captured and attach it to the raw file.

When you view a raw image on your camera's LCD screen, you are seeing a downsized version of the JPEG preview included in the raw file along with the actual sensor data and EXIF information that records shooting information about the camera/sensor and the state of the camera and its settings at the time the photo was taken.

When you open a "raw" file on your computer you see one of two different things:

  • The preview jpeg image created by the camera at the time you took the photo. The camera used the settings in effect when you took the picture and appended it to the raw data in the raw file.

  • A conversion of the raw data by the application you used to open the "raw" file. When you open a 12-bit or 14-bit 'raw' file in your photo application on the computer, what you see on the screen is an 8-bit rendering of the demosaiced and gamma converted raw file, not the actual monochromatic Bayer-filtered linear 14-bit file. As you change the settings and sliders the 'raw' data is remapped and rendered again in 8 bits per color channel.

Which you see will depend on the settings you have selected for the application with which you open the raw file. Many raw conversion applications will display the JPEG preview until the application has time to apply the current settings within the application and render its own interpretation of the raw data based on those settings.

Now that we understand what a raw file contains and what it doesn't, how do we get from 14 bits of monochrome brightness levels to 8-bits of 'Red', 8 bits of 'Green', and 8 bits of 'Blue' for each pixel in a JPEG image?

There are several existing questions here at Photography SE that discuss various ways of demosaicing the monochromatic linear brightness values contain in a raw image file to derive color information. But regardless of which method is used, the end result is that we have three separate values for each pixel in the image: one each for Red, Green, and Blue. At this point, though, these RGB values are still linear. That is, pixels that received twice as many photons will have a number twice as high as those that received half as many. (This is a bit simplistic, because most cameras do "cut off" the lowest analog values from the sensor, which can not be distinguished from the "noise" floor, before analog-to-digital encoding but they don't usually cut them off to near the extent that they will be in further processing steps.)

Color multipliers will be applied to each of the three color channels to compensate for the color temperature and white balance of the light illuminating the scene when the camera captured the photograph. A different set of color multipliers needs to be applied for a scene illuminated by a warm tungsten light bulb than if the same scene were illuminated by a much cooler flash strobe. Setting color temperature (the blue ←→ amber axis) and white balance correction (the green ←→ magenta axis) changes the values of the color channel multipliers.

Black point and White point settings determine what is the highest linear value that will be considered "solid black" and what is the lowest linear value that will be considered "solid white". All values below the black point are converted to "0". All values above the white point are converted to the maximum value. For 8-bit, the maximum value is 2^8 - 1, or 255. For 16-bits the max value is 2^16 - 1, or 16,535. Note that the black and white point are the same in the raw values whether using 8-bit, 16-bit, or an even higher internal bit depth for use during processing. The difference between 8-bit and 16-bit at this point is a difference in the size of each step between consecutive values.

Think of it like a staircase: The black point is how many feet above ground level the bottom step is. The white point is how many feet above ground level the top step is. The bit depth is how many steps the staircase has. If we have a staircase that is 256 feet from the bottom to the top, at 8-bits (0-255 are 256 distinct values) each of the 256 steps would be one foot in height. If we have a staircase that is the same 256 feet from bottom to top, at 16-bits (0-16,535 or 16,536 distinct values) we would have 256 steps per foot! These small gradations are important when we do the next step.

The next step is what we call gamma correction. Please note that this step is not the same process that takes place when a video signal is modified by a device's graphics processing unit depending on what type of display device the signal is being sent, even though the term is the same. Though similar in some respects, they are two distinct steps in the processing chain between the raw information captured by a camera's sensor and the image displayed on a screen.

During image processing, gamma correction applies a response curve to the linear values in order to make them more like the logarithmic way the human eye/brain system perceives differences in brightness levels of light. When we change the "contrast" setting on our camera, we are changing the shape of the gamma correction curve applied to the raw information coming off of the sensor. When we change "highlight" or "shadow" levels in post-processing applications, we are changing the shape of the top and bottom, respectively, of the gamma curve being applied. With many raw conversion applications, we can even use a "Curves" tool that allows us to draw differently shaped gamma curves for each of the Red, Green, and Blue channels.

By having a large number of very small steps, we can "stretch" the values along one part of the curve while "compacting" values along other parts of the curve without leaving large jumps from one step to the next.

enter image description here

For further reading here at Photography SE:

What's the point of capturing 14 bit images and editing on 8 bit monitors?
Why can software correct white balance more accurately for RAW files than it can with JPEGs?
RAW to TIFF or PSD 16bit loses color depth
Why are Red, Green, and Blue the primary colors of light? (Hint: They're not)
What does an unprocessed RAW file look like?

\$\endgroup\$
2
  • \$\begingroup\$ Thank you for the detailed answer. Are there any books on this subject which you wrote in this answer? \$\endgroup\$
    – MOON
    Jun 4, 2023 at 13:48
  • 1
    \$\begingroup\$ More a huge number of academic papers, white papers, and online resources from academic and research specialists that I've read over the years than traditionally published books. You might find much of the background information at Photons to Photos helpful. Clarkvision (I'm not related to Roger Clark) is another. \$\endgroup\$
    – Michael C
    Jun 6, 2023 at 2:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.