-2
\$\begingroup\$

I'm planning to acquire between 50k and 200k image per day with a 50MPixels (or 68MPixels or 130MPixels) sensor; I'll be acquiring the raw data (10 or 12 or 14 bits) from the sensor through SLVS-EC and create a raw file of my own design. The raw bitrate from the sensor may go up to 75.2 Gbps.

I may have to store 50k-250k images per day (eg., 17.5TB if 250k images are 70MB-50MPixels each). I need to keep high quality images (in particular, colors must remain accurate and textures fully detailed, hence the lossless or only light loss and nothing below 10 bits per channel), and a flexibility in edition (hence the raw).

Also images will share a lot, since I may have 2-24Hz framerate at capture; also a first processing will drop (delete) between 10% and 50% of images, so a keyframe based compression may not be suitable.

Since I need to keep the storage cost as low as possible without doing too hard compression (maybe go below 30-50MB per raw image). I'm planning to allow compression within this raw file, this compression can be lossless or lightly lossy. I'm thinking about wavelet and auto learnt dictionaries (patchs and sparse coding) for the compression, but this is not a requirement.

I will not release any sdk or raw image, so there are no need or requirement on the standard and adoption side. I'll very likely use an FPGA for signal processing (up to 75.2Gbps from the sensor), since I need very high IO and fast signal processing, and the whole package will be embedded, and as compact as possible and reasonnably light (say less than 1-2kg).

About the images, it will be natural environment with natural day light; it may include shadows and sky with sun, and hence high dynamic, but also rich (high frequency) textures which must be preserved. So likely I won't add further denoising, but I want to keep the fexibility with color processing: in particular the ability to change the signal amplification and the white/black balance.

Do you have thoughts and pieces of advice about the compression strategy for this raw format ? In particular do you think video compression algorithms (eg., HEVC) could be adapted to raw bayered data ?

\$\endgroup\$
17
  • 2
    \$\begingroup\$ I’m voting to close this question because it is not exactly about photography but about compression algorithms. \$\endgroup\$ Feb 8, 2022 at 13:40
  • 2
    \$\begingroup\$ Yes, it just feels like the wrong "forum" to me. What topics can I ask about here? says that "Photography Stack Exchange is for professional, enthusiast and amateur photographers", whereas this question is very much more towards the "engineering" side of image capture / image processing. \$\endgroup\$
    – osullic
    Feb 8, 2022 at 14:53
  • 1
    \$\begingroup\$ It feels to me like it would be a better fit for Signal Processing, judging by their on topic list - dsp.stackexchange.com/help/on-topic \$\endgroup\$
    – Tetsujin
    Feb 8, 2022 at 16:12
  • 2
    \$\begingroup\$ "won't use existing proprietary formats" – That's the problem DNG is supposed to solve. It has compression support, and you can use existing libraries. \$\endgroup\$
    – xiota
    Feb 9, 2022 at 7:17
  • 2
    \$\begingroup\$ I’m voting to close this question because it isn't about producing still photographs as described in this community's guidelines. Per the OP it is about "... thoughts and pieces of advice about compression for raw data and that's it." \$\endgroup\$
    – Michael C
    Feb 11, 2022 at 11:11

2 Answers 2

0
\$\begingroup\$

Don't.

Entire companies, industries, standards, and fortunes are bought and sold upon what you want to do.

There's a reason H.266 is expensive in processing power- squeezing that little bit more out is computationally expensive for little gain.

Storage is cheap (as was said). Buy 2x what you need, and keep a copy.

\$\endgroup\$
1
  • \$\begingroup\$ With 18TB per day of images, storage is critical, so is compression. \$\endgroup\$
    – Soleil
    Feb 8, 2022 at 17:37
0
\$\begingroup\$

A raw format is already compressed, so don't expect too much gain with lossless compression. For instance the Canon format derives from TIFF and likely uses LZW compression; the raw files from my camera are about half the size you would expect (Pixel count × 3 × bit depth).

If you use lossy compression, you would have to check the effects on demosaicing algorithms.

Storage is cheap.


Edit:

In your case, the problem won't be storage, but I/O bandwidth... 18TB/day is around 200MB/s. Given the size I assume this will not be on local disks... 200MB/s sustained requires a dedicated I/O architecture, likely tape libraries somewhere (plural, because backup). Source: I worked on the digital video infrastructure for a large TV channel (4TB/day of video intake).

Of course you can compress by a factor of two, but you'll have to find an algorithm that doesn't burn too much CPU (and you still have to manage a 200MB/s intake bandwidth, and enough CPUs...).

You should also wonder about the output bandwidth (because all this data will be used at some point). In the video infrastructure described above, it was estimated that each input (write) was read 3.75 times in its lifespan. This is another I/O bandwidth to take in account.

Finally, what kind of data center you have in the end to process/use all these images?

\$\endgroup\$
10
  • \$\begingroup\$ I'm creating my own raw format, so no: I'll have to compress it myself. With 18TB per day of images, storage is critical, so is compression. \$\endgroup\$
    – Soleil
    Feb 8, 2022 at 17:36
  • 1
    \$\begingroup\$ See augmented answer \$\endgroup\$
    – xenoid
    Feb 8, 2022 at 18:02
  • \$\begingroup\$ Well, no, bandwidth is not the question, I'll have an FPGA for the sensor readout processing and an SSD sata RAID array to sustain >1GB/s for writing (OP updated) since nvme is still expensive. \$\endgroup\$
    – Soleil
    Feb 8, 2022 at 18:09
  • 1
    \$\begingroup\$ From personal experience: don't believe figures, do real benchmarks... \$\endgroup\$
    – xenoid
    Feb 8, 2022 at 18:18
  • 1
    \$\begingroup\$ "the raw files from my camera are about half the size you would expect (Pixel count × 3 × bit depth)." Ummm... Canon raw files (and pretty much everyone else's when Bayer masked sensors are used) have a single monochrome luminance value per photosite (a/k/a pixel well - only digitally stored images have pixels). No need to multiply bit-depth by three, since there is only one 14-bit value per photosite. \$\endgroup\$
    – Michael C
    Feb 11, 2022 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.