1
\$\begingroup\$

I'm trying to reduce the total size of my backups without sacrificing quality, and more importantly, without discarding photos1.

There are plenty of image formats that offer great compression, or settings that avoid visible quality loss, but I couldn't find a compression method that exploits similarities between images. A lot of my pictures are from sequences with slightly different angles, lighting, poses, and camera settings, and it seems wasteful that they are compressed separately.

This would obviously make each image file depend on the previous one, or have a single large file for a sequence/entire archive. But as long as the process is reversible, I don't care if they are individually readable or not. Processing time of many hours is also ok, but anything over a day or two is too much.

I have come up with, but not yet tried, two hacky solutions:

  • Undo the JPEG compression (save as bitmap, or undo just the last encoding step) and compress everything into a single 7zip file. Would probably not compress much, since it's a generic algorithm.
  • Use all pictures as individual frames of a video file, and rely on video compression to identify redundancies over "time". Would have to fiddle with settings to avoid quality loss, split into multiple videos, skip "single" images, and try to minimize encoding time.

Are there any established solutions for reducing the size of similar images? Or any reason why the methods above would/would not work?


1 There's already software that can merge a sequence of similar images into a single (arguably) better image, and these tools will only get better. So for now, I'd rather save all the seemingly duplicated family portraits.

\$\endgroup\$
14
  • 1
    \$\begingroup\$ I’m voting to close this question because software recommendations are out of topic here. Better ask here: softwarerecs.stackexchange.com \$\endgroup\$ Sep 29, 2023 at 12:07
  • \$\begingroup\$ @RomeoNinov As far as I could find there's no software to recommend. Would it help if I focused on the method/process of the solution? In this regard my question seems to be on par with other questions for these tags. \$\endgroup\$
    – BoppreH
    Sep 29, 2023 at 13:09
  • 4
    \$\begingroup\$ I immediately thought that what you are trying to do here sounds to me like video compression really. I've never heard of anyone doing this with photos - probably because nobody does it. My opinion - bite the bullet and use "normal" archive methods. Storage is, generally, not expensive. Maybe analyse the way you are shooting - maybe you don't need to produce all this content in the first place? I would avoid the option of converting JPEG to BMP because you'll lose the metadata in the files probably. \$\endgroup\$
    – osullic
    Sep 29, 2023 at 13:15
  • 1
    \$\begingroup\$ Are there video codecs whose aim is to losslessly compress video by taking advantage of differences between frames? I'm not sure about that, but I would say that video compression has different aims to still image compression because of the way humans perceive "moving" images. \$\endgroup\$
    – osullic
    Sep 29, 2023 at 13:17
  • 2
    \$\begingroup\$ As Romeo Ninov noted, either this is a software recommendation question; or it's asking about theoretical compression techniques. If it's the latter, it's speculative and discussion-oriented. In both cases, I believe this is off-topic for Photo-SE. \$\endgroup\$
    – scottbb
    Sep 29, 2023 at 20:53

2 Answers 2

2
\$\begingroup\$

A few suggestions:

  1. Max out the dictionary size option on the compression tool you use. This allows more internal re-use of common sequences found in the data. It can waste space on a few small text documents, but with many big photos, the dictionary size is relatively small compared to the data contained, and will pay for itself quickly.

  2. Compress each scene/batch in a different archive. This relates to #1; allowing the parts of the 16x16 block data that appear frequently to be replaced by a dictionary entry. If all the photos are about the same, it can really squeeze the redundancy out much more effectively than if you have a group of say vacation pics with food, people, sunsets, landscapes, and lots of other random subjects.

  3. Like 2, compress jpeg and raw files separately. This increases redundancy.

  4. Try several different compression engines. Don't assume zip or rar or z7 will work the best on everything. If you use command-line tools, it's pretty simple to just use all of them on each batch, all you have to do is wait and delete the largest archives of the grouping.

  5. Have reasonable expectations. I doubt you'll do better than to half the footprint, and even that could be a tall order depending on the camera, image, and algo. Many raw files these days are already compressed in-camera, so there's less overhead to wring out.

  6. Experimental. Try using a batch tool to combine many images into a large image file, then try different compression methods designed for images on that file. As a bonus, this should make non-image compression more efficient as well, but it might be a lot of manual work to save just a little bit; something to consider if you're long on time and low on cash/space.

\$\endgroup\$
0
\$\begingroup\$

In general, the general similarity of the images is not important for compression. If you generate two images in GIMP:

  • random noise
  • random noise overlaid with some illustration (in additive mode for example)

and save them as JPEG they will occupy almost same space on disk. Miniature details are many times more important for file size than illustration-style image (which could represent the similarity between the photos), that's why adding it on top of noise won't change anything.

Photos are not illustrations, it's the miniature details and noise which take up most of the disk space.


However, there are compression algorithms which are a lot better than JPEG, and you might explore those. Additionally, Adobe DNG Converter is able to compress raw images, and they might take even less space than JPG because JPG is a debayerized file.

\$\endgroup\$
6
  • \$\begingroup\$ I understand why illustrations take less space, but I don't see how this relates to multi-photo compression. \$\endgroup\$
    – BoppreH
    Sep 29, 2023 at 14:37
  • \$\begingroup\$ Compression depends on entropy. The more entropic the image, the larger the file size. Just because 6 photos contain the same bird in a bush does not mean the entropy can be averaged in any way, other than by using video codecs, which expect each subsequent image to have slight differences & a lot of 'sameness'. \$\endgroup\$
    – Tetsujin
    Sep 29, 2023 at 14:46
  • \$\begingroup\$ @BoppreH I suggested a simple experiment which exactly answers your question. Miniature details are many times more important for file size than details of illustration-style image, that's why adding it on top of noise won't change anything. \$\endgroup\$ Sep 29, 2023 at 16:01
  • 2
    \$\begingroup\$ Video codecs rely on the image barely changing between frames - they draw the difference & discard the rest [to over-simplify]. If there's a lot of change, the entire frame needs a fresh start. Google key-frame. \$\endgroup\$
    – Tetsujin
    Sep 29, 2023 at 16:16
  • 1
    \$\begingroup\$ Using Adobe DNG convertor to convert raw files to DNG also discards almost all of the maker notes section of the EXIF Info. If all you ever use to process images are Adobe products, this doesn't matter since Adobe products ignore, for the most part, maker notes information. But if you use other image processing applications that do use info from the maker notes section, then you'll lose the ability to leverage that (sometimes valuable) info when using those tools. \$\endgroup\$
    – Michael C
    Oct 1, 2023 at 7:25

Not the answer you're looking for? Browse other questions tagged or ask your own question.