Reduce filesize of 4000 images

[abduct]

I have 4000 images that total 8.9gb in size. Most are jpg but there may be the a few png scattered throughout.

The 4000 images are broken up into groups each group having it's own directory within the parent directory.

The images are mostly the same size (dimensions) most of which are 2037x3056, the next most being 4074x3056, and finally there are a few at 1200x1800.

The images are black and white except for maybe about 150 which are color.

My objectives are:

1) convert all files that are not jpg to jpg
2) reduce the file size of all images without reducing quality or image dimensions (I heard the -strip option might reduce the file size a lot without impacting quality)

If 2) is not feasible then a reduction of the file dimensions (shrinking to a new resolution) would take priority over reducing image quality. All image must maintain the same resolution ratio.

Which applications of the imagemagick suite is suitable for this batch task and how can I add this recursive behavior so that it applies to all images from a base directory and sub directories?

Thanks.

[fmw42]

You can use mogrify to process a whole directory of images. See http://www.imagemagick.org/Usage/basics/#mogrify

-strip will remove most meta data including profiles. So you may want to be careful about that or reset your color profiles after doing a -strip. But you need to be careful or test if the JPG files are CMYK or sRGB before doing the latter. I would suggest you convert all files to sRGB with profiles. You should test on a few images using convert/magick (depending upon whether you use IM 6 or IM 7) to see the effects of -strip.

You should also check your quality values to be sure you do not already have too low a quality value.

What is your IM version and platform?

[snibgo]

As Fred says, especially the caution on stripping profiles. I suppose you want all results to be encoded as sRGB.

"Mogrify" can process the files (possibly with wildcards) in a directory, processing all files in the same way. Put this in a shell loop to do subdirectories.

I suppose these are photographs? Or, at any rate, like photos? How much do ou want to compress them by?

JPEG is lossy compression, so it changes images. The greater the compression, the greater the change. Changes (by definition) lower the quality, but you might find a small amount of change to be acceptable. Personally, I find a 1% RMSE change to ordinary photographs is usually invisible.

IM has no command to "reduce JPEG quality to give the most compression within a maximum 1% change". This can be done in a script or compiled program.

Reducing the size, in pixels, is easy. But if you want "the least reduction that meets some criteria", that would have to be scripted.

Any script that processed images differently would use "magick" or "convert", not "mogrify".

[abduct]

You can use mogrify to process a whole directory of images. See http://www.imagemagick.org/Usage/basics/#mogrify

-strip will remove most meta data including profiles. So you may want to be careful about that or reset your color profiles after doing a -strip. But you need to be careful or test if the JPG files are CMYK or sRGB before doing the latter. I would suggest you convert all files to sRGB with profiles. You should test on a few images using convert/magick (depending upon whether you use IM 6 or IM 7) to see the effects of -strip.

You should also check your quality values to be sure you do not already have too low a quality value.

What is your IM version and platform?

Thanks for the reply.

My versions are:
(On Linux)
Version: ImageMagick 6.9.7-4 Q16 x86_64 2017-01-29 http://www.imagemagick.org

(On Windows)
ImageMagick-7.0.3-7-portable-Q16-x64

Either platform is viable for the task.

According to identify these are the following formats of the images:

The color title images:
a.jpg JPEG 2037x3056 2037x3056+0+0 8-bit sRGB 1.914MB 0.047u 0:00.052

The black and white images that make up 98% of the images:
b.jpg JPEG 2037x3056 2037x3056+0+0 8-bit Gray 256c 1.201MB 0.063u 0:00.070

The odd closing title images:
d.jpg JPEG 1200x1800 1200x1800+0+0 8-bit sRGB 39.9KB 0.016u 0:00.012

From this I gather that any images of color are using the sRGB profile, and any of the main content is using a Gray profile.

After running convert with -strip I don't see any diference in quality, but the file size actually increased:

convert -strip a.jpg aa.jpg
identify.exe aa.jpg
aa.jpg JPEG 2037x3056 2037x3056+0+0 8-bit sRGB 1.987MB 0.000u 0:00.000

Any other information you might require for aiding me in this topic?

As Fred says, especially the caution on stripping profiles. I suppose you want all results to be encoded as sRGB.

"Mogrify" can process the files (possibly with wildcards) in a directory, processing all files in the same way. Put this in a shell loop to do subdirectories.

I suppose these are photographs? Or, at any rate, like photos? How much do ou want to compress them by?

JPEG is lossy compression, so it changes images. The greater the compression, the greater the change. Changes (by definition) lower the quality, but you might find a small amount of change to be acceptable. Personally, I find a 1% RMSE change to ordinary photographs is usually invisible.

IM has no command to "reduce JPEG quality to give the most compression within a maximum 1% change". This can be done in a script or compiled program.

Reducing the size, in pixels, is easy. But if you want "the least reduction that meets some criteria", that would have to be scripted.

Any script that processed images differently would use "magick" or "convert", not "mogrify".

These are mostly white comic book panels an example of which can be found here: https://i.imgur.com/3zsFosH.jpg (be aware that I don't know if the host compresses or modifies the images uploaded).

I am basically looking for the most compression I can get without any visible notice in image degradation. If this is done via cutting the resolution of the image down a bit, or increasing the jpeg compression then it is fine. I know that image degradation and visible defects are subjective so hopefully a solution will come about which I can set different options of compression and compare to see which I am comfortable with using.

[fmw42]

One other caution, testing for profiles and/or colorspace cannot be done within mogrify. It is a rather simplistic tool that simply processed files -- no conditionals, except for resizing if the image is larger than some dimension. So if you need to do that, then you would have to write a script loop over your directory. Scripting is OS specific, so that is why we asked about your platform.

Note also that reprocessing jpegs even with the same compression level might reduce the quality some, because it has to be decompressed and then recompressed.

[fmw42]

You would have to test what amount of compression (-quality) value to use that achieves your desired visual quality needs. If you are on Unix-like system, then I have a script, downsize, that will resize to achieve a certain file size. See my link below. But this was not designed particularly for jpeg files, though you could test it. But it would need to be run in a bash shell script to loop over a given directory.

[abduct]

You would have to test what amount of compression (-quality) value to use that achieves your desired visual quality needs. If you are on Unix-like system, then I have a script, downsize, that will resize to achieve a certain file size. See my link below. But this was not designed particularly for jpeg files, though you could test it. But it would need to be run in a bash shell script to loop over a given directory.

Thanks I will check it out. Is there any other options other than -quality (and the one used for resizing dimensions) that I should look into that may play a factor in reducing file size?

[fmw42]

You might want to look into non-imagemagick tools that will optimize jpg better than ImageMagick. See http://www.imagemagick.org/Usage/formats/#jpg_non-im

[fmw42]

I am not a jpg expert, but there is also this option. http://www.imagemagick.org/script/comma ... ing-factor

[fmw42]

You might also want to review http://www.imagemagick.org/Usage/formats/#jpg

[abduct]

I'll do some more research and see if anyone else has some more input on this topic for the sample image.

I simply ran

convert -quality 10% input.jpg output.jpg

and on the mostly white line art image as provided above i got 238kb in file size as compared to 1178kb. When comparing the images zoomed in, they were still very similar with little to no noticeable defects. This must be due to the face that the image is mostly white in color and the black line art is faint/thin enough that any pixels compressed away are unnoticeable. I might even be bold enough to say that it almost makes the image better in terms of quality due to rouge pixels that seem to "dirty" up the line art get removed causing a slightly cleaner line.

As for testing on the colored image, using 10% for quality caused a large blocking of pixels and it was noticeable when zoomed into the image as defects.

Now unfortunately it is one thing to segregate the colored images out since they are in the same position in the directory for all image groups, but there are some images that are mostly black or contain gradients as seen in the lower right of the provided image.

So...

Is there a way to use image magick to tell if a image is mostly white or mostly black to apply different quality attributes to images? For example if the image is highly white, apply a huge compression, where if the image is hugely or even moderately black provide less compression.

Or even better yet is there a way to tell imagemagick to only apply quality compression on specific areas of the image? Mean while is there also a way to find dense dark areas in an image to not apply compression to? That would be awesome.

Thanks again for your time.

[fmw42]

convert -quality 10% input.jpg output.jpg

Proper IM syntax is to read a raster input before applying any settings or operators. So

Code: Select all

convert input.jpg -quality 10% output.jpg

For IM 6, this is likely not a problem, since it is more forgiving than IM 7.

Is there a way to use image magick to tell if a image is mostly white or mostly black to apply different quality attributes to images? For example if the image is highly white, apply a huge compression, where if the image is hugely or even moderately black provide less compression.

Yes, but in IM 6 that would mean testing as a separate step and then processing in a conditional. So it would mean scripting rather than mogrify.

Or even better yet is there a way to tell imagemagick to only apply quality compression on specific areas of the image?

Not really.

[abduct]

Yes, but in IM 6 that would mean testing as a separate step and then processing in a conditional. So it would mean scripting rather than mogrify.

Conditionals are fine. I'll likely iterate over the files via bash and can apply the conditionals on a per file basis.

What command line flags should I be looking at to tell if the image is mostly white or mostly black/grey.

[fmw42]

check if it is colorspace=gray or type=grayscale. then check if the mean value of all the pixels in the image are above some desired threshold. See string formats %[colorspace] %[type] and either %[mean] or %[fx:mean] at http://www.imagemagick.org/script/escape.php and http://www.imagemagick.org/script/fx.php

%[mean] will give values between 0 and quantum range (65535 for Q16 IM) and %[fx:mean] will be between 0 and 1

Code: Select all

mean=`convert image -format "%[fx:mean]" info:`

puts the mean into the variable mean. Then you can use that in some conditional.

Similarly, check the colorspace and the type to be sure at least one is colorspace=Gray or the other is type=Grayscale. It may be cap specific.

[abduct]

So I tried another approach that a friend suggested.

Rather than keeping the files as JPG I attempted to convert a small batch of copies to PNG, then run optipng tool to further apply lossless compression on the data.

Code: Select all

mogrify -resize x1200 -filter triangle -format png -posterize 24 *.jpg
optipng -o7 *.png

I chose x1200 due that is the height of my reading tablet. So on top of keeping the resolution fairly high, it should prevent the software from having to blow up the images causing defects.

Although I noticed that the there is a slight bluring of the image of the mogrify images. Although bluring aside I was able to reduce the file size of the images by about 300%.

Is there a specific flag I could research that maybe applies different algorithms to keep the sharpness up?

http://imgur.com/a/5V9jf

Here is an album of a handful of images if you want to test on them.