Corrupt Images With B&W Conversion

[andrewstonewall]

I have found a bug that affects ImageMagic randomly when converting to a B&W image.

The bug can be reproduced with the sample images PIC1.BMP and PIC2.BMP found in the ZIP file at this location:

http://drop.io/ImageMagickBugSampleImages

At this location you will find ImageMagickBugSampleImages.zip for download.

If you use these two commands to convert the images to bitonal TIF files:

convert -format tif -compress group4 pic1.bmp pic1.tif

convert -format tif -compress group4 pic2.bmp pic2.tif

(I have also included thed PIC1.TIF and PIC2.TIF output files in my ZIP file).

You will notice that PIC1.TIF is all white. PIC2.TIF is what is expected. Randomly changing PIC1.BMP stops the error from occurring.

Currently I get this bug in about 1 in 100 files I'm converting.

Adding other parameters like -DEPTH or -COLORS will change which images are affected by the bug (along with the order of the commands on the command-line) but there doesn't seem to be a way to make this bug stop affecting my conversion scripts.

I am using release ImageMagick-6.6.4-Q16 on Win32 but can reproduce this on older versions also.

Can somebody please explain why this error is occurring?

Thanks

[andrewstonewall]

Hi,

Could someone please confirm whether they can reproduce this issue with the commands and image files that I've provided?

I've managed to reproduce this on several Win32XP boxes and a Win2k3 Server all running the latest Win32 build ImageMagick-6.6.4-Q16 and one running a slightly older build.

Thanks

[magick]

Try

convert pic1.bmp -ordered-dither 3x3 -compress group4 pic1.tif

[andrewstonewall]

Thanks for the reply Magick - that command did create a valid file using the alternate dithering technique.

My issues with this work-around are:

1. The files are not a nice to view (this was the sole reason I gave up on LeadTools and switched to ImageMagick as it had a much better normalize and B&W dither)
2. The conversion process has already been approved by the customer and I cannot swap the dithering technique at this point
3. Randomly changing other command-line options also makes this bug go away and this doesn't give me confidence that a root cause has been found
4. These files are also much larger than the ??riemersma?? dithered files I've already had the customer approve

Can you suggest how I get to the bottom of the issue?

Did you manage to reproduce the issue on your computer?

Thanks

[andrewstonewall]

Further to my last post this command also works, using the dither technique I like:

convert pic1.bmp -dither riemersma -compress group4 -colors 4 pic1.tif

This command produces a good image.

This command below produces an empty white file though:

convert pic1.bmp -dither riemersma -compress group4 -colors 256 pic1.tif

[andrewstonewall]

So I guess the point I'm making is that the dithering technique doesn't seem to be proven to be the issue as changing a single command in the rendering pipeline reintroduces the issue with no apparent causality (-colors 4 vs -colors 256).

And even if the issue is down to the dithering routine I'm selecting (the default one which gives great results) you have to wonder what can be done to fix such an easily reproducible bug that occurs randomly with such a common conversion scenario i.e. taking a color image and making it B&W with the default dither routine handling the color resolution down-scaling.

I'm amazed that others performing B&W conversions haven't struck this bug - perhaps this is something they just don't realize is happening.

[andrewstonewall]

I've managed to get a copy of the ImageMagick source code running in a debugger on my machine (using the VisualStaticMT.sln) and wouldn't mind looking into this problem myself as I'm a software developer by profession.

Can anyone recommend a piece of code to start looking at given the symptoms that I've described?

Given that this issue looks to be connected with the dithering routines, I've run a few traces looking at the quantize.c code but can't find any detailed information on how this works so any pointers from any developers would be appreciated.

Thanks

[magick]

Take a look at the color reduction algorithm in magick/quantize.c. Its not a bug per se, instead the color reduction algorithm is reducing the multi-color image down to one color-- white. There are other ways to produce bitonal images such as ordered dithered or thresholding.

[andrewstonewall]

I think that this is definitely a bug as there is other associated buggy behavior whenever this error occurs.

What happens is that the output files created from a monochrome conversion are sometimes abnormally bloated in size whenever you hit this error and a compressed output file is produced and sometimes a zero length file also appears.

Logically an all white image should compress very small but instead, the compressed file is actually larger than it would be if there were no compression at all indicating that something is going seriously wrong under the covers and that the all-white image isn't by design of the color quantizing methods. At other times, depending on what image format is used as the source image the compressed file is zero length and therefore invalid.

Let me demonstrate - if you take the two sample images I uploaded before:

http://drop.io/ImageMagickBugSampleImages -> ImageMagickBugSampleImages.zip -> pic1.bmp
http://drop.io/ImageMagickBugSampleImages -> ImageMagickBugSampleImages.zip -> pic2.bmp

and run these commands:

convert pic1.bmp pic1.tif
convert pic2.bmp pic2.tif

convert pic1.tif -compress Group4 pic1a.tif
convert pic2.tif -compress Group4 pic2a.tif

You will find that the after doing a straight uncompressed BMP to uncompressed TIF convert you will have four files all of about 11mb as expected (pic1.tif, pic1.bmp, pic2.tif & pic2.bmp)

Once you convert the new uncompressed TIFF files to monochrome Group4 TIFF files pic1a.tif becomes zero length while pic2a.tif is valid 400kb file. No TIFF file should be Zero length as this isn't valid even for an all-white image.

This result isn't always found when I run this on several machines so it may be likely you can reproduce this exact problem.

As I stated earlier, at other times with other images all-white images are made massive when they should be 1kb and also, curiously, sometimes when the conversion is going OK and there is a valid B&W image, the compressed file is many times larger than it should be and requires re-compression.

What is more reproducible is the bloating bug - for example take this image I've just posted online:

http://drop.io/ImageMagickBugSampleImages -> pic3.tif

Now try this command:

convert pic3.tif -compress Group4 pic3a.tif

You will end up with a 654KB file - a funny co-incidence as I find many files convert to this strange size (strange given that the file is all white and always 654kb on the images I'm testing with).

identify -verbose pic3a.tif

Shows this is a Group4 file.

identify -format "%k" pic3a.tif

Shows a unique color count of 1.

Re-saving the file with Irfanview as a 2248x1625 Group4 TIFF creates a 440 byte file as expected.

Saving the file as an uncompressed mono BMP yeilds a 450kb file making the 654kb file seem bloated even for an uncompressed format.

I look forward to hearing you comments on these findings, along with any ideas on good primers on how the ImageMagick source code is structured so that I might be able to look at this myself.

Maybe there's something going wrong between the rendering pipeline and the save routine? Just a thought.

Thanks

[andrewstonewall]

Also, slightly changing the same image makes mono conversion suddenly work. This isn't expected behavior as a slight change shouldn't make the quantize routine suddenly elect that the entire image should be white.

For example, if you use the modified PIC1.BMP found here:

http://drop.io/ImageMagickBugSampleImages -> pic1-slightly-different.bmp

and run this command:

convert pic1-slightly-different.bmp -compress Group4 pic1-slightly-different.tif

The slightly different file suddenly converts properly.

[andrewstonewall]

Can anyone on the forum please confirm whether they can reproduce any of the errors that I have highlighted?

Thanks