Error in image output using compare and a fuzz value

Post any defects you find in the released or beta versions of the ImageMagick software here. Include the ImageMagick version, OS, and any command-line required to reproduce the problem. Got a patch for a bug? Post it here.
Locked
Steve2.0
Posts: 10
Joined: 2020-02-21T08:29:11-07:00
Authentication code: 1152

Error in image output using compare and a fuzz value

Post by Steve2.0 »

Creating this bug report based on feedback given in the following user thread:
https://www.imagemagick.org/discourse-s ... =1&t=37585

Compare images using a fuzz value, resultant pixel errors in the output image and the number of reported errors (ae) don't match.

Unexpected behavior seen using version 7.0.9-25 Q16 x64 2020-02-21
Expected behavior seen using version* 7.0.7-11 Q16 x64 2017-11-12
Both are windows builds running on a fully updated Win10

Simple example:

Code: Select all

magick logo: logo.png
magick logo: -attenuate 1% -seed 42 +noise gaussian logo_noise.png
magick compare -verbose -metric ae -fuzz 15% -compose src -highlight-color white -lowlight-color black logo.png logo_noise.png diff_logo.png
Output from 7.0.9-25 Q16 x64 2020-02-21
.\logo.png PNG 640x480 640x480+0+0 8-bit sRGB 256c 27398B 0.016u 0:00.010
.\logo_noise.png PNG 640x480 640x480+0+0 8-bit sRGB 659953B 0.016u 0:00.025
Image: .\logo.png
Channel distortion: AE
red: 144
green: 484
blue: 1256
all: 1256


Check using a histogram:

Code: Select all

magick diff_logo.png -format "%c" histogram:info:
306766: ( 0, 0, 0) #000000 black
434: (255,255,255) #FFFFFF white

I would have expected the 'all' pixel count to match the number of white pixels in the resultant hard difference image.

Trying this with 7.0.7-11 Q16 x64 2017-11-12 results in the 'all' pixel ae result matching the number of white pixels, in this example the ae result is 27085 pixels with exactly the same number of pixels showing in the output image.
The fact the ae results differ between versions is not an issue for me and quite possibly expected as the fuzz algorithm is improved.

Steve

*Yes I know this version is really very old but that's how long it's taken me to update my image analysis tool :D , it's quite possible the expected behavior is seen in much newer versions too.

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Error in image output using compare and a fuzz value

Post by fmw42 »

I get similar results using IM 7.0.9.25 and IM 6.9.10.95 Q16 Mac OSX Sierra

Code: Select all

magick logo: logo.png
magick logo: -seed 100 -attenuate 1% +noise gaussian logo_noise.png
magick compare -verbose -metric ae -fuzz 15% -compose src -highlight-color white -lowlight-color black logo.png logo_noise.png diff_logo.png
logo.png PNG 640x480 640x480+0+0 8-bit sRGB 256c 27398B 0.010u 0:00.007
logo_noise.png PNG 640x480 640x480+0+0 8-bit sRGB 656662B 0.010u 0:00.013
Image: logo.png
Channel distortion: AE
red: 141
green: 448
blue: 1195
all: 1195
logo.png=>diff_logo.png PNG 640x480 640x480+0+0 8-bit sRGB 27398B 0.350u 0:00.105

Code: Select all

magick diff_logo.png -format "%c" histogram:info:
306810: (0,0,0) #000000 black
390: (255,255,255) #FFFFFF white

User avatar
magick
Site Admin
Posts: 11254
Joined: 2003-05-31T11:32:55-07:00

Re: Error in image output using compare and a fuzz value

Post by magick »

Convert the image colorspace to gray to get expected results:

Code: Select all

$ magick compare -verbose -metric ae -fuzz 15% -compose src -highlight-color white \
  -lowlight-color black logo.png logo_noise.png -colorspace gray diff_logo.png
logo.png PNG 640x480 640x480+0+0 8-bit sRGB 256c 27398B 0.010u 0:00.005
logo_noise.png PNG 640x480 640x480+0+0 8-bit sRGB 655820B 0.010u 0:00.010
Image: logo.png
  Channel distortion: AE
    gray: 2094
    all: 2094
logo.png=>diff_logo.png PNG 640x480 640x480+0+0 8-bit sRGB 27398B 0.150u 0:00.034
$ magick diff_logo.png -format "%c" histogram:info:
    305106: (0,0,0) #000000 black
      2094: (255,255,255) #FFFFFF white

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Error in image output using compare and a fuzz value

Post by fmw42 »

Please explain what is happening when the images are not grayscale. I do not see why the white from the histogram should not match the "all" value from the compare ae metric.

User avatar
magick
Site Admin
Posts: 11254
Joined: 2003-05-31T11:32:55-07:00

Re: Error in image output using compare and a fuzz value

Post by magick »

Distortion is computed on a channel by channel basis for each pixel. The composite (all) distortion is accounting for differences across all channels. For grayscale, a single channel image, the composite distortion is the same value as the single gray channel since there is only one channel. Its all about how you want to define the composite pixel (all). If we combine all the channels of a pixel by computing its grayscale intensity, we get the same results as the single gray channel for all, that is 2094. How should the composite pixel be computed for AE?

Steve2.0
Posts: 10
Joined: 2020-02-21T08:29:11-07:00
Authentication code: 1152

Re: Error in image output using compare and a fuzz value

Post by Steve2.0 »

First thanks to both for looking into this.
I think the grayscale solution might work for me - will test this method against the method using the older version of the tool.

However I'll admit I am not understanding the explanation - it would be great if the relationship between the ae result and the number of pixel in the image can be explained. Why are they different and if this is correct behavior what does each value mean.

Edit - forgot to add - why is the pixel error value 2094 using the gray colorspace correct?

Thanks again
Steve

User avatar
magick
Site Admin
Posts: 11254
Joined: 2003-05-31T11:32:55-07:00

Re: Error in image output using compare and a fuzz value

Post by magick »

Pixels are made up of channels (e.g. red, green, blue). Each image processing algorithm may treat each channel independently or may treat it holistically where all channels are combined. A typical method for combining channels is to convert the red, green, and blue channels to grayscale so instead of dealing with three independent values, there is a single value, the pixel intensity. For comparison, the individual values are reported against the specified metric (e.g. absolute error) as well as the composite of all the channels. How the composite pixel is reported is a judgement call. For the AE metric, the composite pixel, denoting the distortion, is the maximum distortion of any one channel. We could instead report the difference across all channels by instead looking for differences in the pixel intensity between the source and reconstructed images. Alternatively, this of course can already be done by converting the image to grayscale as we previously suggested. Since there is a solution to the problem you presented, it likely does not make sense to change how the composite pixel is computed-- it might break existing scripts that rely on the current behavior.

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Error in image output using compare and a fuzz value

Post by fmw42 »

But then for the color case, how is it decided which pixels to make white. It seems to disagree with the ae count method. The issue, I believe is not how it is computed for ae count in the color case, but that it does not seem consistent with what is put in the output image.

Steve2.0
Posts: 10
Joined: 2020-02-21T08:29:11-07:00
Authentication code: 1152

Re: Error in image output using compare and a fuzz value

Post by Steve2.0 »

I have a reasonable understanding of how the fuzz algorithm is working within the RGB colorspace, I also accept that this can and probably should (improve) from version to version.
But I am still not understanding why the ae result gives one value for errored pixels and the output image is showing another different value. If I could understand the relationship between the ae result and the pixels in the output image I may be able to make a simple modification to my analysis tool.

With the older version (7.0.7-11) I'd assumed fuzz calculate, per channel, the number of pixels that deviate beyond a threshold. The 'all' results was the total number of pixels that deviate in any of the color channels.
The output image from the compare contained a visual representation of the ae 'all' result.

This understanding seemed to hold true for all of my testing (many thousands of images).

Now with 7.0.9-25 that understanding is not holding true and I don't know what the connection is between the ae 'all' result and the output image.

As you suggest I will try the conversion to the grayscale colorspace but I am not yet 100% confident the conversion won't loose important detail I need. I'm working with both 8 and 16 bit depth images.

Briefly my tool is determining a subset of the worst errors in a image and using this to supply targeted cropped sections to another algorithm. In some scenarios I need to be able to detect very small deltas where I then categorize structural issues that need investigation from expected "noise" like differences.

Steve

User avatar
magick
Site Admin
Posts: 11254
Joined: 2003-05-31T11:32:55-07:00

Re: Error in image output using compare and a fuzz value

Post by magick »

Thanks for the problem report. We can reproduce it and will have a patch to fix it in GIT master branch @ https://github.com/ImageMagick/ImageMagick later today. The patch will be available in the beta releases of ImageMagick @ https://www.imagemagick.org/download/beta/ by sometime tomorrow.

Locked