Occational random line through images

[XDAMDaniel]

We have an issue where a line randomly appears through a jpeg that is scaled down by convert. This problem is some what rare as we have to run 500 to 1000 images through our processing procedure to replicate the issue and each image is scaled down to five different sizes. It doesn't seem to matter what image is used the problem can be replicated using the same image 500 times or using 500 different images.

We believe this may be related to or a duplicate of the issue mentioned in: viewtopic.php?t=16496. Another member of our team requested assistance with the same issue almost a year ago: viewtopic.php?t=27641, but was unable to find a solution. At the time we resorted to simply reprocessing images with the symptom but that is no longer a viable solution.

Our current platform is Centos 7 running in the Amazon EC2 environment. We have installed numerous different versions of ImageMagick and have determined that version 6.7.4-4 is the last version where we do not observe the problem. Note that 6.7.4-4 will not build on Centos 7 due to incompatibility with the version of zlib. So, part of the experiment was performed on a Centos 6.5 server specifically dedicated to troubleshooting this problem. We have verified though that the line problem is consistent between later versions of ImageMagick regardless of whether the OS is Centos 6.5 or 7.

Interestingly we also noted a decrease in performance in all versions after 6.7.3-3. Our processing sequence consistently took 3 to 5 seconds in 6.7.3-3 and 17 to 20 seconds in 6.7.3-4 and later. This may or may not be related but seems like it is worth a mention.

Currently we have a Centos 7 server setup with ImageMagick 6.9.3-8 and we can reproduce the issue by running the following command up to 1000 times:

Code: Select all

 convert -limit map 512mb -limit memory 512mb -limit thread 1 -size '2000x2000' -quality 85 -flatten -scale '2000x2000' '1000.jpg'  $filePart1$i$fileExt;

We are prepared to provide any additional information that maybe helpful in triangulating and solving this issue up to and including providing a test bed server where the issue can reliably be duplicated.

[fmw42]

There is no input image for -scale and -flatten. You need to read or create your input images before processing with -scale and -flatten. -size 2000x2000 does not create an image unless it is followed by xc:somecolor or just xc:. Also -flatten should come after -scale and before the output image. I do not see what you are flattening, if you only have one image 1000.jpg?

Please clarify your command or provide a full command line.

[XDAMDaniel]

I apologize for the confusion about the command. I copied it out of my bash script and the path/name of the destination file was stored in variables. I have modified the command as suggested but 11 images through the batch I'm still seeing an image with a line through it. Here is the modified command:

Code: Select all

convert -limit map 512mb -limit memory 512mb -limit thread 1  -size '2000x2000' -quality 85 -scale '2000x2000' '1000.jpg' -flatten  testImage11.jpg;

As a note: "flatten" is present because we use the same command with other image types like TIFF and PSD. I believe we are under the impression that it is ignored in image types where it is not relevant.

[snibgo]

If you include "-synchronize", does that help? See http://www.imagemagick.org/script/comma ... ynchronize

[fmw42]

I still do not see where -size is creating an image. Perhaps I am mistaken, but I have always assumed that -size only creates new images if you follow it with xc:somecolor. Is it creating a fully transparent image of that size for you? Otherwise, I do not think it does anything. Furthermore, you need to put -scale and -flatten after you read your input image 1000.jpg for proper IM 6 syntax. IM 6 may be forgiving, but best to follow the proper syntax.

Please explain what you are trying to do with -size 2000x2000!

[XDAMDaniel]

If you include "-synchronize", does that help? See http://www.imagemagick.org/script/comma ... ynchronize

I tried -synchronize but I'm still seeing lines. Ninety images in on the first test batch and four images in on the second batch.

[XDAMDaniel]

I still do not see where -size is creating an image. Perhaps I am mistaken, but I have always assumed that -size only creates new images if you follow it with xc:somecolor. Is it creating a fully transparent image of that size for you? Otherwise, I do not think it does anything. Furthermore, you need to put -scale and -flatten after you read your input image 1000.jpg for proper IM 6 syntax. IM 6 may be forgiving, but best to follow the proper syntax.

Please explain what you are trying to do with -size 2000x2000!

Well I think I mentioned before that we process a number of different file types with this command. I'm pretty sure the reason we have the -size is in there for file types other than jpg. We are not creating a new image with this command, 1000.jpg is the source image and we are scaling the destination image test1.jpg to 2000x2000. I went ahead and ran a test without the -size and the first image processed with this command has a line in it.

Code: Select all

convert -limit map 512mb -limit memory 512mb -limit thread 1 -quality 85 -scale '2000x2000' '1000.jpg' -flatten test1.jpg;

[snibgo]

Does it happen with all inputs or just this one? Please post 1000.jpg.

[fmw42]

convert -limit map 512mb -limit memory 512mb -limit thread 1 -quality 85 -scale '2000x2000' '1000.jpg' -flatten test1.jpg;

Does it happen with this command in proper IM 6 syntax?

Code: Select all

convert -limit map 512mb -limit memory 512mb -limit thread 1 '1000.jpg' -scale '2000x2000' -flatten -quality 85 test1.jpg;

And as snibgo requested, can you post your 1000.jpg to some place such as dropbox.com and put the URL here so we can test ourselves.

[snibgo]

With IM v6.9.2-5 on Windows 8.1, I've tested with 1000 iterations. I get the same correct result every time. My test script, BAT syntax, was:

Code: Select all

%IM%convert -size 1000x1000 xc: 1000.jpg

%IM%convert ^
  -limit map 512mb -limit memory 512mb -limit thread 1  -size 2000x2000 ^
  -quality 85 -scale 2000x2000 ^
  1000.jpg -flatten  ^
  testImage11.jpg

set CNT=0
:loop
%IM%convert ^
  -limit map 512mb -limit memory 512mb -limit thread 1  -size 2000x2000 ^
  -quality 85 -scale 2000x2000 ^
  1000.jpg -flatten  ^
  testImage11B.jpg

for /F "usebackq" %%L in (`%IM%convert ^
  testImage11.jpg testImage11B.jpg ^
  -format "DIFF=%%[distortion]" -compare ^
  info:`) do set %%L

set /A CNT+=1

if %CNT% LSS 1000 if %DIFF%==0 goto loop

echo DIFF=%DIFF%

[XDAMDaniel]

Please post 1000.jpg.

As requested here is a link to the 1000.jpg: https://www.dropbox.com/s/rpg9uja8s56fi7i/1000.jpg?dl=0

I'll experiment with some of the suggested command reformats and see what the results yield.

[XDAMDaniel]

Ok I ran another batch of images using the suggested ImageMagick 6 syntax and image 121 has a line in it. Here is a link to the problem image: https://www.dropbox.com/s/msvbixa5c0o7j ... 1.jpg?dl=0

Here is the command used to generate the problem image:

Code: Select all

convert -limit map 512mb -limit memory 512mb -limit thread 1 '1000.jpg' -scale '2000x2000' -flatten -quality 85 testImage121.jpg;

[fmw42]

IM 6.9.3.8 Q16 Mac OSX

JPG* JPEG rw- Joint Photographic Experts Group JFIF format (90)



I ran 25 iterations of

Code: Select all

cd
cd desktop/test
for ((i=0; i<25; i++)); do
convert -limit map 512mb -limit memory 512mb -limit thread 1 '1000.jpg' -scale '2000x2000' -flatten -quality 85 test_$i.jpg
done
cd

and no lines?

Does it happen if you just do

Code: Select all

convert -limit map 512mb -limit memory 512mb -limit thread 1 '1000.jpg' -quality 85 test.jpg

Does it happen every time your run your command? Does it happen with other image formats or other JPG images?

What version of libjpg? Use

Code: Select all

convert -list format

to get that. Is it the same on all your systems?

There were some versions of IM that had striping issue as I recall. But I do not recall which. You might search the changelog at http://www.imagemagick.org/script/changelog.php to see if anything was listed.

[XDAMDaniel]

I ran 25 iterations of

Code: Select all

cd
cd desktop/test
for ((i=0; i<25; i++)); do
convert -limit map 512mb -limit memory 512mb -limit thread 1 '1000.jpg' -scale '2000x2000' -flatten -quality 85 test_$i.jpg
done
cd

and no lines?

Ok thanks for trying that. Unfortunately 25 iterations is probably not enough. It really takes 200 to 500 iterations to be likely to encounter the issue and 1,000 or more to have any confidence that it's not there. For instance in my last batch, the first image with a line in it was 112 iterations through the batch, the next is 177 and the next was 286. There have been times I've seen as may as 400 images go through before seeing a line appear in one of them. So the frequency of corrupted images can be very unpredictable.

Does it happen if you just do

Code: Select all

convert -limit map 512mb -limit memory 512mb -limit thread 1 '1000.jpg' -quality 85 test.jpg

I tried 1,000 images with that command and I didn't see any lines. So it seems we only see the problem when -scale is present. As a note we scale images to several different sizes and have seen the line present in any of the various sizes (but not all of the sizes).

Does it happen every time your run your command? Does it happen with other image formats or other JPG images?

No it doesn't happen every time we run the command. It happens at very random and unpredictable intervals. And, yes, we do see the lines with other source image formats. To verify I ran the command 1,000 time on a tif and I do have lines in some of the resulting files. The destination image is always a jpeg, we never convert to any other format. Hopefully I can help explain the situation a little better by noting that we process literally tens of thousands of files with this command. These are all of different source file types, raw types, tiff, jpeg, pdf, and even frames from videos. If the file is not a type supported by convert we first convert it to a tiff or jpeg using ffmpeg, ghost script, dcraw, etc. The line issue can appear in any of these files and it's appearance seems to be totally random. We have seen this issue frequently enough in enough various files that it seems likely that if we processed any image 1,000 times or more we could produce a destination image with a line in it. If we see an image pop up in one of the systems to has a line in it we can simply run the original source image through processing again and the line problem is gone. Alternatively we have scenarios where an image successfully proccesses without a line and then is processed again for another part of the workflow and the resulting image has a line in it. Does that help to clarify some of the questions about different files and types?

What version of libjpg?

The output from convert -list format is: JPG* rw- Joint Photographic Experts Group JFIF format (62)

[fmw42]

62 is rather old. Perhaps you should try upgrading and see if that fixes the issue? Nevertheless, if it does not happen without -scale, then I do not see a connection. Sorry if you have answered this, but IM 6.7.x is very old. Perhaps and upgrade to a current version might help along with newer versions of your delegates.