jpeg corruption

[miha]

I sometimes get corrupted jpegs when converting from a lossless format and the image is large and has lots of fine detail. With quality 100% the result is corrupted. With 99% or 98% it usually is correct although some cases needed going even lower. What could be the problem for this? Internal buffer overflow? How can it be avoided while staying at 100% and not reducing detail in the image before the conversion?

[Bonzo]

Corrupted in what way?
What code are you using?
Can you try it on another server or localhost?
What is your lossless format?
What is you output format?
What Imagemagick version are you using?
What platform and code language are you using?

[miha]

- Image is black or one colour or totally desaturated after some initial block of correct lines, one or multiple corrupt areas possible
- for example gm convert -size 32768x24576 -quality 100% -depth 8 rgb:$1 $1_32K.jpg (can happen with smaller sizes too)
- yes
- raw bytes
- jpg
- Version: ImageMagick 6.8.7-5 Q16 x86_64 2013-12-12
- Centos 6.8, bash script

[Bonzo]

I see gm so assume you are using Graphicsmagick; what happens if you use Imagemagick directly?

I do not know about gm but with imagemagick the input image usually comes right after convert and the quality goes just before the save. For jpg the quality setting would be 100 not 100%

[glennrp]

With either IM or GM the input image (rgb:$1) would have to come after the -size option. With such large images you might need to use -limit to avoid running out of resources.

[miha]

It happens with GM or IM. And the file comes after the size option.
Current limits
File Area Memory Map Disk Thread Throttle Time
--------------------------------------------------------------------------------
768 270.59GB 126GiB 252GiB unlimited 48 0 unlimited

[snibgo]

32768x24576

This is nearly a billion pixels. With IM v6, this would take 8 GB. The jpeg compression code will also need memory, and I don't know if it fails gracefully if it can't allocate the memory. I suspect it doesn't, and that's the problem you are hitting.

I suggest you set the various "-limit" options so the pixels are written to disk instead of memory, freeing memory for the jpeg code.

An alternative might be to do this in two stages, with mpc as the intermediate:

Code: Select all

convert -size 32768x24576 -depth 8 rgb:$1 temp.mpc
convert temp.mpc -quality 100% $1_32K.jpg 

(My comments apply only to ImageMagick. I know nothing about GraphicsMagick.)

[miha]

I have 128 GB RAM. Not enough?

[snibgo]

I wouldn't know. Your operating system probably has tools that will tell you how much is used.

[miha]

I just had another case. There was enough RAM for sure as reported by htop. 100% quality and 99% failed. 98% worked. Must be an internal bug.

[snibgo]

Does it fail consistently for a given input? If so, I suggest you put it online somewhere for the developers to examine. And give the exact command you used.