Errors on batch processing

Post any defects you find in the released or beta versions of the ImageMagick software here. Include the ImageMagick version, OS, and any command-line required to reproduce the problem. Got a patch for a bug? Post it here.
Post Reply
mgo
Posts: 11
Joined: 2019-03-17T02:55:46-07:00
Authentication code: 1152

Errors on batch processing

Post by mgo » 2019-03-17T03:14:31-07:00

I am getting once every few days (let's say every 30 batch runs) a lot of errors from ImageMagick.
The errors cannot be reproduced right after, so I suspect batch processing nature of the problem.
The processed files are correct, and the errors cannot be reproduced while processing a single file.

I will post the errors, perhaps you will have an idea what happens.
Batch run 1:

Code: Select all

composite: improper image header `/home/events/2019/03/15/14652/872287/dsc-1-13_baae18_940x627_border.png' @ error/png.c/ReadPNGImage/4291.
composite:  `/home/events/2019/03/15/14652/872287/dsc-1-13_baae18_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872288/dsc-1-14_2835e3_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872288/dsc-1-14_2835e3_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872292/dsc-1-18_7ca1e0_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872292/dsc-1-18_7ca1e0_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
convert: Premature end of JPEG file `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
convert: Corrupt JPEG data: premature end of data segment `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
convert: Premature end of JPEG file `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
convert: Corrupt JPEG data: premature end of data segment `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
convert: Premature end of JPEG file `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
convert: Corrupt JPEG data: premature end of data segment `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
convert: Premature end of JPEG file `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
convert: Corrupt JPEG data: premature end of data segment `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872294/dsc-1-20_4266c2_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872294/dsc-1-20_4266c2_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
Not a JPEG file: starts with 0x00 0x00
convert: IDAT: CRC error `/home/events/2019/03/15/14652/872295/dsc-1-21_317185_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
convert: no images defined `/home/events/2019/03/15/14652/872295/dsc-1-21_317185_940x627_border.jpg' @ error/convert.c/ConvertImageCommand/3300.
Aborted
Not a JPEG file: starts with 0x00 0x00
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872297/dsc-1-23_1e454f_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872297/dsc-1-23_1e454f_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872299/dsc-1-25_f28d7c_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872299/dsc-1-25_f28d7c_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872300/dsc-1-26_a3fe1f_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872300/dsc-1-26_a3fe1f_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
The commands are nothing special:
composite with -gravity SouthEast creating the _border.png file (why "improper image header" since the file is created by IM?)
and then convert, which scales/crops.

snibgo
Posts: 12147
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Errors on batch processing

Post by snibgo » 2019-03-17T03:29:12-07:00

The error messages are consistent with running an IM command on an input image file before it has been fully written. Perhaps your batch process detects a file and starts processing it before a previous process has finished writing it.
snibgo's IM pages: im.snibgo.com

mgo
Posts: 11
Joined: 2019-03-17T02:55:46-07:00
Authentication code: 1152

Re: Errors on batch processing

Post by mgo » 2019-03-18T08:33:07-07:00

Dear Snibgo. I will investigate this possibility. It is likely. Thank you!

mgo
Posts: 11
Joined: 2019-03-17T02:55:46-07:00
Authentication code: 1152

Re: Errors on batch processing

Post by mgo » 2019-03-21T09:12:30-07:00

Hmm. Does convert/composite end before file bas heen fully written?

snibgo
Posts: 12147
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Errors on batch processing

Post by snibgo » 2019-03-21T09:53:58-07:00

It depends on your computer and operating system.

On my computer (HP laptop, running Windows 8.1) when I run a series of commands in one window, each command won't start until the previous command has fully written and closed any files.

But I might run a series of commands in one window, and another series of commands in another window. The first window might have a command that creates a file, then takes time writing it (eg a large PNG file). The second window might have a command that waits until it sees a PNG file has been created, and then tries to read and process it. But that can happen before the PNG file has been totally written, so the file appears to be corrupt.

A simple workaround is to create an extra command at the end of the first series that writes another file, eg "FINISHED.TXT". The second series of commands doesn't start until it sees that FINISHED.TXT exists.
snibgo's IM pages: im.snibgo.com

mgo
Posts: 11
Joined: 2019-03-17T02:55:46-07:00
Authentication code: 1152

Re: Errors on batch processing

Post by mgo » 2019-04-03T09:54:16-07:00

I am processing the images on a Linux server, from a PHP process. Calling exec('convert and then exec('composite.
I wonder if process ends before the file is fully written, but I guess it's possible. Depends on write time, right?

snibgo
Posts: 12147
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Errors on batch processing

Post by snibgo » 2019-04-03T10:17:00-07:00

I don't use PHP so can't advise. The question is whether exec works asynchronously, ie whether one can start before the previous one has finished.
snibgo's IM pages: im.snibgo.com

mgo
Posts: 11
Joined: 2019-03-17T02:55:46-07:00
Authentication code: 1152

Re: Errors on batch processing

Post by mgo » 2019-04-03T11:55:43-07:00

Dear Snibgo,

Thanks for helping out.

Exec works synchronously. For some reason convert process ends with success status before file is fully written.

Post Reply