4meg image creates multi-gigabyte temp file & crashes

[treaves]

I process lots of images. Last week I was trying to create thumbnails for about 3.2 million PNG line art images. Most of these images are fairly small in size (they are bi-toanl, and in the 10's of k). I wrote a script to convert the images. It kept crashing. At first, I thought it was something with my bash script (which walked the directories & used convert). So I rewrote to use mogrify on a per-directory basis, and the same thing happened.

Upon investigation, what is/was happing is that there would be an image that would create these mulit-gigabyte tmp files, filling the file system, and the kernel would kill the process. I was able to find several of these images. One of them, a 4meg PNG, when I simply use identify, creates a temp file:
-rw------- 1 bv6679 23903448660 Sep 18 09:56 /tmp/magick-RaU0gd0O
and the command just stalls (it does not return any output).

I've do doubt these images are corrupt. But how am I to know that an image is corrupt before this happens? When the OS kills the process, the temp file is left behind, so I'm not even sure I could write a script to moniter for this behaviour.

This seems a very serious defect to me. I've searched for similar issues, and I find refernece to them, but, not to this exact one. I've tried IM versions from 6.2 up through the current 6.7, and they all behave this way.

[magick]

Not a defect. Your PNG image is huge and ImageMagick will attempt to process the image based on its dimensions. While ImageMagick attempts to allocate the pixel cache, it can either complete, run out of temporary disk space, or the OS can kill it. If the OS kills it, the temporary file must be removed manually because the kill signal cannot be intercepted by a user application. Less severe signals are intercepted and any temporary files are removed by ImageMagick before the process exits.

You have two solutions. One is the identify the image before you process it and if it exceeds a certain management size, don't convert it. If you identify an image without specifying the -verbose option (this is called pinging the image), the image pixel cache is not created so even with corrupt images, the process completes. Say your corrupt PNG image is 250000 x 250000 pixels wide. Your script can then skip this image as corrupt based on the large image dimensions.

Another solution is to set resource limits. To prevent excess memory usage, set your memory limit to say 16GB. Now set the disk limit to 4GB. ImageMagick will exit if the disk limit is exceeded. A 250000 x 2500000 pixel image will require more than 4GB of disk so ImageMagick exits with a "cache resource exhausted" exception.

[treaves]

O.K., good ideas. thanks.

[canavan]

This problem can be mitigated - at least on POSIX systems - by deleting the temporary file as soon as it is created. This will actually free the diskspace occupied by the file as soon as it is closed or the ImageMagick terminates. We're using the patch below for automated/unattended use of ImageMagick. I have no idea how or if this works in Windows or other operating systems where opened files cannot be deleted.

Code: Select all

--- ImageMagick-6.8.0-7/magick/resource.c	2012-11-20 16:41:17.000000000 +0100
+++ ../ImageMagick-6.8.0-7/magick/resource.c	2012-12-04 19:39:49.388473225 +0100
@@ -491,6 +491,8 @@
   UnlockSemaphoreInfo(resource_semaphore);
   (void) AddValueToSplayTree(temporary_resources,ConstantString(path),
     (const void *) NULL);
+
+  unlink(path);
   return(file);
 }

[canavan]

Is there any chance to get the above patch or something equivalent included in ImageMagick?

[magick]

No because some temporary files are closed and then reopened. If its reopened, an exception is thrown because it would have already been deleted. In addition, the patch is not cross-platform-- it does not work under Windows.