identify on xml file

[bananas2]

sometimes our system runs identify on files to check their type. everything is ok, except xml files (doesnt depend on file extension eg xml, inx...):

identify -verbose /disk03/volume03/32_HAZ_Harke_03.xml
identify: Must specify image size `/tmp/magick-XX6jrUmW'.

and temp file is never deleted.

os linux
im 6.3.5

[magick]

We tried to reproduce the problem you reported with the latest ImageMagick release and could not. XML files returned an exception and any temporary files were removed as expected.

[bananas2]

file content build.xml

<?xml version="1.0" encoding="UTF-8"?>

<!--
Copyright 2007 Medsea Business Solutions S.L.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

-->

<project name="MimeUtil" default="default" basedir=".">

<description>Mime Type Utility.</description>

<!-- Module name -->
<property name="module.name" value="mime-util"/>


<!-- Loads environment variables as properties -->
<property environment="env"/>

<!-- Directory mapping -->
<property name="root.dir" location="${basedir}"/>

<property name="src.dir" location="src"/>
<property name="build.dir" location="build"/>
<property name="dist.dir" location="dist"/>
<property name="test.dir" location="test"/>
<property name="lib.dir" location="lib"/>
<property name="docs.dir" location="docs"/>

<property name="build.classes.dir" location="${build.dir}/classes"/>
<property name="build.test.classes.dir" location="${build.classes.dir}/test"/>

<property name="javadocs.dir" location="${docs.dir}/javadocs"/>

<property name="test.src.dir" location="${test.dir}/src"/>
<property name="test.data.dir" location="${test.dir}/data"/>
<property name="test.reports.dir" location="${docs.dir}/testreports"/>

<!-- User's properties -->
<property file="build.properties"/>

<!-- names of targets -->
<property name="jar_file" value="mime-util.jar" />

<!-- Initializes the working directories -->
<target name="do-init">
<tstamp/>
<mkdir dir="${build.classes.dir}"/>
<mkdir dir="${dist.dir}"/>
</target>

<!-- Clean build environment -->
<target name="do-clean">
<delete dir="${build.dir}"/>
<delete dir="${dist.dir}"/>
</target>

<!-- Compile the sources -->
<target name="do-compile" depends="do-init">
<javac srcdir="${src.dir}"
destdir="${build.classes.dir}"
debug="${javac.debug}"
verbose="${javac.verbose}"
deprecation="${javac.deprecation}">
<include name="**/*.java"/>
</javac>
</target>

<target name="create-jar" depends="do-compile">
<jar destfile="${dist.dir}/${jar_file}">
<fileset dir="${build.classes.dir}">
<include name="**/*.class"/>
</fileset>
<fileset dir="${src.dir}">
<include name="**/*.properties"/>
<include name="**/magic.mime"/>
</fileset>
</jar>
</target>

<!-- Default target -->
<target name="default" description="Default target." depends="all"/>

<!-- Clean ANT working directories -->
<target name="clean" depends="do-clean" description="Clean ANT working directories."/>

<!-- Generates distribution files. -->
<target name="dist" depends="create-jar" description="Generates distribution files."/>

<!-- Clean, build, deploy, test, docs the module. -->
<target name="all" description="Clean, build the module." depends="clean, dist"/>

</project>

output :

on windows :
D:\Projects\ISY2_0_0\dist\fs\volume1>identify -verbose build.xml
identify: Must specify image size `magick-nmEv7yzi'.

instead of identify: no decode delegate for this image format

vaersion:
D:\Projects\ISY2_0_0\dist\fs\volume1>identify -version
Version: ImageMagick 6.3.6 11/02/07 Q8 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2007 ImageMagick Studio LLC


on linux(my system not customer):

-bash-2.05b$ identify -verbose build.xml
ERROR: meta.c (179): wmf_header_read: this isn't a wmf file
identify: Delegate failed `"wmf2eps" -o "%o" "%i"'.
identify: unable to open image `/tmp/magick-XX6FkeXF': No such file or directory.
identify: UnableToOpenBlob `/tmp/magick-XX6FkeXF': No such file or directory.

seems to be no access rights

version:

-bash-2.05b$ identify -version
Version: ImageMagick 6.3.6 11/16/07 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2007 ImageMagick Studio LLC

--------------------

customer logs:

''2008-02-14 16:10:56,494 INFO IMCallableBuilder - identify -verbose 02_SchwVZBuetzow_01.xml
''2008-02-14 16:10:56,697 ERROR extract() IMCallableBuilder - identify: Must specify image size `/tmp/magick-XXQx5md1'.

--------------------

what version do you test?

[bananas2]

customer system:

On system with heavy load a lot of temp files are created in very short time. On
one system it takes ~5minutes sometimes, until the root filesystem is full,
because some temp files are several GB in size.

Version: ImageMagick 6.3.5 09/05/07 Q16

probably not xml files cause this problem, but error logs doesnt help? a've tested many file types (indd, psd, tiff ...)

i've seen types of temp files on our test system:
0-size
small-size 30-40 kb

still its not clear for me how Gb temp file can be created.

[magick]

We're using ImageMagick 6.3.8-7, the latest release. Your command works fine. It returns an exception as expected (since XML files are not recognized by ImageMagick) and does not leave any temporary files around.

Regarding filling up the filesystem. ImageMagick can consume huge amounts of resources if the images processed are large/huge (see http://www.imagemagick.org/script/architecture.php). The solution is to limit the number of simultaneous ImageMagick processes running, ensure all ImageMagick temporary files are being clean-up when the process exits, and of course buy more disk. We just bought a 1TB disk for $349.

[bananas2]

is there any way to debug or log creation of temp files? to determine processing of which file caused creation of temp file?

only 4 IM processes were killed when hard was full, but there were many small sized files instead few huge (Gb)

can you give example file(point url) which cause imageMagick to consume huge amounts of resources?

thanks.

[magick]

Add -debug resource to your command-line to monitor the creation/destruction of ImageMagick temporary files.

[signsrus]

I know this probably doesn't help much but I encountered the same problem. It is actually a very serious problem that needs to be addressed. A very large (over 1GB!) temp file was written to the /tmp file. Why is this so bad? Because it filled up that partition AND it is used by PHP to write session files. So nobody can login into our site! I already deleted the file and everything was fine. It has not happened since and hasn't happened ever. Let me know if I can provide any more information. Version 6.0.7 4/19/08. Thanks

[magick]

ImageMagick provides resource limits which can prevent runaway image processing jobs. See http://www.imagemagick.org/script/archi ... .php#cache for a discussion of the pixel cache.

Unfortunately, your version of ImageMagick has a minimum limit of 1GB of disk, meaning the pixel cache would need to exceed 1GB of temporary storage before ImageMagick exits with a "resource exhausted" message. Recent versions permit specifying a limit in bytes. Here is an example:

• convert -limit memory 32mb -limit map 64mb -limit disk 256mb image.pdf image.png

The image.pdf pixels are cached to memory unless it consumes more than 32MB of memory, otherwise it is written to disk and memory-mapped unless it exceeds 64MB of memory. For that case, the pixels are cached to disk unless they consume more than 256MB of disk. For that case, ImageMagick exits with a "resource exhaused" exception (and deletes any temporary disk space it created).

[signsrus]

Cool, that helps a lot. I will add those attributes to my script. Thank you.

Are you saying our version CAN or CANNOT have resource limitations specified in the script?

If I am unable to specify limits in the script, is there a config file I can edit?

What about using the 'identify' command. Would I need to include those limits to that too.

What limits do you suggest.

This is the output of 'free' command:

root@ds37 [~]# free
total used free shared buffers cached
Mem: 505316 452160 53156 0 26496 214772
-/+ buffers/cache: 210892 294424
Swap: 2048248 58736 1989512

Wow! Looks like there is only 53K free RAM. Is that right?

Here is the output of 'df' command:

root@ds37 [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda5 2016016 692776 1220828 37% /
/dev/sda1 101086 27774 68093 29% /boot
none 252656 0 252656 0% /dev/shm
/dev/sda8 60166880 12444500 44666016 22% /home
/dev/sda7 1510032 36256 1397068 3% /tmp
/dev/sda3 4032124 2716404 1110892 71% /usr
/dev/sda2 7052496 1566312 5127936 24% /var
/tmp 1510032 36256 1397068 3% /var/tmp

What do you think we should do?

Is there an attribute that tells it to use a different temporary directory instead of /tmp?

One more thing...is .ai files supported? If not, do you see when they might?

Thanks.

Shawn

[magick]

The resource limits work for your version of ImageMagick, its just that you can only specify disk resources in GBs. So if you set the disk limit to 1, ImageMagick will exit if the pixel cache exceeds 1GB. With modern releases you can specify in bytes (e.g. 256mb).

You can set an alternative path to temporary storage with the MAGICK_DEBUG environment variable.

[signsrus]

Nice. Where do I change that variable?

[magick]

Its an environment variable. One way:

• export MAGICK_TMPDIR=/data
  convert ...

for example.

[signsrus]

Awesome. Thank you so much. Any tips on upgrading my version?

[signsrus]

What do you suggest using for limits if we allow .psd, .ai, .eps, .jpg, .bmp, .tiff, .pdf, .png and .gif files up to 16MB to be uploaded and processed by ImageMagick???

I have:

(Where thisfile.psd <= 16MB)

-convert -limit map 16mb -limit disk 32mb thisfile.psd -resize '960x960>' thatfile.png

Thank you,

Shawn