convert is toooooo slow with many images

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
saulortega
Posts: 3
Joined: 2015-06-13T17:30:27-07:00
Authentication code: 6789

convert is toooooo slow with many images

Post by saulortega »

Hello.

Version: ImageMagick 6.6.9-7 2014-03-06 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2011 ImageMagick Studio LLC
Features: OpenMP

I'm running this command:

Code: Select all

convert *.JPG -morph 2 -monitor -normalize mov/%05d.jpg
With 10 images, it takes 15 seconds. With 30 images, it takes 1 hour!. :shock: :shock: :shock:

With 10 images, it do not writes on disk. With 30 images (~1.2 MB) it writes 400 MB in temp disk. With 50 images 3 GB! :shock:


Why???!!!
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: convert is toooooo slow with many images

Post by fmw42 »

You are reading every image into memory. That will slow things down. See http://www.imagemagick.org/Usage/files/#massive for processing large image(s).
saulortega
Posts: 3
Joined: 2015-06-13T17:30:27-07:00
Authentication code: 6789

Re: convert is toooooo slow with many images

Post by saulortega »

Code: Select all

However your system may reject large memory requests and still force IM to cache the image to disk (about 1000 times slower).
:shock: :shock: :shock: That explains it.

Is not there some way to tell IM not load all images in memory at once but in twos? :)
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: convert is toooooo slow with many images

Post by magick »

You could write a shell script to call the convert utility for every two images. You might get some improvement by upgrading to the latest ImageMagick release @ 6.9.1-5. You might get some improvement by forcing all pixel access to disk (to prevent swap thrashing) by adding "-limit memory 2mb -limit map 2mb" to your command line. You might get some improvement by setting the MAGICK_TMPDIR environment variable to a partition other than /tmp, instead use /data/tmp, for example.
saulortega
Posts: 3
Joined: 2015-06-13T17:30:27-07:00
Authentication code: 6789

Re: convert is toooooo slow with many images

Post by saulortega »

Well, I programmed my own script. Now I takes 1 hour processing 1000 images.
Nemo_bis
Posts: 11
Joined: 2012-06-16T01:13:18-07:00
Authentication code: 13

Re: convert is toooooo slow with many images

Post by Nemo_bis »

Another experience: to create a PDF from 24 JPG files, convert eats 3 GiB RAM and 4 GiB swap, bringing the machine to swapdeath, and is eventually killed by Linux.

This makes sense given viewtopic.php?f=1&t=28438&p=126258 says we need 8 bytes per pixel and these are 35 MPx images: 35*10**6*24*8/1024**3 = 6 GiB memory needed. And I'd need to convert ~300 such images, not just 24!

Changing the resource limits makes it die faster but not complete the job, it just can't take this much memory on my 8 GB RAM machine.

Code: Select all

$ identify -version
Version: ImageMagick 6.8.8-10 Q16 x86_64 2015-03-10 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2014 ImageMagick Studio LLC
Features: DPC Modules OpenMP
Delegates: bzlib cairo djvu fftw fontconfig freetype gslib jng jpeg lcms ltdl lzma openexr pangocairo png ps rsvg tiff webp wmf x xml zlib

$ convert -list resource
  File       Area     Memory        Map       Disk   Thread  Throttle       Time
--------------------------------------------------------------------------------
 38400   15.948GB  7.4266GiB  14.853GiB  unlimited        2         0  unlimited

$ time convert 000*jpg output.fromjpeg75.pdf 
Ucciso

real    7m23.713s
user    0m21.734s
sys     0m13.826s

$ time convert -limit memory 7GiB 000*jpg output.fromjpeg75.pdf
Ucciso

real    0m36.269s
user    0m22.991s
sys     0m10.424s
In contrast mogrify completes the job in a reasonable time, but then I have to merge the individual PDF files. As magick suggested in another thread about large collections of TIFF, I tried tiffcp + tiff2pdf instead and it's indeed way faster: 1 min instead of 4 for the same PDF.

Code: Select all

$ time mogrify -format jpg -quality 75 000*tif

real    1m46.553s
user    1m6.503s
sys     0m17.276s

$ time mogrify -format pdf 000*jpg

real    4m10.036s
user    3m19.764s
sys     0m23.655s

$ time tiffcp -c lzw 000*tif out.lzw.tiff
real    0m57.404s
user    0m35.480s
sys     0m4.750s

$ time tiff2pdf -o output.jpeg75.pdf -j -q 75 -u m -F out.lzw.tiff 

real    0m55.016s
user    0m49.864s
sys     0m4.687s

$ time tiff2pdf -o output.lossless.pdf -z -u m -F out.lzw.tiff 

real    4m15.793s
user    4m1.839s
sys     0m9.028s
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: convert is toooooo slow with many images

Post by magick »

Note, ImageMagick resource limits are enforced for ImageMagick, not its delegate libraries. We cannot control, for example, how much resources Ghostscript consumes or the JPEG library.
Post Reply