using disk for processing of huge images

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
rogach
Posts: 2
Joined: 2018-01-29T09:50:51-07:00
Authentication code: 1152

using disk for processing of huge images

Post by rogach » 2018-01-29T11:32:46-07:00

Is there a way to tell ImageMagick to use in-memory processing for smaller images, but use disk when files are too big?

For example, I have a following command that requires a lot (~35Gb) of memory:

Code: Select all

$ time convert xc:white -extent 30000x30000 white.png
convert: unable to write pixel cache '/tmp/magick-30697WJTJFL2B7hAI': No space left on device @ error/cache.c/WritePixelCachePixels/5621.
convert: unable to extend cache 'white.png': No space left on device @ error/cache.c/OpenPixelCache/3684.
convert: memory allocation failed `white.png' @ error/png.c/WriteOnePNGImage/9262.

real	0m32.975s
user	0m22.634s
sys	0m10.289s
As you can see, it fails to finish. I tried using -limit options:

Code: Select all

$ time convert -limit memory 8gb -limit map 8gb -limit area 1000MP xc:white -extent 30000x30000 white.png
convert: unable to write pixel cache '/tmp/magick-27824B6CVaIqN2A0C': No space left on device @ error/cache.c/WritePixelCachePixels/5621.
convert: unable to write pixel cache '/tmp/magick-278242gCe9p110hp2': No space left on device @ error/cache.c/WritePixelCachePixels/5621.
convert: memory allocation failed `white.png' @ error/png.c/WriteOnePNGImage/9262.

real	0m33.451s
user	0m22.736s
sys	0m10.581s
The result stays exactly the same - I guess because /tmp is not actual disk, but tmpfs on most systems and is kept in memory.

Well, since we really want to use actual disk, we should instruct ImageMagick to use proper on-disk temp directory - so I tried setting registry:temporary-path:

Code: Select all

$ time convert -limit memory 8gb -limit map 8gb -limit area 1000MP -define registry:temporary-path=/var/tmp xc:white -extent 30000x30000 white.png

real	11m41.129s
user	1m37.469s
sys	1m41.773s
Now it works correctly. However, if I now use this options on small processing loads (that do fit into memory limit), performance is down 4 times (and it's on fast SSD, on slower disks it would be even worse):

Code: Select all

$ time convert xc:white -extent 10000x10000 white.png

real	0m7.166s
user	0m6.383s
sys	0m1.942s
$ time convert  -limit memory 8gb -limit map 8gb -limit area 1000MP -define registry:temporary-path=/var/tmp xc:white -extent 10000x10000 white.png

real	0m26.342s
user	0m6.815s
sys	0m6.935s

Thus my question - why ImageMagick still uses temporary directory, if processing fits into memory? Is there a way to make it skip disk if there is plenty of memory?

These test were done using ImageMagick 7.0.7-22, on Linux 4.14.13-1-ARCH.

snibgo
Posts: 10482
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: using disk for processing of huge images

Post by snibgo » 2018-01-29T16:09:49-07:00

Weirdly, the case matters. Use "8GB" (capitals) not "8gb" (lower case).
snibgo's IM pages: im.snibgo.com

rogach
Posts: 2
Joined: 2018-01-29T09:50:51-07:00
Authentication code: 1152

Re: using disk for processing of huge images

Post by rogach » 2018-01-29T23:05:48-07:00

Wow! Thanks!

Post Reply