C++ API graceful handling of MAGICK_TMPDIR becoming full

Questions and postings pertaining to the development of ImageMagick, feature enhancements, and ImageMagick internals. ImageMagick source code and algorithms are discussed here. Usage questions which are too arcane for the normal user list should also be posted here.
Post Reply
whatdoido
Posts: 25
Joined: 2011-01-22T16:54:46-07:00
Authentication code: 8675308

C++ API graceful handling of MAGICK_TMPDIR becoming full

Post by whatdoido » 2017-09-22T02:45:20-07:00

I have a C++ process on Linux that processes a large number of images - when TMPDIR / MAGICK_TMPDIR is full the process crashes with SIGBUS

I can make this situation appear more readily by creating a small 1MB tmpfs filesystem and pointing MAGICK_TMPDIR at it - the crash/output is produced with that setup

Code: Select all

Thread 1066 "imgcat" received signal SIGBUS, Bus error.
[Switching to Thread 0x7fff6ae31700 (LWP 9427)]
0x00007fffbddf561e in ReadJPEGImage () from /usr/lib64/ImageMagick-6.9.9/modules-Q16/coders/jpeg.so
(gdb) where
#0  0x00007fffbddf561e in ReadJPEGImage () from /usr/lib64/ImageMagick-6.9.9/modules-Q16/coders/jpeg.so
#1  0x00007ffff73f396e in ReadImage () from /lib64/libMagickCore-6.Q16.so.5
#2  0x00007ffff73be7a0 in BlobToImage () from /lib64/libMagickCore-6.Q16.so.5
#3  0x00007ffff7bb1107 in Magick::Image::read(Magick::Blob const&) () from /lib64/libMagick++-6.Q16.so.8
#4  0x00007ffff7bb1169 in Magick::Image::Image(Magick::Blob const&) () from /lib64/libMagick++-6.Q16.so.8
#5  0x000000000041db18 in ImgThumbGen::_genthumbnail (this=0xb726e0, path_=".thumbs/6091047-6450.jpg",
At frame 5

Code: Select all

Magick::Image  img(Magick::Blob( preview_.pData(), preview_.size() ));
the size of buffer provided is 4,296,964 bytes so not large/inline with the last several hundred images that were processed.

Is there a recommended way to trap and handle the condition whereby ImageMagick fails with generating its tmp files in TMPDIR/MAGICK_TMPDIR locations?

Using ImageMagick 6.9.9-13 Q16 on Fedora26 (from their official repos)
Last edited by whatdoido on 2017-09-22T03:33:38-07:00, edited 1 time in total.

User avatar
magick
Site Admin
Posts: 10690
Joined: 2003-05-31T11:32:55-07:00

Re: C++ API graceful handling of MAGICK_TMPDIR full

Post by magick » 2017-09-22T03:28:07-07:00

To avoid SIGBUS, either point MAGICK_TMPDIR to a partition with plenty of free space or lose some performance by eliminating memory mapping. That is, set your MAP limit to 0. From the command-line its -limit MAP 0. Now when the resource limits are reached, you should get a graceful exit on I/O error.

whatdoido
Posts: 25
Joined: 2011-01-22T16:54:46-07:00
Authentication code: 8675308

Re: C++ API graceful handling of MAGICK_TMPDIR becoming full

Post by whatdoido » 2017-09-22T03:44:45-07:00

Thanks for the response - I understand that to avoid this problem, its best to point at a large enough partition for internal magick operations.

However, by default on linux, it appears that magick uses /tmp. On many systems, /tmp is 1-2GB in size - with this default setup, my program crashes with SIGBUS which is not great. When you say:
lose some performance by eliminating memory mapping. That is, set your MAP limit to 0. From the command-line its -limit MAP 0
what is the equivalent when using the C++ API? And what happens to Magick API calls under this situation?

For instance, in the call to:

Code: Select all

Magick::Image  img(Magick::Blob( preview_.pData(), preview_.size() ));
would an exception (and if so, what?) is thrown?

Interestingly enough, when running my process a second time when MAGICK_TMPDIR was already full, I didn't get the SIGBUS, rather messages:

Code: Select all

Magick: unable to extend cache `': No space left on device @ error/cache.c/OpenPixelCache/4074

User avatar
magick
Site Admin
Posts: 10690
Joined: 2003-05-31T11:32:55-07:00

Re: C++ API graceful handling of MAGICK_TMPDIR becoming full

Post by magick » 2017-09-22T04:28:01-07:00

Found a better solution-- set this policy

Code: Select all

<policy domain="cache" name="synchronize" value="True"/>
See https://www.imagemagick.org/script/security-policy.php. Set to True to ensure all image data is fully flushed and synchronized to disk. There is a performance penalty, however, the benefits include ensuring a valid image file in the event of a system crash and early reporting if there is not enough disk space for the image pixel cache.

whatdoido
Posts: 25
Joined: 2011-01-22T16:54:46-07:00
Authentication code: 8675308

Re: C++ API graceful handling of MAGICK_TMPDIR becoming full

Post by whatdoido » 2017-09-22T06:36:46-07:00

Thanks - I set this value in /etc/ImageMagick-6/policy/xml and it does not bus error but still dies.

I still think there is a bug - the BUS error in original stack trace happens when, at the start of the program, the MAGICK_TMPDIR has capacity (ie not full). As the process runs, IM creates the tmp files of size

Code: Select all

11653 open("/tmp/magick-9076bm47ay5yKLRB", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
11653 fchmod(3, 0600)                   = 0
11653 lseek(3, 0, SEEK_END)             = 0
11653 pwrite64(3, "\0", 1, 289218559)   = 1
The size of each tmp file is constant - in the BUS errror situation, it only happens when we can't do alloc this file. It seems there should be a better way to

If I run the program on the same input set with MAGICK_TMPDIR at 100% the BUS erorr does not happen.

Post Reply