Extremely slow TIFF conversion (Ubuntu)

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
slowtiff2
Posts: 2
Joined: 2018-11-19T12:54:42-07:00
Authentication code: 1152

Extremely slow TIFF conversion (Ubuntu)

Post by slowtiff2 »

I am seeing general slowness with ImageMagick, sometimes a coredump, and seems to really only happen when dealing with TIFF files.

This occurs from the command-line and using PHP.

I read about a possible issue with OpenMP, and so compiled from source "--without-openmp".

So two questions:
  • how do I verify that OpenMP is "off"?
  • General request for help troubleshooting.

This is a VM running on Azure.

Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial


Here's a command showing how long it's actually taking. The source TIFF is black and white, and 2mb in size.

Attempting to convert causes "Aborted (core dumped)"

Code: Select all

time /usr/local/bin/convert -monitor +dither -map pattern:gray50 339483.tiff out.jpg
load image[339483.tiff]: 19600 of 19601, 100% complete
load image[gray50]: 31 of 32, 100% complete
classify image colors[gray50]: 31 of 32, 100% complete

Aborted (core dumped)

real    12m25.320s
user    0m2.162s
sys     0m2.896s
Here's a command getting image height:

Code: Select all

time /usr/local/bin/convert 339483.tiff -format "%w x %h" info:
22907 x 19601
real    0m15.548s
user    0m0.677s
sys     0m0.981s

edit:
Version: ImageMagick 7.0.8-14 Q16 x86_64 2018-11-19 https://imagemagick.org
Copyright: © 1999-2018 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI
Delegates (built-in): bzlib djvu fftw fontconfig freetype jbig jng jpeg lcms lqr lzma openexr pangocairo png tiff wmf x xml zlib
Last edited by slowtiff2 on 2018-11-20T05:23:20-07:00, edited 1 time in total.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Extremely slow TIFF conversion (Ubuntu)

Post by snibgo »

What version of IM?
slowtiff2 wrote:how do I verify that OpenMP is "off"?

Code: Select all

convert -version
Look under "Features:", and check for "OpenMP".

Your tiff is quite large, 22907 x 19601 pixels. With IM v6 Q16 integer, this needs 6 GB memory. If you don't have enough, this should use disk, which is slower but shouldn't cause a crash.
snibgo's IM pages: im.snibgo.com
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Extremely slow TIFF conversion (Ubuntu)

Post by fmw42 »

how do I verify that OpenMP is "off"?
You can also just limit the threads to 1.

Code: Select all

MAGICK_THREAD_LIMIT=1 convert ....
or

Code: Select all

convert -limit thread 1 ....
slowtiff2
Posts: 2
Joined: 2018-11-19T12:54:42-07:00
Authentication code: 1152

Re: Extremely slow TIFF conversion (Ubuntu)

Post by slowtiff2 »

Thanks to both of you.

I have confirmed that OpenMP is not listed in the features (see my edit above with output of -version).

Including "-limit thread 1" seems to run faster (but I have a very small number of tests) so I will look at adding that to the overall configuration.

Here's the crappy part:
Overall, everything related to ImageMagick seems faster today. Was I blaming ImageMagick when really it was something else? quite possible, though I do not know what that could have been.

Will ImageMagick automatically use disk if RAM is not available? I wonder if there was some slowness with disk I/O? Unfortunately this is an azure VM and I have zero control (and really no access) to see what was going on.


/dev/sda:
Timing cached reads: 16908 MB in 1.99 seconds = 8482.64 MB/sec
Timing buffered disk reads: 98 MB in 3.05 seconds = 32.15 MB/sec


I guess I continue on and see if it comes up again.

Thank you again... very appreciative of your fast response and your time helping.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Extremely slow TIFF conversion (Ubuntu)

Post by snibgo »

If your IM doesn't have OpenMP, then it doesn't multithread, and "-limit threads 1" will make no difference.

On a virtual machine, you share physical resources with other processes. So performance (as measured by a wall clock) will vary.

If you don't have enough RAM for pixel caches, IM will use disk, provided disk usage isn't limited by policy.xml or elsewhere. Use "-list resource" to see the limits.

Disks that physically spin are massively slower than RAM. And this could be worse on a virtual machine -- many processes may be competing for the slow disk access.
snibgo's IM pages: im.snibgo.com
Post Reply