Problem converting large JPGs

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
pnt103
Posts: 4
Joined: 2018-08-01T08:51:20-07:00
Authentication code: 1152

Problem converting large JPGs

Post by pnt103 » 2018-08-01T09:48:30-07:00

I'm trying to resize several large B&W images using convert, but it keeps core-dumping. Small files work OK, though. Typical image file is around 12MB, 19200 x 13100 pixels, 16-bit greyscale, JPG format. This is using ImageMagick 6.6.9-7 on a machine running Ubuntu 12.04.5 LTS, with 6GB RAM and 1.8TB of free space on the disk partition I'm using.

I've checked ulimit, and there are no limits on memory or disk use, tried setting MAGICK_TMPDIR to the directory I'm in (and which the images are in) so there's plenty of disk space available. Here's what happens:

$ convert -debug cache LR01.jpg -resize 900 LR01_900.jpg
2018-08-01T17:08:27+01:00 0:00.000 0.000u 6.6.9 Cache convert[26572]: cache.c/DestroyPixelCache/1540/Cache
destroy
2018-08-01T17:08:27+01:00 0:00.000 0.000u 6.6.9 Cache convert[26572]: cache.c/ExtendCache/4020/Cache
extend LR01.jpg[0] (/raid2TB/pete/tmp/magick-0iQ1wOxO[4], disk, 2.5423GBB)
2018-08-01T17:08:27+01:00 0:00.000 0.000u 6.6.9 Cache convert[26572]: cache.c/OpenPixelCache/4246/Cache
open LR01.jpg[0] (/raid2TB/pete/tmp/magick-0iQ1wOxO[4], disk, 19285x13183 2.5423GBB)
Segmentation fault (core dumped)

The ultimate goal is to create two reduced-size copies for web use, one about 900 pixels wide for display, and another as a smaller thumbnail. What am I missing?

User avatar
fmw42
Posts: 25042
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Problem converting large JPGs

Post by fmw42 » 2018-08-01T11:50:32-07:00

What is your version of libjpeg? You can find that from

Code: Select all

convert -list format
Perhaps you need to upgrade it.

Otherwise, check your policy.xml file for other limitations such as on width or height or memory, etc, though you seem to have perhaps checked that already.

If you want someone to test, then post your input JPG to some free hosting service that will not modify it and put the URL here.

On Linux, the version 6.6.9.7 would appear to be ancient. But they do not change versions when they patch. So the more important fact would be the date associated with the version. Be sure you have the latest patch from your Linux distro.

User avatar
magick
Site Admin
Posts: 10984
Joined: 2003-05-31T11:32:55-07:00

Re: Problem converting large JPGs

Post by magick » 2018-08-02T04:08:22-07:00

Agreed. Try upgrading to the latest release. If that fails, let us know and we will investigate further.

pnt103
Posts: 4
Joined: 2018-08-01T08:51:20-07:00
Authentication code: 1152

Re: Problem converting large JPGs

Post by pnt103 » 2018-08-02T08:02:35-07:00

Thank you replying so quickly.

"convert -list format" just lists (a lot of) formats without giving versions:

Code: Select all

JPEG* JPEG      rw-   Joint Photographic Experts Group JFIF format (80)
JPG*  JPEG      rw-   Joint Photographic Experts Group JFIF format (80)
JPX*  JP2       rw-   JPEG-2000 File Format Syntax
My libjpg is /usr/lib/x86_64-linux-gnu/libjpeg.so.8.0.2

ImageMagick itself is 8:6.6.9.7-5ubuntu3.9 with libMagickCore.so.4.0.1 and libMagickWand.so.4.0.1, or from "convert -version":

Code: Select all

Version: ImageMagick 6.6.9-7 2017-03-14 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2011 ImageMagick Studio LLC
Features: OpenMP
Clearly a fairly old but not (?) ancient version, probably because although I only installed it a few days ago, I did it using apt-get and my version of Ubuntu Linux isn't very new (12.04 LTS, whereas the current is 18.04 LTS - 2 or 3 LTS versions different). I wonder why they don't update application version numbers. I'm not sure what difference upgrading the O/S would make to installing a pre-built ImageMagick - but upgrading the O/S would entail big changes to the whole system.

Anyway, it's the current version as far as Ubuntu thinks I should have. Otherwise I think I'd have to build ImageMagick from source; not impossible but I'd like to avoid that if I can.

For completeness, here's the important part of policy.xml:

Code: Select all

<policymap>
  <!-- <policy domain="system" name="precision" value="6"/> -->
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <!-- <policy domain="resource" name="memory" value="2GiB"/> -->
  <!-- <policy domain="resource" name="map" value="4GiB"/> -->
  <!-- <policy domain="resource" name="area" value="1gb"/> -->
  <!-- <policy domain="resource" name="disk" value="16eb"/> -->
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <!-- <policy domain="resource" name="thread" value="4"/> -->
  <!-- <policy domain="resource" name="throttle" value="0"/> -->
  <!-- <policy domain="resource" name="time" value="3600"/> -->
</policymap>
If my files are roughly 19200 x 13500 pixels in 16-bit greyscale, I reckon they should be a little over half a gig uncompressed, so the limits seem OK. The machine has 6GB RAM, 6GB swap space, over 80GB free in /tmp, and 1.7GB free on the drive the images are on.

Any suggestions? Do you have any particular reason to think upgrading would make a difference to what should be a straightforward operation?

snibgo
Posts: 11710
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Problem converting large JPGs

Post by snibgo » 2018-08-02T08:35:58-07:00

Your policy.xml has all lines commented-out. What does "convert -list resource" say?

I've never used patched versions of IM (except where I have made the patches). You version has obviously been partially patched (because the version number is in the source code, and that hasn't been patched). My question would be: how do we know that a consistent set of patches has been applied?
snibgo's IM pages: im.snibgo.com

pnt103
Posts: 4
Joined: 2018-08-01T08:51:20-07:00
Authentication code: 1152

Re: Problem converting large JPGs

Post by pnt103 » 2018-08-02T08:57:18-07:00

Oh, spit. Sorry, guys, it's too hot here, and I only just realised that policy has all the relevant parts commented out. And apparently that was the problem. I edited it to remove the comments and adjust the values, and now I can do what I want. I've not tried everything, but "so far, so good". fmw42, thank you for pointing me in the right direction.

pnt103
Posts: 4
Joined: 2018-08-01T08:51:20-07:00
Authentication code: 1152

Re: Problem converting large JPGs

Post by pnt103 » 2018-08-02T09:03:11-07:00

snibgo wrote:
2018-08-02T08:35:58-07:00
Your policy.xml has all lines commented-out. What does "convert -list resource" say?
Yes, I realised that just before you posted. For the sake of completeness:

Code: Select all

$ convert -list resource
File         Area       Memory          Map         Disk    Thread         Time
-------------------------------------------------------------------------------
 768          3GB         3GiB         8GiB   13.877788EiB         4         3600
Before I removed the comment marks and edited the values it was:

Code: Select all

$ convert -list resource
File         Area       Memory          Map         Disk    Thread         Time
-------------------------------------------------------------------------------
 768     1.5321GB    5.7077GiB    11.415GiB    unlimited         2    unlimited
Just needs a bit of tweaking now.

User avatar
fmw42
Posts: 25042
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Problem converting large JPGs

Post by fmw42 » 2018-08-02T11:16:58-07:00

I would also point out that your libjpeg at 80 is old. The current version is 90. So you might also consider upgrading it. Also the date of you patch of ImageMagick is 3/14/2017. That is over a year ago. So perhaps there is a more current patch. You should check with your Linux distro people to find out if there is a newer version, if you still have trouble.

Post Reply