Not using all available resources

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
moocow89
Posts: 7
Joined: 2018-01-16T16:00:03-07:00
Authentication code: 1152

Not using all available resources

Post by moocow89 » 2018-01-16T16:08:01-07:00

Hi,

My machine has 64gig of ram but using "list resource" it appears only 16gig is available to be used.
I've checked the FAQ and reviewd my policy.xml (fresh install, no restrictions in place) and am now out of ideas. Is there something I need to do to utilise the additional ram?

I've been trying to render an SVG at a (stupidly?) high DPI and been running out of memory.

Regards,
Rob

Windows 10 Professional x64

Version: ImageMagick 7.0.7-21 Q16 x64 2018-01-06 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2015 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Visual C++: 180040629
Features: Cipher DPC Modules OpenMP
Delegates (built-in): bzlib cairo flif freetype gslib jng jp2 jpeg lcms lqr openexr pangocairo png ps raw rsvg tiff webp xml zlib

magick convert -list resource
Resource limits:
Width: 214.748MP
Height: 214.748MP
Area: 34.3268GP
Memory: 15.9847GiB
Map: 31.9693GiB
Disk: unlimited
File: 1536
Thread: 8
Throttle: 0
Time: unlimited

User avatar
fmw42
Posts: 23055
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Not using all available resources

Post by fmw42 » 2018-01-16T17:24:55-07:00

Your memory limitation from Imagemagick is about 16GB. You can edit your policy.xml file to increase the IM Memory limitation. see https://www.imagemagick.org/script/resources.php

What do you get from

identify -verbose your.svg

and what density are you trying to use. Perhaps the rasterized SVG file is larger than 16 GB.

Even so, the processing should use disk space if it runs out of memory, but it might be incredibly slow.

moocow89
Posts: 7
Joined: 2018-01-16T16:00:03-07:00
Authentication code: 1152

Re: Not using all available resources

Post by moocow89 » 2018-01-17T00:25:08-07:00

Thanks for the quick reply - I tried changing the policy.xml again and this time it's showing up correctly when I "list policy", clearly user error

moocow89
Posts: 7
Joined: 2018-01-16T16:00:03-07:00
Authentication code: 1152

Re: Not using all available resources

Post by moocow89 » 2018-01-18T03:24:19-07:00

Hmm - I did a fresh install and I'm back to having limited memory, even though I believe I've edited the policy.xml corretly:

policy.xml

Code: Select all

<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="d:/temp/"/> -->
  <policy domain="resource" name="memory" value="50GiB"/>
  <!-- <policy domain="resource" name="map" value="4GiB"/> -->
  <!-- <policy domain="resource" name="width" value="10MP"/> -->
  <!-- <policy domain="resource" name="height" value="10MP"/> -->
  <!-- <policy domain="resource" name="area" value="1GB"/> -->
  <!-- <policy domain="resource" name="disk" value="16EB"/> -->
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <!-- <policy domain="resource" name="thread" value="4"/> -->
  <!-- <policy domain="resource" name="throttle" value="0"/> -->
  <!-- <policy domain="resource" name="time" value="3600"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->
  <!-- <policy domain="coder" rights="none" pattern="MVG" /> -->
  <!-- <policy domain="delegate" rights="none" pattern="HTTPS" /> -->
  <!-- <policy domain="path" rights="none" pattern="@*" /> -->
  <policy domain="cache" name="shared-secret" value="passphrase" stealth="true"/>
</policymap>

Code: Select all

D:\temp>magick convert -list policy

Path: C:\Program Files\ImageMagick-7.0.7-Q16\policy.xml
  Policy: Resource
    name: memory
    value: 50GiB

Path: [built-in]
  Policy: Undefined
    rights: None

Code: Select all

D:\temp>magick convert -list resource
Resource limits:
  Width: 214.748MP
  Height: 214.748MP
  Area: 34.3268GP
  Memory: 15.9847GiB
  Map: 31.9693GiB
  Disk: unlimited
  File: 1536
  Thread: 8
  Throttle: 0
  Time: unlimited

Code: Select all

D:\temp>magick -density 3000 fig.svg -resize 3000x3000 image.png
magick: memory allocation failed `fig.svg' @ error/svg.c/ReadSVGImage/3092.
magick: no images found for operation `-resize' at CLI arg 4 @ error/operation.c/CLIOption/5266.
What am I missing?

moocow89
Posts: 7
Joined: 2018-01-16T16:00:03-07:00
Authentication code: 1152

Re: Not using all available resources

Post by moocow89 » 2018-01-18T11:15:47-07:00

A further data point, if I set memory to <16GB then the policy is correctly applied - so it looks like I'm capped at 16GB for some reason - is it perhaps I need a contiguous ram range? I did a fresh reboot but that didn't apepar to help.

With the policy at 10GB I get the following as expected:

Code: Select all

D:\temp>systeminfo |find "Available Physical Memory"
Available Physical Memory: 60,219 MB

Path: C:\Program Files\ImageMagick-7.0.7-Q16\policy.xml
  Policy: Resource
    name: memory
    value: 10GB

D:\temp>magick convert -list resource
Resource limits:
  Width: 214.748MP
  Height: 214.748MP
  Area: 34.3268GP
  Memory: 10000000000B
  Map: 31.9693GiB
  Disk: unlimited
  File: 1536
  Thread: 8
  Throttle: 0
  Time: unlimited

User avatar
fmw42
Posts: 23055
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Not using all available resources

Post by fmw42 » 2018-01-18T11:48:54-07:00

The policy.xml file can be in several places. You must edit the main one. See https://www.imagemagick.org/script/resources.php. I do not know the Windows file system to tell you where that is. The other locations can only make the limitation smaller.

moocow89
Posts: 7
Joined: 2018-01-16T16:00:03-07:00
Authentication code: 1152

Re: Not using all available resources

Post by moocow89 » 2018-01-18T15:23:36-07:00

Thanks for the suggestions, I've taken a look at the docs here: https://www.imagemagick.org/script/resources.php and they suggest the following search order for Windows:

Code: Select all

$MAGICK_CONFIGURE_PATH
<windows registry>
$PREFIX/config
$USERPROFILE/.config/ImageMagick
<client path>
Just to be sure I dug out SysInternals Process Monitor and traced magick.exe, that confirmed it searched in the following order
  • C:\Program Files\ImageMagick-7.0.7-Q16\policy.xml (exists)
  • C:\Users\rob\AppData\Local\ImageMagick\policy.xml (did not exist)
after this it stopped looking

I tried setting the environment variable "MAGICK_MEMORY_LIMIT" to "50GB" and that appears to work:

Code: Select all

C:\Users\rob>set MAGICK_MEMORY_LIMIT=50GB

C:\Users\rob>magick -list resource
Resource limits:
  Width: 214.748MP
  Height: 214.748MP
  Area: 34.3268GP
  Memory: 50000000000B
  Map: 31.9693GiB
  Disk: unlimited
  File: 1536
  Thread: 8
  Throttle: 0
  Time: unlimited
sadly it didn't solve the OOM when trying very high density rendering of an svg:

Code: Select all

D:\temp>magick -density 3000 fig-trunc.svg -resize 1200x1200 fi3000.png
magick: memory allocation failed `fig-trunc.svg' @ error/svg.c/ReadSVGImage/3092.
magick: no images found for operation `-resize' at CLI arg 4 @ error/operation.c/CLIOption/5266.
I've also noticed corruption in the rendering of SVGs with density > 1900, as the density increase more and more of the bottom of the rendered image is chopped off - see below for 1800, 2100 and 2400 densities (from the same source SVG) - all very strange

1800Image
2100Image
2400Image

User avatar
fmw42
Posts: 23055
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Not using all available resources

Post by fmw42 » 2018-01-18T15:41:29-07:00

Looks like you have RSVG as a delegate. Do you have Inkscape on your system. Perhaps RSVG is limiting it. Inkscape is probably a better renderer. What is your version of RSVG? Is it up to date?

When you add -verbose to your command, what does it say about the Renderer?

moocow89
Posts: 7
Joined: 2018-01-16T16:00:03-07:00
Authentication code: 1152

Re: Not using all available resources

Post by moocow89 » 2018-01-18T16:12:58-07:00

I'm happy to keep trying as many suggestions as you can come up with, although I'm going to give IM a whirl on Ubuntu and see how that looks

Code: Select all

D:\temp>magick -list format | find "SVG"
     MSVG  rw+   ImageMagick's own SVG internal renderer
      SVG  rw+   Scalable Vector Graphics (RSVG 2.40.18)
     SVGZ  rw+   Compressed Scalable Vector Graphics (RSVG 2.40.18)
     
I didn't install RSVG myslef (assume it came with the IM install?)

verbose at 1800

Code: Select all

D:\temp>magick convert -verbose -density 1800 fig-trunc.svg -resize 1200x1200 fi1800.png
fig-trunc.svg SVG 22500x22500 22500x22500+0+0 16-bit sRGB 942570B 12.328u 0:12.351
fig-trunc.svg=>fi1800.png SVG 22500x22500=>1200x1200 1200x1200+0+0 16-bit sRGB 1100650B 35.859u 0:05.024
verbose at 3000

Code: Select all

D:\temp>magick convert -verbose -density 3000 fig-trunc.svg -resize 1200x1200 fi3000.png
convert: memory allocation failed `fig-trunc.svg' @ error/svg.c/ReadSVGImage/3092.
convert: no images defined `fi3000.png' @ error/convert.c/ConvertImageCommand/3275.

User avatar
fmw42
Posts: 23055
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Not using all available resources

Post by fmw42 » 2018-01-18T17:02:06-07:00

22500*22500 = 506250000 or about 506 Mpixels (time 6 for 3 channels at 16-bits per channel = 15 GB) just for the input rasterized SVG file (not counting the 1200x1200 pixel output image, nor for any intermediate image that may be needed). But that should be enough if you have 64 GB RAM and can use it all.

User avatar
fmw42
Posts: 23055
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Not using all available resources

Post by fmw42 » 2018-02-01T12:40:19-07:00

Perhaps you are running out of space in the MAGICK_TEMPORARY_PATH you have assigned or the default on unix is often /tmp. Check to se that it does not get overloaded or have large files left there, especially from ImageMagick. See https://www.imagemagick.org/script/reso ... nvironment

Post Reply