ImageMagick memory consumption

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Locked
dkode
Posts: 9
Joined: 2018-04-23T11:26:53-07:00
Authentication code: 1152

ImageMagick memory consumption

Post by dkode »

Configuration :
OS: CentOs

identify -version

Version: ImageMagick 7.0.6-7 Q16 x86_64 2017-08-12 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC HDRI Modules OpenMP
Delegates (built-in): bzlib cairo djvu fftw fontconfig fpx freetype gslib jng jpeg lcms ltdl lzma pangocairo png ps tiff webp wmf x xml zlib



identify -list resource
Resource limits:
Width: 107.374MP
Height: 107.374MP
Area: 33.3722GP
Memory: 15.5401GiB
Map: 31.0803GiB
Disk: unlimited
File: 48750
Thread: 4
Throttle: 0
Time: unlimited


____________________________________________________________________________________________________________________________

I wanted to know how to know the total memory, map and disk used by an imagemagick command.

I tried few options like -debug cache

Can anybody tell me how to exactly interpret the output?

Also I tried valgrind <command> to know the total heap used.

Please guide in this respect as i need to figure out sane resource limits for my application.

Also are the resource limits on per command basis ?

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: ImageMagick memory consumption

Post by fmw42 »

Resource limits are set in your policy.xml file. There are Environmental variable that can be set per session. See https://www.imagemagick.org/script/reso ... nvironment. I do not know the you can change resources per command line. One of the IM developers or users more expert than I would have to respond.

dkode
Posts: 9
Joined: 2018-04-23T11:26:53-07:00
Authentication code: 1152

Re: ImageMagick memory consumption

Post by dkode »

Hi Fred,

I know how to set resource limits, my query is how to get the exact amount of memory used by a command as i am not able to fully comprehend debug cache's output. I need to reach a conclusion for setting resource limits but since i am not fully confident in getting resource consumption metrics for a command, i need some help in this area.

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: ImageMagick memory consumption

Post by fmw42 »

I understand. You will need a response from one of the developers or some one more knowledgeable that me. Sorry I cannot help.

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: ImageMagick memory consumption

Post by fmw42 »

Have you tried -debug resource? See https://www.imagemagick.org/script/comm ... .php#debug. I don't know if that helps or not. I have to used it before.

snibgo
Posts: 13034
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: ImageMagick memory consumption

Post by snibgo »

dkode wrote:I tried few options like -debug cache. Can anybody tell me how to exactly interpret the output?
The question is vague. The output might contain:

Code: Select all

2018-05-13T17:53:41+01:00 0:00.186 0.172u 7.0.7 Cache magick[3916]: cache.c/OpenPixelCache/3582/Cache

  open pink[0] (Heap Memory, 1000x1000x4 32000000B)
This says at a certain time the function OpenPixelCache is allocating heap memory for the image pink[0], for one million pixels each of 4 channels, and it needs 32 million bytes.

What else needs interpreting?

IM needs memory for pixels, and other structures. For small numbers of large images we might ignore "other structures" and just consider pixels. Memory needed per pixel depends on v6 or v7, and number of channels, and HDRI or not, and Q-number.

Your operating system will have facilities that tell you total memory usage per process, eg the Windows "tasklist" command. My process module "shell" enables us to run those commands at any stage during the execution of an IM command.
dkode wrote:Also are the resource limits on per command basis ?
I'm unsure what you mean. Limits can be set per command. If you have two or more IM commands running simultaneously, the limits apply to the commands individually, not collectively. An IM process has no knowledge about any other IM processes that might also be running.
snibgo's IM pages: im.snibgo.com

dkode
Posts: 9
Joined: 2018-04-23T11:26:53-07:00
Authentication code: 1152

Re: ImageMagick memory consumption

Post by dkode »

Hi Snibgo,

I tried -debug resource and got following output

2018-05-14T19:47:10+05:00 0:00.210 0.010u 7.0.6 Resource convert[4343]: resource.c/AcquireMagickResource/319/Resource
Width: 600B/0P/107.374MP
2018-05-14T19:47:10+05:00 0:00.210 0.010u 7.0.6 Resource convert[4343]: resource.c/AcquireMagickResource/319/Resource
Height: 457B/0P/107.374MP
2018-05-14T19:47:10+05:00 0:00.210 0.010u 7.0.6 Resource convert[4343]: resource.c/AcquireMagickResource/319/Resource
Area: 3290400B/3290400P/17.1799GP
2018-05-14T19:47:10+05:00 0:00.210 0.010u 7.0.6 Resource convert[4343]: resource.c/AcquireMagickResource/319/Resource
Memory: 3290400B/3290400B/8GiB
2018-05-14T19:47:10+05:00 0:00.270 0.020u 7.0.6 Resource convert[4343]: resource.c/AcquireMagickResource/319/Resource
Memory: 822600B/4113000B/8GiB
2018-05-14T19:47:10+05:00 0:00.270 0.020u 7.0.6 Resource convert[4343]: resource.c/RelinquishMagickResource/968/Resource
Memory: 822600B/3290400B/8GiB
2018-05-14T19:47:10+05:00 0:00.300 0.040u 7.0.6 Resource convert[4343]: resource.c/RelinquishMagickResource/968/Resource
Memory: 3290400B/0B/8GiB

Can you tell the interpretation of the line?

Memory: 822600B/3290400B/8GiB
Last part is RESOURCE_LIMIT. what about other two.

8GB is the MEMORY_LIMIT, one of them is memory used but what is other one? Please tell if you know.

dkode
Posts: 9
Joined: 2018-04-23T11:26:53-07:00
Authentication code: 1152

Re: ImageMagick memory consumption

Post by dkode »

I got the answer for above question myself by looking at MagickCore code https://www.imagemagick.org/api/MagickC ... ource.html .

The format is <Memory in the current step>/<Total memory used>/ MEMORY_LIMIT

Example:
total acquired=OB
Max limit=8GiB ( to check run : identify -list resource )

2018-05-14T19:47:10+05:00 0:00.210 0.010u 7.0.6 Resource convert[4343]: resource.c/AcquireMagickResource/319/Resource
Memory: 3290400B/3290400B/8GiB
---Acquired memory of size 3290400B, also total acquired=0B+3290400B=3290400B ---

2018-05-14T19:47:10+05:00 0:00.270 0.020u 7.0.6 Resource convert[4343]: resource.c/AcquireMagickResource/319/Resource
Memory: 822600B/4113000B/8GiB
---Acquired memory of size 822600B, total acquired=3290400B+822600B=4113000B ---

2018-05-14T19:47:10+05:00 0:00.270 0.020u 7.0.6 Resource convert[4343]: resource.c/RelinquishMagickResource/968/Resource
Memory: 822600B/3290400B/8GiB
---Released memory of size 822600B, total acquired=4113000B-822600B=3290400B---

2018-05-14T19:47:10+05:00 0:00.300 0.040u 7.0.6 Resource convert[4343]: resource.c/RelinquishMagickResource/968/Resource
Memory: 3290400B/0B/8GiB
---Released memory of size 3290400B, total acquired=3290400B-3290400B=0B---

So at the end all memory taken up has been released to the os.

Locked