Limiting resources for large images efficiently

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
meraj
Posts: 2
Joined: 2017-05-24T11:28:42-07:00
Authentication code: 1151

Limiting resources for large images efficiently

Post by meraj » 2017-05-24T11:45:13-07:00

I am trying to convert an image with following details:
Geometry: 15360x8640+0+0
Resolution: 300x300
Depth: 8-bit
Pixels: 132710400

I initially tried with this command:

Code: Select all

convert /tmp/16MB-test.jpg -quality 95 -sampling-factor 4:2:2 -background white -flatten -resize 960 /tmp/960x960/16MB-test-preview.jpg
The file is converted successfully but I noticed it is taking upto 2GB memory. Which is above my expectations and I was expecting it to take ~300MB memory. I tried to convert same image by limiting memory, map and disk as per my guess but it didn't work. I tried with the following but it fails saying cache resources exhausted.

Code: Select all

convert /tmp/16MB-test.jpg  -limit memory 256MB -limit map 512MB -limit disk 500MB -quality 95 -sampling-factor 4:2:2 -background white -flatten -resize 960 /tmp/960x960/16MB-test-preview.jpg
convert /tmp/16MB-test.jpg  -limit memory 512MB -limit map 1GB -limit disk 500MB -quality 95 -sampling-factor 4:2:2 -background white -flatten -resize 960 /tmp/960x960/16MB-test-preview.jpg
I am not sure why it is failing and how I can limit the memory and disk as per my requirements and still be able to convert this image and some other large size images. We need to run this program on AWS Lambda where resources are limited Max memory is 1.5GB while storage is 500MB.

Can anyone suggest in achieving max out of this limitation? Thanks

User avatar
fmw42
Posts: 25419
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Limiting resources for large images efficiently

Post by fmw42 » 2017-05-24T12:04:39-07:00

You need at least 15360*8640*3*2=796262400 or nearly 800 MB. The 3 is for 3 channels of 15360*8640 pixels. The 2 is that you need both the input and output images in memory before writing the output. But 800 MB does not include the decompression of the jpg, which could be quite substantial and could account for another factor of 5 to get to 3.2 GB or perhaps even more. IM will use as much memory as you allow and then use disk space to handle the rest. So it could be slow if you do not have enough RAM to hold two copies of your decompressed image.

I defer to others more knowledgeable than I on such matters of large image processing.

snibgo
Posts: 12025
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Limiting resources for large images efficiently

Post by snibgo » 2017-05-24T12:30:57-07:00

The input and outpts are jpg, which is 8 bits/channel. What Q-number IM are you using? Q8 will use half the memory of Q16, with no loss of quality for this command.

In general, IM runs as fast as it can. You can limit the memory usage if you want, which will slow it down. But if you also limit the disk usage, this can stop it from working, as you have found.
snibgo's IM pages: im.snibgo.com

User avatar
fmw42
Posts: 25419
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Limiting resources for large images efficiently

Post by fmw42 » 2017-05-24T12:38:02-07:00

Thanks snibgo. That does point out a mistake I made above. I forgot to multiply by 8 or 16. So for Q16 IM (default). My results above should have (taking into account decompression to full 3 * 16-bits-per-channel) been.

15360*8640*3*16*2=8493465600 or about 8.5 GB

snibgo
Posts: 12025
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Limiting resources for large images efficiently

Post by snibgo » 2017-05-24T13:27:25-07:00

For v6 Q16, the basic calculation I use is each image needs 8 bytes/pixel, so 15360*8630*8/1e9 = 1.0604544 GB.

The "-flatten" is pointless, as the JPG input is fully opaque, but I doubt that IM is smart enough to realise this. So this takes the same space again. Removing "-background white -flatten" will probably save memory.

Then the first image is released, and the result of the "-resize" is smaller. So the peak requirement is 2.12 GB.

I don't know how much memory is needed to decompress the input JPG, or to compress the output JPG.
snibgo's IM pages: im.snibgo.com

User avatar
fmw42
Posts: 25419
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Limiting resources for large images efficiently

Post by fmw42 » 2017-05-24T14:21:47-07:00

snibgo wrote:
2017-05-24T13:27:25-07:00
For v6 Q16, the basic calculation I use is each image needs 8 bytes/pixel
Why 8 bytes per pixel? Each pixel is 3 channels x 16 bits-per-channel = 48 bits = 6 bytes? But then you need to multiply by two for the output image in memory before writing to disk? So 12 bytes.

What am I missing?

snibgo
Posts: 12025
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Limiting resources for large images efficiently

Post by snibgo » 2017-05-24T15:11:00-07:00

IM v6 stores non-palette sRGB (or RGB) images internally with 4 channels: RGBA.
snibgo's IM pages: im.snibgo.com

User avatar
fmw42
Posts: 25419
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Limiting resources for large images efficiently

Post by fmw42 » 2017-05-24T15:55:50-07:00

Got it. Thanks. I forgot about that.

meraj
Posts: 2
Joined: 2017-05-24T11:28:42-07:00
Authentication code: 1151

Re: Limiting resources for large images efficiently

Post by meraj » 2017-05-25T02:53:57-07:00

Thanks for the details information regarding how the logic works.

I am using ImageMagick 6.9.8-3 Q16 x86_64 2017-04-24 version.
For v6 Q16, the basic calculation I use is each image needs 8 bytes/pixel, so 15360*8630*8/1e9 = 1.0604544 GB.
Why does it take 8 bytes/pixel when the depth is only 1 byte. Can you explain this logic a bit more? As I was calculating that it should take 1 byte/pixel by taking into account each pixel's depth (8-bits) and then multiplied by 3 channels.

Also if there is a mode where bytes are packed so conversions can still happen with limited memory / disk but in more time?

Thanks

snibgo
Posts: 12025
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Limiting resources for large images efficiently

Post by snibgo » 2017-05-25T07:20:42-07:00

Q16 means that pixels are stored internally (in memory) with 16 bits (2 bytes) per channel. Similarly, Q8 store pixels with 8 bits (1 byte). In your case, Q8 is sufficient.

In the other direction, Q32 and Q64 are available. Some applications need this higher precision.

So Q16 takes 2 bytes per channel per pixel. For non-palette images, IM v6 stores pixels with 4 channels (RGBA) or 5 channels (CMYKA).

IM V7 may use less memory than v6 when an image has less than 4 channels, but I haven't tested this.
snibgo's IM pages: im.snibgo.com

User avatar
magick
Site Admin
Posts: 11043
Joined: 2003-05-31T11:32:55-07:00

Re: Limiting resources for large images efficiently

Post by magick » 2017-05-25T08:07:31-07:00

IMv7 uses one channel for grayscale images, two if an alpha channel is included.

Post Reply