AWS Lambda with IM7

Questions and postings pertaining to the development of ImageMagick, feature enhancements, and ImageMagick internals. ImageMagick source code and algorithms are discussed here. Usage questions which are too arcane for the normal user list should also be posted here.
Post Reply
malinlesinge
Posts: 6
Joined: 2019-10-18T01:29:25-07:00
Authentication code: 1152

AWS Lambda with IM7

Post by malinlesinge » 2019-10-18T01:48:16-07:00

Hi,

I am using ImageMagick in a AWS Lambda to crop png to tiles with that command:

Code: Select all

/opt/bin/magick convert /tmp/1571383180912/1571383180912_300dpi.png -monitor -crop 512x512 -set filename:tile %[fx:page.x/512]_%[fx:page.y/512] +repage +adjoin "/tmp/1571383180912/1571383180912_300dpi/tile-300-%[filename:tile].png"
Everything is working so far, I just have issues with huge png (I have a 8mo file, 26 000*10 000 pixels)
I am getting the following error :

Code: Select all

convert: unable to write pixel cache '/tmp/magick-24phDuETFQt7OM': No space left on device @ error/cache.c/WritePixelCachePixels/5823.
convert: no images defined `/tmp/1571383180912/1571383180912_300dpi/tile-300-%[filename:tile].png' @ error/convert.c/ConvertImageCommand/3300.
I pretty much already know why : Lambda have only 512mb space available on /tmp file, and /tmp is the only FS we get the access to.

My question is, can I remove the disk pixel cache and only use memory cache ? (I can get up to 3gb)
Time is not really an issue, I just want it to get done.

Thank you for your help.

User avatar
magick
Site Admin
Posts: 11077
Joined: 2003-05-31T11:32:55-07:00

Re: AWS Lambda with IM7

Post by magick » 2019-10-18T03:54:37-07:00


malinlesinge
Posts: 6
Joined: 2019-10-18T01:29:25-07:00
Authentication code: 1152

Re: AWS Lambda with IM7

Post by malinlesinge » 2019-10-18T09:06:21-07:00

Thanks for the link !
I'm trying to implement the distribute cache, but I got an error and I don't really know how to fix it :

Code: Select all

convert: DistributedPixelCache '52.208.221.223' @ error/distribute-cache.c/ConnectPixelCacheServer/243.
convert: cache resources exhausted `/tmp/1571390578996/1571390578996_100dpi.png' @ error/cache.c/OpenPixelCache/3850.
convert: cache resources exhausted `/tmp/1571390578996/1571390578996_100dpi/tile-100-%[filename:tile].png' @ error/cache.c/OpenPixelCache/3850.
convert: cache resources exhausted `/tmp/1571390578996/1571390578996_100dpi/tile-100-%[filename:tile].png' @ error/cache.c/OpenPixelCache/3850.
convert: cache resources exhausted `/tmp/1571390578996/1571390578996_100dpi/tile-100-%[filename:tile].png' @ error/cache.c/OpenPixelCache/3850.
convert: memory allocation failed `/tmp/1571390578996/1571390578996_100dpi/tile-100-%[filename:tile].png' @ error/png.c/WriteOnePNGImage/9345.
convert: Wrote palette index exceeding num_palette `/tmp/1571390578996/1571390578996_100dpi/tile-100-0_0.png' @ warning/png.c/MagickPNGWarningHandler/1748.
This is my command :

Code: Select all

/opt/bin/magick convert /tmp/1571390578996/1571390578996_100dpi.png -limit memory 2000mb -limit map 200mb -limit disk 200mb -define registry:cache:hosts=52.208.221.223:6666 -monitor -crop 1024x1024 -set filename:tile %[fx:page.x/1024]_%[fx:page.y/1024] +repage +adjoin "/tmp/1571390578996/1571390578996_100dpi/tile-100-%[filename:tile].png"
Is there any way to see in my remote host logs or anything ?

Thanks

malinlesinge
Posts: 6
Joined: 2019-10-18T01:29:25-07:00
Authentication code: 1152

Re: AWS Lambda with IM7

Post by malinlesinge » 2019-10-22T05:22:31-07:00

Hi again,
Sorry for double post, I just got an update

So I found the source of my previous error, it was because the wrong policy.xml was taken, so the secret was not right.
But I got another error now :

Code: Select all

Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 0 of 1169, 00% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 50 of 1169, 04% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 100 of 1169, 08% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 150 of 1169, 12% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 200 of 1169, 17% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 250 of 1169, 21% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 300 of 1169, 25% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 350 of 1169, 29% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 400 of 1169, 34% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 450 of 1169, 38% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 500 of 1169, 42% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 550 of 1169, 47% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 600 of 1169, 51% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 650 of 1169, 55% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 700 of 1169, 59% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 750 of 1169, 64% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 800 of 1169, 68% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 850 of 1169, 72% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 900 of 1169, 77% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 950 of 1169, 81% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 1000 of 1169, 85% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 1050 of 1169, 89% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 1100 of 1169, 94% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 1150 of 1169, 98% complete
Save/Image//tmp/1571673148318/1571673148318_100dpi[tile-100-0_0.png]: 1168 of 1169, 100% complete

convert: unable to open pixel cache '52.208.221.223': No such file or directory @ error/cache.c/OpenPixelCache/3802.
convert: cache resources exhausted `/tmp/1571673148318/1571673148318_100dpi.png' @ error/cache.c/OpenPixelCache/3850.
convert: cache resources exhausted `/tmp/1571673148318/1571673148318_100dpi/tile-100-%[filename:tile].png' @ error/cache.c/OpenPixelCache/3850.
convert: memory allocation failed `/tmp/1571673148318/1571673148318_100dpi/tile-100-%[filename:tile].png' @ error/png.c/WriteOnePNGImage/9345.
convert: Wrote palette index exceeding num_palette `/tmp/1571673148318/1571673148318_100dpi/tile-100-0_0.png' @ warning/png.c/MagickPNGWarningHandler/1748.
As the logs goes, the cropping seems to be working well. It gets to 100% but then it crashes, and I got no error on the remote host. The process stays up and nothing is logged in.

Is there something I am missing ? (for the record, I am trying to crop a really small image which should be fine)

User avatar
magick
Site Admin
Posts: 11077
Joined: 2003-05-31T11:32:55-07:00

Re: AWS Lambda with IM7

Post by magick » 2019-10-22T05:50:14-07:00

You may be running out of space on your /tmp folder. Try pointing the pixel cache to an alternative location with plenty of free space, e.g. -define registry:temporary-path=/data/tmp.

malinlesinge
Posts: 6
Joined: 2019-10-18T01:29:25-07:00
Authentication code: 1152

Re: AWS Lambda with IM7

Post by malinlesinge » 2019-10-22T06:50:56-07:00

The /tmp on my remote host has 6gb of free space, that should be enough.
I just have limited memory, only 300mb but I'm setting a really low memory limit so it caches to disk anyway.

With the -debug 'All' option I managed to get more informations (which are not really helpfull though):

Code: Select all

magick convert -limit memory 10mb -limit map 5000mb -limit disk 5000mb -limit width 1000MP -limit height 1000MP -limit area 1000MP -debug "All" -distribute-cache 6666
2019-10-22T13:45:42+00:00 0:02.874 0.000u 7.0.8 Exception convert[3661]: cache.c/OpenPixelCache/3671/Exception
  WidthOrHeightExceedsLimit `'
2019-10-22T13:45:42+00:00 0:02.938 0.000u 7.0.8 Exception convert[3661]: cache.c/OpenPixelCache/3671/Exception
  WidthOrHeightExceedsLimit `'
FYI, when I try the convert command from my Lambda to the host remote pixel-cache directly, and with the same file, it's working fine :

Code: Select all

magick convert /home/ec2-user/1571750340659_300dpi.png -monitor -crop 512x512 -set filename:tile %[fx:page.x/512]_%[fx:page.y/512] +repage +adjoin "/tmp/tile-300-%[filename:tile].png"
Mogrify/Image//home/ec2-user[1571750340659_300dpi.png]: 34 of 35, 100% complete
Write/Image//tmp[tile-300-%[filename:tile].png]: 34 of 35, 100% complete
[ec2-user@ip-172-31-26-177 ~]$ ls /tmp/
tile-300-0_4.png                                                         tile-300-1_4.png  tile-300-2_4.png  tile-300-3_4.png  tile-300-4_4.png  tile-300-5_4.png  tile-300-6_4.png
tile-300-0_0.png                                                         tile-300-1_0.png  tile-300-2_0.png  tile-300-3_0.png  tile-300-4_0.png  tile-300-5_0.png  tile-300-6_0.png
tile-300-0_1.png                                                         tile-300-1_1.png  tile-300-2_1.png  tile-300-3_1.png  tile-300-4_1.png  tile-300-5_1.png  tile-300-6_1.png
tile-300-0_2.png                                                         tile-300-1_2.png  tile-300-2_2.png  tile-300-3_2.png  tile-300-4_2.png  tile-300-5_2.png  tile-300-6_2.png
tile-300-0_3.png                                                         tile-300-1_3.png  tile-300-2_3.png  tile-300-3_3.png  tile-300-4_3.png  tile-300-5_3.png  tile-300-6_3.png
Thank you so much for your help !

snibgo
Posts: 12301
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: AWS Lambda with IM7

Post by snibgo » 2019-10-22T07:37:55-07:00

malinlesinge wrote:WidthOrHeightExceedsLimit
I suggest you insert "-list resource" as the first option in your magick command. This might explain the limit problem.
snibgo's IM pages: im.snibgo.com

malinlesinge
Posts: 6
Joined: 2019-10-18T01:29:25-07:00
Authentication code: 1152

Re: AWS Lambda with IM7

Post by malinlesinge » 2019-10-22T08:08:13-07:00

Alright, so on my Lambda I have : (I only want to use memory, got almost no storage spage)

Code: Select all

Width: 107.374MP
Height: 107.374MP
Area: 6.61642GP
List length: unlimited
Memory: 3000000000B
Map: 0B
Disk: 0B
File: 768
Thread: 1
Throttle: 0
Time: unlimited
And on my distributed cache, I have : (I only want to use disk, got almost no memory)

Code: Select all

Width: 107.374MP
Height: 107.374MP
Area: 2.06302GP
List length: unlimited
Memory: 0B
Map: 1.92133GiB
Disk: 5242880000B
File: 768
Thread: 1
Throttle: 0
Time: unlimited
considering I am using this to setup the distributed-cache host :

Code: Select all

/magick convert -limit memory 0 -limit map 5000MB -limit disk 5000MB -limit width 1000MP -limit height 1000MP -limit area 1000MP -debug "All" -distribute-cache 6666
I don't really see anything wrong about the height or width limit, 107MP should be way enough. The file I'm testing with is something around 3000*3000x pixels

malinlesinge
Posts: 6
Joined: 2019-10-18T01:29:25-07:00
Authentication code: 1152

Re: AWS Lambda with IM7

Post by malinlesinge » 2019-11-08T01:27:41-07:00

Hey again,

Do you have any idea ? I'm still stuck on this..

Thanks

Post Reply