Convert big PSD (1.7GB) to JPG hangs

Post any defects you find in the released or beta versions of the ImageMagick software here. Include the ImageMagick version, OS, and any command-line required to reproduce the problem. Got a patch for a bug? Post it here.
Post Reply
Swoops
Posts: 11
Joined: 2017-01-19T05:13:30-07:00
Authentication code: 1151

Convert big PSD (1.7GB) to JPG hangs

Post by Swoops »

Hello,

Whenever I execute one of the bad commands it looks like it gets stuck in some form of infinite loop. When running 'top', this is the end result:

PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
159 135 root D 20044m 59% 2 0% convert ./huge-document.psd -flatten ./huge-document.jpg
143 135 root T 20044m 59% 2 0% convert ./huge-document.psd ./huge-document.jpg

Without the -flatten option and using the preview layer (the good command) all is fine.
Any ideas would be helpful :).

Version: 7.0.4-4
Environment: Alpine 3.5 in Docker 1.12
Machine: AWS EC2 c4.4xlarge
Memory: 30GB
Storage: AWS EFS
Cores: 16
Settings:
----------> MAGICK_AREA_LIMIT: 8GiB
----------> MAGICK_HEIGHT_LIMIT: 300MP
----------> MAGICK_MAP_LIMIT: 16GiB
----------> MAGICK_MEMORY_LIMIT: 4GiB
----------> MAGICK_TEMPORARY_PATH: /tmp
----------> MAGICK_THREADS_LIMIT: 6
----------> MAGICK_TMPDIR: /tmp
----------> MAGICK_WIDTH_LIMIT: 300MP
----------> OMP_NUM_THREADS: 6

Download link: http://s3-eu-west-1.amazonaws.com/image ... cument.psd

Bad command 1: convert ./huge-document.psd ./huge-document.jpg
Bad command 2: convert ./huge-document.psd -flatten ./huge-document.jpg
Good command: convert ./huge-document.psd[0] ./huge-document.jpg

Configuration:
Path: /usr/local/lib/ImageMagick-7.0.4//config-Q16HDRI/configure.xml

Name Value
-------------------------------------------------------------------------------
CC gcc
CFLAGS -I/usr/include/libxml2 -I/usr/include/cairo -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/librsvg-2.0 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/gdk-pixbuf-2.0 -pthread -I/usr/include/libpng16 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/harfbuzz -I/usr/include/OpenEXR -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/freetype2 -I/usr/include/libpng16 -fopenmp -Wno-deprecated-declarations -Wno-error=unused-variable -Wall -mtune=core2 -fexceptions -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16
CODER_PATH /usr/local/lib/ImageMagick-7.0.4/modules-Q16HDRI/coders
CONFIGURE ./configure '--with-gslib=yes' '--with-x=yes' '--with-rsvg=yes' '--without-magick-plus-plus' 'CC=gcc' 'CFLAGS=-Wno-deprecated-declarations -Wno-error=unused-variable'
CONFIGURE_PATH /usr/local/etc/ImageMagick-7/
COPYRIGHT Copyright (C) 1999-2016 ImageMagick Studio LLC
CPPFLAGS -I/usr/local/include/ImageMagick-7
CXX g++
CXXFLAGS -g -O2
DEFS -DHAVE_CONFIG_H
DELEGATES bzlib mpeg fftw fontconfig freetype gslib jng jpeg openexr pango png ps rsvg tiff webp wmf x xml zlib
DISTCHECK_CONFIG_FLAGS 'CC=gcc' 'CFLAGS=-Wno-deprecated-declarations -Wno-error=unused-variable' --disable-deprecated --with-quantum-depth=16 --with-jemalloc=no --with-umem=no --with-autotrace=no --with-fontpath= --with-perl=no
DOCUMENTATION_PATH /usr/local/share/doc/ImageMagick-7
EXEC-PREFIX /usr/local
EXECUTABLE_PATH /usr/local/bin
FEATURES DPC HDRI Cipher OpenMP
FILTER_PATH /usr/local/lib/ImageMagick-7.0.4/modules-Q16HDRI/filters
GIT_REVISION 11509
HOST x86_64-unknown-linux-gnu
INCLUDE_PATH /usr/local/include/ImageMagick-7
LDFLAGS -L/usr/local/lib
LIB_VERSION 0x704
LIB_VERSION_NUMBER 7,0,4,4
LIBRARY_PATH /usr/local/lib/ImageMagick-7.0.4
LIBS -ltiff -lfreetype -ljpeg -lgs -lpng16 -lfftw3 -lfontconfig -lfreetype -lwebp -lwmflite -lXext -lX11 -lbz2 -lIlmImf -lImath -lHalf -lIex -lIexMath -lIlmThread -lpangocairo-1.0 -lpango-1.0 -lgobject-2.0 -lglib-2.0 -lintl -lcairo -lrsvg-2 -lm -lgio-2.0 -lgdk_pixbuf-2.0 -lgobject-2.0 -lglib-2.0 -lintl -lcairo -lxml2 -L/lib -lz -lm -lgomp
NAME ImageMagick
PCFLAGS -fopenmp -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16
PREFIX /usr/local
QuantumDepth 16
RELEASE_DATE 2017-01-16
SHARE_PATH /usr/local/share/ImageMagick-7
SHAREARCH_PATH /usr/local/lib/ImageMagick-7.0.4/config-Q16HDRI
TARGET_CPU x86_64
TARGET_OS linux-gnu
TARGET_VENDOR unknown
VERSION 7.0.4
WEBSITE http://www.imagemagick.org

Path: [built-in]

Name Value
-------------------------------------------------------------------------------
FEATURES OpenMP
NAME ImageMagick
QuantumDepth 16
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Convert big PSD to JPG hangs

Post by snibgo »

The PSD is too large for me to download. How many pixels does it have? How much real memory do you have? Perhaps the commands that read all layers need more memory than you have, so IM will store the pixels on disk, which takes forever, giving the appearance of an infinite loop.
snibgo's IM pages: im.snibgo.com
Swoops
Posts: 11
Joined: 2017-01-19T05:13:30-07:00
Authentication code: 1151

Re: Convert big PSD (1.7GB) to JPG hangs

Post by Swoops »

The resolution is 10115 × 20000. The moment everything is started and we are about to run the convert it has 28GB of memory available. I am pretty sure it is not just taking a long time, as we let it run for days actually. Also its not writing anything to disk the moment it hangs and uses no CPU at all.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Convert big PSD (1.7GB) to JPG hangs

Post by snibgo »

You have 28 GB, but various limits in place. You might try relaxing those limits.

Q16 v6 takes 8 bytes/pixel. (I'm not sure about v7.) So that is 1.6 GB per layer. How many layers?
snibgo's IM pages: im.snibgo.com
Swoops
Posts: 11
Joined: 2017-01-19T05:13:30-07:00
Authentication code: 1151

Re: Convert big PSD (1.7GB) to JPG hangs

Post by Swoops »

There are 7 layers. The reason we put these limits in place as we don't want the server to crash, as the server is used as a rendition generator and most likely have to deal with multiple conversions at the same time. Will try to do the same task with 1952 GB of memory and 128 cores (AWS EC2 x1.32xlarge), without any limits in magick config :).
Swoops
Posts: 11
Joined: 2017-01-19T05:13:30-07:00
Authentication code: 1151

Re: Convert big PSD (1.7GB) to JPG hangs

Post by Swoops »

Ok, it seems like you are right. Now it did finish the conversion in 10 minutes. Sorry for reporting it as a bug. Will try to tweak to the optimal setting.
Last edited by Swoops on 2017-01-19T08:26:53-07:00, edited 1 time in total.
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: Convert big PSD (1.7GB) to JPG hangs

Post by magick »

Large images force the pixel cache on disk rather than memory. Since I/O is a thousand times slower than memory, conversion of the image is expected to be slow but still complete. You can extract the composite image with this command (bypassing the layers):

Code: Select all

convert -limit area 1GB -debug cache './huge-document.psd[0]' huge-document.jpg
We converted the image on a SSD and the process completed in one minute. On a SATA disk, it took 5 minutes to complete.
Swoops
Posts: 11
Joined: 2017-01-19T05:13:30-07:00
Authentication code: 1151

Re: Convert big PSD (1.7GB) to JPG hangs

Post by Swoops »

Hi magick, thanks for giving it a try. For that command it also works perfect for us. However does it also finish so well for you when you run either of these?:

Bad command 1: convert ./huge-document.psd ./huge-document.jpg
Bad command 2: convert ./huge-document.psd -flatten ./huge-document.jpg

With these it never completes. Not even after days with the system as described in first post.
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: Convert big PSD (1.7GB) to JPG hangs

Post by magick »

On an SSD, the flatten command ran in 5 and 1/2 minutes:

Code: Select all

convert -limit area 1GB ./huge-document.psd -flatten huge-document.jpg
We're using 7.0.4-4 with HDRI-enabled on a Linux desktop.

You can track progress of your command by adding '-debug cache' to your command line.
Swoops
Posts: 11
Joined: 2017-01-19T05:13:30-07:00
Authentication code: 1151

Re: Convert big PSD (1.7GB) to JPG hangs

Post by Swoops »

Ok, I got the bad command completed in 50 seconds, by disabling all disk cache and pure heap memory cache. The heap memory cache is 4GiB, as before. Writing to disk, especially AWS EFS, was the root cause of this. Thanks Magick for pointing me to the debug option, that really helped.
Post Reply