Convert does not finish

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
groovejunkk
Posts: 7
Joined: 2017-07-19T00:19:09-07:00
Authentication code: 1151

Convert does not finish

Post by groovejunkk »

Hello Guys,

I'm quite new to computer science at all so please excuse any mistakes you might see.
I wanted to build a script to unify images in bash, so I did the following:

Code: Select all

#!/bin/bash

for pic in /some/dir/*.jpg; do
        echo $pic
        convert  $pic -resize 2000x2000^ $pic
        echo "resize done"
        convert -verbose -units PixelsPerInch $pic -resample 300   $pic
done

When I run the script it starts the second convert on the first image but never finishes it.
This is what my output is, due to the verbose flag

Code: Select all

/some/dir/448528.jpg JPEG 2000x2000 2000x2000+0+0 8-bit sRGB 3.208MB 0.080u 0:00.080
I wrote the second convert to bring my images up to 300 dpi from 96 dpi.

I run ImageMagick 6.9.7-4 Q16 x86_64 20170114 on debian stable.
Any help is of course greatly appreciated.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Convert does not finish

Post by snibgo »

groovejunkk wrote:... but never finishes it.
I suppose that means it was doing work for a long time, and you killed it.

You claim the input was 96 dpi, but have you checked this? Resampling from 96 dpi to 300 dpi will increase the number of pixels by a factor approx 3 in each dimension, so from 2000x2000 to 6000x6000. That shouldn't be a problem, but if the input was 1 dpi the output would be 600000x600000 which would be a problem.
snibgo's IM pages: im.snibgo.com
groovejunkk
Posts: 7
Joined: 2017-07-19T00:19:09-07:00
Authentication code: 1151

Re: Convert does not finish

Post by groovejunkk »

Yes I have checked the dpi for the input pictures. My input pictures are approximately 1200x2400px at 96dpi or some at 72dpi which i just found out.
Never finishing in my case means I left it running for a good 45 minutes with no result. This seemed to slow on a quadcore cpu with 16gb of ram in my opinion.

I just ran identify -verbose on one of the pictures (already resized to 2000x2000) but the data seems alright to me

Code: Select all

Image: 448528.jpg
  Format: JPEG (Joint Photographic Experts Group JFIF format)
  Mime type: image/jpeg
  Class: DirectClass
  Geometry: 2000x2000+0+0
  Resolution: 72x72
  Print size: 27.7778x27.7778
  Units: PixelsPerInch
  Type: TrueColor
  Endianess: Undefined
  Colorspace: sRGB
  Depth: 8-bit
  Channel depth:
    red: 8-bit
    green: 8-bit
    blue: 8-bit
  Channel statistics:
    Pixels: 4000000
    Red:
      min: 161 (0.631373)
      max: 255 (1)
      mean: 218.122 (0.855379)
      standard deviation: 10.5832 (0.0415026)
      kurtosis: 0.0607825
      skewness: -0.0176774
      entropy: 0.835328
    Green:
      min: 151 (0.592157)
      max: 255 (1)
      mean: 208.093 (0.81605)
      standard deviation: 11.0974 (0.0435192)
      kurtosis: 0.132938
      skewness: 0.0657159
      entropy: 0.823453
    Blue:
      min: 152 (0.596078)
      max: 255 (1)
      mean: 210.699 (0.826271)
      standard deviation: 11.152 (0.0437331)
      kurtosis: 0.164899
      skewness: -0.0369316
      entropy: 0.824477
  Image statistics:
    Overall:
      min: 151 (0.592157)
      max: 255 (1)
      mean: 212.305 (0.832567)
      standard deviation: 10.9472 (0.0429301)
      kurtosis: 1.00205
      skewness: -0.0164577
      entropy: 0.827753
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Background color: white
  Border color: srgb(223,223,223)
  Matte color: grey74
  Transparent color: black
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 2000x2000+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: JPEG
  Quality: 99
  Orientation: TopLeft
  Properties:
    date:create: 2017-07-19T09:29:03+02:00
    date:modify: 2017-07-19T09:29:03+02:00
    exif:BitsPerSample: 8, 8, 8
    exif:ColorSpace: 65535
    exif:Compression: 1
    exif:Copyright: Copyright (C) reserved
    exif:DateTime: 2014:12:08 15:19:49
    exif:ExifImageLength: 1060
    exif:ExifImageWidth: 1060
    exif:ExifOffset: 344
    exif:ImageLength: 1060
    exif:ImageWidth: 1060
    exif:Model: 
    exif:Orientation: 1
    exif:PhotometricInterpretation: 2
    exif:PlanarConfiguration: 1
    exif:ResolutionUnit: 2
    exif:SamplesPerPixel: 3
    exif:Software: Adobe Photoshop CC (Macintosh)
    exif:thumbnail:Compression: 6
    exif:thumbnail:JPEGInterchangeFormat: 482
    exif:thumbnail:JPEGInterchangeFormatLength: 1392
    exif:thumbnail:ResolutionUnit: 2
    exif:thumbnail:XResolution: 72/1
    exif:thumbnail:YResolution: 72/1
    exif:XResolution: 72/1
    exif:YResolution: 72/1
    icc:copyright: Copyright (C) 2007 by Color Solutions, All Rights Reserved. License details can be found on: http://www.eci.org/eci/en/eciRGB.php
    icc:description: eciRGB v2
    icc:manufacturer: eciRGB v2
    icc:model: eciRGB v2
    jpeg:colorspace: 2
    jpeg:sampling-factor: 1x1,1x1,1x1
    signature: 3d8fcd1bd2bf9b913bccddad2b6472d5c1ff94ee39613f0313d87fd4e6bafd77
  Profiles:
    Profile-8bim: 4098 bytes
    Profile-exif: 1880 bytes
    Profile-icc: 1992 bytes
    Profile-iptc: 50 bytes
      City[1,90]: 0x00000000: 254700                                        -%
      City[1,90]: 0x00000000: 254700                                        -%
      unknown[2,0]: 
      Copyright String[2,116]: Copyright (C) reserved
    Profile-xmp: 5054 bytes
  Artifacts:
    filename: 448528.jpg
    verbose: true
  Tainted: False
  Filesize: 3.208MB
  Number pixels: 4M
  Pixels per second: 40MB
  User time: 0.110u
  Elapsed time: 0:01.099
  Version: ImageMagick 6.9.7-4 Q16 x86_64 20170114 http://www.imagemagick.org
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Convert does not finish

Post by snibgo »

I would expect a handful of seconds.

What is it doing in that time? Do you have temporary files in /tmp? What does "convert -list resource" say? On my machine, this is:

Code: Select all

Resource limits:
  Width: 214.7MP
  Height: 214.7MP
  Area: 8.4474GP
  Memory: 3.9336GiB
  Map: 7.8672GiB
  Disk: unlimited
  File: 1536
  Thread: 8
  Throttle: 0
  Time: unlimited
snibgo's IM pages: im.snibgo.com
groovejunkk
Posts: 7
Joined: 2017-07-19T00:19:09-07:00
Authentication code: 1151

Re: Convert does not finish

Post by groovejunkk »

Code: Select all

Resource limits:
  Width: 20KP
  Height: 20KP
  Area: 512MP
  Memory: 6GiB
  Map: 512MiB
  Disk: 10GiB
  File: 49152
  Thread: 4
  Throttle: 0
  Time: unlimited
  
I suppose the width and height parameters are a little small but still extensive enough for 6000x6000.
The map and area parameters seem a little far off to me. As I understood the map determines the size of the pixel cache in memory, so a value like this shouldn't be a problem with pictures between 3 and 4 MB. Still I am uncertain about the area value.

I found that in the policy.xml there is

Code: Select all

<policy domain="resource" name="area" value="512MB"/>
 
But in -identify resource it says MP instead of MB.


When the command runs I get a temp file named magick-126892JjPX_qof01W, but I'm not sure how to investigate the file further.
groovejunkk
Posts: 7
Joined: 2017-07-19T00:19:09-07:00
Authentication code: 1151

Re: Convert does not finish

Post by groovejunkk »

I just had a quick look at the process in htop and it shows 0% CPU and 0% memory load for the convert process.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Convert does not finish

Post by snibgo »

groovejunkk wrote:... a temp file named magick-126892JjPX_qof01W
That's probably a pixel cache, on disk because IM can't get enough real memory. Your limits look okay, so something else is limiting your process.
How large is it, in bytes? If your IM is Q16 integer, divide bytes by 8 to get pixels.
snibgo's IM pages: im.snibgo.com
groovejunkk
Posts: 7
Joined: 2017-07-19T00:19:09-07:00
Authentication code: 1151

Re: Convert does not finish

Post by groovejunkk »

Code: Select all

du -b 448528.jpg
23808547        448528.jpg
So the first Image has 23808547 bytes
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Convert does not finish

Post by snibgo »

Divide by 8 for the number of pixels, and that's approx 1200*2400 pixels. So IM is asking for memory for this pixel cache, and your OS is declining the request, so IM is using disk.

What OS sysadmin tools should you use to diagnose and fix this? Sorry, I don't know.
snibgo's IM pages: im.snibgo.com
groovejunkk
Posts: 7
Joined: 2017-07-19T00:19:09-07:00
Authentication code: 1151

Re: Convert does not finish

Post by groovejunkk »

But shouldn't disk space also be somewhat reasonably fast? I've got a pretty fast SSD in my machine..
Anyway: Thank you for your time and help I think I know some people to ask when it comes to debian sysadmin tools :-)
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Convert does not finish

Post by snibgo »

Yes, I think your process isn't just slow, but stopped. I don't know why, only that IM is asking for resources it isn't getting. My last serious work on Unix was 22 years ago, so I don't know where to begin looking. Sorry.
snibgo's IM pages: im.snibgo.com
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Convert does not finish

Post by fmw42 »

Both these commands work fine for me on IM 6.9.9.0 Q16 Mac OSX.

Code: Select all

for pic in /Users/fred/desktop/test/*.jpg; do
echo $pic
convert $pic -resize 2000x2000^ $pic
echo "resize done"
convert -verbose $pic -units PixelsPerInch -resample 300 $pic
done

Code: Select all

for pic in /Users/fred/desktop/test/*.jpg; do
echo $pic
convert $pic -resize 2000x2000^ -verbose -units PixelsPerInch -resample 300 $pic
echo "done"
done
Except that the -resample did nothing. See below regarding your input files having no units or density specified.


Have you tried just copying and pasting the code into a terminal window rather than calling it as a script?

What dimensions are your input images?

Do you really want to resample the images or do you just want to have the resized results assigned 300 dpi. If the latter then just do

Code: Select all

convert $pic -resize 2000x2000^ -units PixelsPerInch -density 300 $pic
Another possible issue is that your input images have no units define and possibly no density or density=1. That may cause an issue on your version of IM. You can check by identify -verbose input.jpg. So the proper way to do your command would be to make sure you have assigned a density and units to your input images or the results of your resized images. Nominal density is 72. But if your input files have no units, then the density that it might have is useless or possibly it gets assigned a density of 1.

Code: Select all

for pic in /Users/fred/desktop/test/*.jpg; do
echo $pic
convert $pic -resize 2000x2000^ -units PixelsPerInch -density 72 $pic
echo "resize done"
convert -verbose $pic -units PixelsPerInch -resample 300 $pic
done
Now the -resample makes the output images much larger and cases the script to run much slower. Output sizes are >10,000 due to going from 72 dpi to 300 dpi. That is why I asked if you really wanted the resample and not just setting the output density.
groovejunkk
Posts: 7
Joined: 2017-07-19T00:19:09-07:00
Authentication code: 1151

Re: Convert does not finish

Post by groovejunkk »

After some intense reading about images i learned, that I indeed, did not want to resample the Images.
Anyway I want to thank you for your kind help and have to say I appreciate this community and will try to become part of it.
Post Reply