How to convert a very large image in Linux, using -limits or setting environment variables.

[briligg]

I am trying to convert a 22 GB tif file into a smaller downsampled version, and to convert it to png and jpg.

I don't use the terminal all that much, so far all my attempts have yielded strings of errors. Here is the most recent one:

Code: Select all

convert Lunar_LRO_LrocKaguya_DEMmerge_60N60S_512ppd.tif -limit memory 12GB -limit map 25GiB -limit disk 30GiB -resize 10% -debug cache MoonRelief.tif

Which gave me:

Code: Select all

2018-05-28T19:20:43-05:00 0:00.000 0.000u 6.9.7 Cache convert-im6.q16[316]: cache.c/DestroyPixelCache/1123/Cache
  destroy 
2018-05-28T19:20:43-05:00 0:00.000 0.000u 6.9.7 Cache convert-im6.q16[316]: cache.c/DestroyPixelCache/1123/Cache
  destroy 
2018-05-28T19:20:43-05:00 0:00.010 0.010u 6.9.7 Cache convert-im6.q16[316]: cache.c/DestroyPixelCache/1123/Cache
  destroy 
convert-im6.q16: Unknown field with tag 33550 (0x830e) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/912.
convert-im6.q16: Unknown field with tag 33922 (0x8482) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/912.
convert-im6.q16: Unknown field with tag 34735 (0x87af) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/912.
convert-im6.q16: Unknown field with tag 34736 (0x87b0) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/912.
convert-im6.q16: Unknown field with tag 34737 (0x87b1) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/912.
convert-im6.q16: Unknown field with tag 42112 (0xa480) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/912.
convert-im6.q16: Unknown field with tag 42113 (0xa481) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/912.
convert-im6.q16: width or height exceeds limit `Lunar_LRO_LrocKaguya_DEMmerge_60N60S_512ppd.tif' @ error/cache.c/OpenPixelCache/3837.
convert-im6.q16: no images defined `MoonRelief.tif' @ error/convert.c/ConvertImageCommand/3258.

I have also tried

Code: Select all

export MAGICK_DISK_LIMIT=25GiB

but then when I check with

Code: Select all

identify -list resource

the output is the same. Should I be doing that in a different directory? My understanding of the whole underlying system is pretty basic, I'm afraid.

I am running Ubuntu 18.04 and my ImageMagick version is 6.9.7-4 Q16

Any help on this is greatly appreciated.

[fmw42]

Most are just warning about meta tags that IM does not understand. The bad one is:

convert-im6.q16: width or height exceeds limit `Lunar_LRO_LrocKaguya_DEMmerge_60N60S_512ppd.tif' @ error/cache.c/OpenPixelCache/3837.

Looks like you are exceeding the width or height limitations of either IM or your system or libtif. See policy.xml for those limits. Your hosting provider if you have one may have set small dimension limits in IM and you will not be able to raise those limit. You would have to check with your provider.

You can check with:

Code: Select all

convert -list resource

For example on my Mac, I have:

Code: Select all

Resource limits:
  Width: 214.7MP
  Height: 214.7MP
  List length: 18.446744EP
  Area: 17.18GP
  Memory: 8GiB
  Map: 16GiB
  Disk: unlimited
  File: 192
  Thread: 4
  Throttle: 0
  Time: unlimited

Perhaps you need to install bigTiff.

[briligg]

It took me a while to locate policy.xml, change its write permissions, and change those settings. Now that I have, yes indeed the resource limits have changed.

To add a bit of detail on the off chance of someone coming along later who can use them, policy.xml on Ubuntu 18.04 is in the /etc/ImageMagick-6 directory. This is what that section of the policy.xml file looks like:

Code: Select all

<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <policy domain="resource" name="memory" value="14GiB"/>
  <policy domain="resource" name="map" value="30GiB"/>
  <policy domain="resource" name="width" value="16MP"/>
  <policy domain="resource" name="height" value="16MP"/>
  <policy domain="resource" name="area" value="40GB"/>
  <policy domain="resource" name="disk" value="30GiB"/>
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <!-- <policy domain="resource" name="thread" value="4"/> -->
  <!-- <policy domain="resource" name="throttle" value="0"/> -->
  <!-- <policy domain="resource" name="time" value="3600"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->
  <!-- not needed due to the need to use explicitly by mvg: -->
  <!-- <policy domain="delegate" rights="none" pattern="MVG" /> -->
  <!-- use curl -->
  <policy domain="delegate" rights="none" pattern="URL" />
  <policy domain="delegate" rights="none" pattern="HTTPS" />
  <policy domain="delegate" rights="none" pattern="HTTP" />
  <!-- in order to avoid to get image with password text -->
  <policy domain="path" rights="none" pattern="@*"/>
  <policy domain="cache" name="shared-secret" value="passphrase" stealth="true"/>
</policymap>

I had to get write permission on the file in order to change it, with

Code: Select all

chmod 777 policy.xml

in the terminal (I was already in the directory where the file is). Then I just put in the values I wanted for memory, map, width, height, area, and disk. And then I changed the write permissions to 744, in the interest of healthy paranoia.

Thanks for your help. Now I'll see if I can process that image.

[briligg]

Alright, now that I've attempted to convert the image again, it returns a new error. I tried on the image I was trying to process yesterday, and when that failed, I tried on an image that is 8 GB. I have 16 GB of RAM, so I thought that might help. The image I was working with earlier is 22 GB.

The command I input was

Code: Select all

convert Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.tif -resize 25% LOLA-MoonHM.tif

And skipping over the warnings it returned due to metadata it didn't recognize, these are the errors returned:

Code: Select all

convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.tif' @ error/cache.c/OpenPixelCache/3982.
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.tif' @ error/cache.c/OpenPixelCache/3982.
convert-im6.q16: no images defined `LOLA-MoonHM.tif' @ error/convert.c/ConvertImageCommand/3258.

convert -list resource returns

Code: Select all

Resource limits:
  Width: 16MP
  Height: 16MP
  Area: 33.532GP
  Memory: 12GiB
  Map: 30GiB
  Disk: 30GiB
  File: 768
  Thread: 8
  Throttle: 0
  Time: unlimited

So that doesn't seem like it should be the problem. I'm doing this on my own machine, not over a network or through a service.
It was mentioned in the answer yesterday that maybe my system doesn't have bigTiff installed. I forgot to check that.

Edit - I have downloaded the libtiff 4.1 library from http://bigtiff.org/ and unzipped it into a folder on my desktop. There is a makefile. I'm working out how to build it and get it working.

[snibgo]

The size of a file in bytes on disk says nothing about memory usage. How many pixels does it have? At 8 bytes per pixel, how many bytes is that? How much free space do you have in /tmp or wherever your temporary files go?

[briligg]

The one that is 8.5 GB is 92160 x 46080 px and it's 16 bit grayscale.

So multiplying those pixels its area is 4.2 GP. I don't know what multiple to put in there to account for the 16 bits, as I don't know how many shades of gray are actually being used.

There is 75 GB of space free on my main partition. That isn't a lot, there is a lot more on my secondary partition, but I don't know that it's possible to use that. There is 16 GB of RAM.

At any rate, I thought the idea was that data is swapped back and forth in order to manage things that are too big. I thought that was the whole idea of doing things from the command line, it isn't necessary to put the whole thing into memory at once.

At the moment I'm still working out how to set up the ability to use the bigTiff library to handle these files. That has to also be a problem.

[snibgo]

You are using v6, Q16, integer not HDRI (I assume), so each pixel needs 8 bytes, so the 92160 x 46080 image needs 34 GB. Yes, if there isn't enough main memory then disk will be used instead, but you have a limit of 30 GiB of disk usage. I suspect that is the problem. (You will also need space for the resized image, of course.)

As the image is grayscale, v7 probably needs just 2 bytes per pixel.

[briligg]

Ok, increasing the disk limit to 70 GB did it. My machine is now working away at resizing the 8.5 GB file to 25%.

What came with Ubuntu 18.04 was v6. Compiling v7 for my system i suppose is an option I could try. It is again something i haven't done before. Or I can figure out what is taking up so much space on my main partition, i was quite surprised it is so full.

Thanks for your help, i have a lot more to do and it's a relief to finally be underway.