Page 1 of 3

resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-26T12:50:44-07:00
by inMotion
I lost tons of work due a hard drive crash - among them several gigapixel images created for a client that I need back. They are stored on gigapan.com but there is no way to download them. I found this script, which lead me to imagemagick. I have downloaded the 256x256 images that make up the gigapixel, but I am having problems merging them with IM. This is what is going on:

D:\gigapan\Jerusalem_Gigapan\Python27\149616>montage -depth 8 -geometry 256x256+ 0+0 -mode concatenate -tile 104x *.jpg stitchX.tiff
montage.exe: UnableToOpenPixelCache `stitchX.tiff': Too many open files @ error/
cache.c/OpenPixelCache/4030.

The total image is 6.33 Gigapixels that is currently in 96,729 256x256px .jpgs. I have 32gb RAM and 120gb HDD space. I'm using ImageMagick-6.9.8-4-Q16-x64-static. Am I doing anything wrong?

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-26T13:02:06-07:00
by fmw42
You may be running out of resources or your JPG file may have limits on the width.

What do you get from

convert -list resource

If it does not show large enough resources to match your RAM and disk, then you should increase your limits by editing the policy.xml file.

I am not sure if JPG has a width limit.

Will your total file size for both the input images and the final output image fit into your RAM? If not, then IM will try to use disk space. So do you have enough free disk space?

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-26T17:50:50-07:00
by magick
Your command works without complaint for us. We're using ImageMagick 6.8.10-8, the current release. We saved all the image file names to a file "filelist" to reduce the command line arguments size, then ran this command:

Code: Select all

$ montage -define registry:temporary-path=/data/tmp -limit memory 16mb -verbose -depth 8 \
  -geometry 256x256+0+0 -mode concatenate -tile 104x @filelist tiff64:stitchX.tiff

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-26T19:18:08-07:00
by inMotion
Hmm. would limiting it to just 16mb exponentially increase the time it takes to process? it already takes hours. why not like 20GB?

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-27T04:08:41-07:00
by magick
Sure, use 20GB. We wanted to post a command that is known to work as a baseline.

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-27T13:07:42-07:00
by inMotion
D:\gigapan\Jerusalem_Gigapan\Python27\143766>montage -define registry:temporary-
path=data/tmp limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode
concatenate -tile 107x @filelist tiff64:stitchX.tiff
montage.exe: unable to open image `limit': No such file or directory @ error/blo
b.c/OpenBlob/2701.
montage.exe: no decode delegate for this image format `' @ error/constitute.c/Re
adImage/504.
montage.exe: unable to open image `memory': No such file or directory @ error/bl
ob.c/OpenBlob/2701.
montage.exe: no decode delegate for this image format `' @ error/constitute.c/Re
adImage/504.
montage.exe: unable to open image `8000mb': No such file or directory @ error/bl
ob.c/OpenBlob/2701.
montage.exe: no decode delegate for this image format `' @ error/constitute.c/Re
adImage/504.
montage.exe: unable to open image `@filelist': No such file or directory @ error
/blob.c/OpenBlob/2701.
montage.exe: no decode delegate for this image format `' @ error/constitute.c/Re
adImage/504.
montage.exe: missing an image filename `tiff64:stitchX.tiff' @ error/montage.c/M
ontageImageCommand/1793.

Am feel like I'm making an obvious and simple mistake....

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-27T13:09:21-07:00
by inMotion
magick wrote: 2018-07-26T17:50:50-07:00 Your command works without complaint for us. We're using ImageMagick 6.8.10-9, the current release. We saved all the image file names to a file "filelist" to reduce the command line arguments size, then ran this command:

Code: Select all

$ montage -define registry:temporary-path=/data/tmp -limit memory 16mb -verbose -depth 8 \
  -geometry 256x256+0+0 -mode concatenate -tile 104x @filelist tiff64:stitchX.tiff
Isn't 7.0.8-8-Q16-x64 the current/latest release?

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-27T13:22:35-07:00
by fmw42
6.8.10.9 is the current beta. 6.8.10.8 and 7.0.8.8 are the current releases.

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-27T14:07:51-07:00
by snibgo
inMotion wrote:...montage -define registry:temporary-path=data/tmp limit memory ...
"-limit" is the option you want, with a "-" at the front. As you have no "-", IM thinks you want to read an image called "limit", and it can't find it.

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-28T15:24:20-07:00
by inMotion
Well, I installed ImageMagick-6.9.9-37-Q16-HDRI-x64-dll

Tried this:
D:\gigapan\Jerusalem_Gigapan\Python27\143766>montage -define registry:temporary-path=data/tmp -limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode concatenate -tile 127x @filelist tiff64:stitchX.tiff

got this:
montage: unable to open image `@filelist': No such file or directory @ error/blob.c/OpenBlob/2761.montage: no decode delegate for this image format `' @ error/constitute.c/ReadImage/504.montage: missing an image filename `tiff64:stitchX.tiff' @ error/montage.c/MontageImageCommand/1795.

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-28T15:31:15-07:00
by fmw42
Try enclosing @filelist in double quotes as "@filelist.txt" with a suffix on the actual file list.

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-30T13:54:47-07:00
by inMotion
That gives me this:

montage: unable to open image `@filelist.txt': No such file or directory @ error/blob.c/OpenBlob/2761.montage: missing an image filename `tiff64:stitchX.tiff' @ error/montage.c/MontageImageCommand/1795.

Thank you so much for your help!

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-07-30T15:06:32-07:00
by fmw42
Your actual file needs to be named filelist.txt. Perhaps you should post a link to your txt file and we can test with it.

Also try putting the @filelist.txt right after montage.

If on IM 7, then use magick montage and not montage.

Another issue might be that your resources have been restricted to prevent the use of @. See your policy.xml file. https://www.imagemagick.org/script/resources.php

Is this line commented out or not?

<!-- <policy domain="path" rights="none" pattern="@*" /> -->

If uncommented, then you are restricted from using it.

If that is the case or even not, then try

Code: Select all

cat path\to\filelist.txt | montage - -define registry:temporary-path=data/tmp -limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode concatenate -tile 127x tiff64:stitchX.tiff

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-08-01T10:58:33-07:00
by inMotion
Well...i think we're getting somewhere. I don't know what you mean by filelist.txt, so i used *.jpg and it seemed to start to work. I had 150GB free when I started this. Do I just need to free up space or is there another issue with the cache I should know about?

This is what I used:

Code: Select all

montage -define registry:temporary-path=/data/tmp -limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode concatenate -tile 127x *.jpg tiff64:stitchX.tiff
And after many hours, I got this:

tiff64:stitchX.tiff=>stitchX.tiff JPEG 32512x204032 8-bit sRGB 76.453u 1:18.385
montage: UnableToExtendCache `tiff64:stitchX.tiff': No space left on device @ er
ror/cache.c/OpenPixelCache/4101.

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Posted: 2018-08-01T22:17:00-07:00
by inMotion
After 8 hours of processing, now i'm getting this error: https://www.dropbox.com/s/ix69yh5qq3aoh ... 1.jpg?dl=0

A ton of these:
montage: unable to open file `magick-16188shKu7TTUf-rE': Too many open files @ error/cache.c/WritePixelCachePixels/5942.
then this:
montage: too many exceptions (exception processing is suspended) @ error/exception.c/ThrowException/969.