resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
inMotion
Posts: 16
Joined: 2018-07-26T12:37:42-07:00
Authentication code: 1152

resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by inMotion »

I lost tons of work due a hard drive crash - among them several gigapixel images created for a client that I need back. They are stored on gigapan.com but there is no way to download them. I found this script, which lead me to imagemagick. I have downloaded the 256x256 images that make up the gigapixel, but I am having problems merging them with IM. This is what is going on:

D:\gigapan\Jerusalem_Gigapan\Python27\149616>montage -depth 8 -geometry 256x256+ 0+0 -mode concatenate -tile 104x *.jpg stitchX.tiff
montage.exe: UnableToOpenPixelCache `stitchX.tiff': Too many open files @ error/
cache.c/OpenPixelCache/4030.

The total image is 6.33 Gigapixels that is currently in 96,729 256x256px .jpgs. I have 32gb RAM and 120gb HDD space. I'm using ImageMagick-6.9.8-4-Q16-x64-static. Am I doing anything wrong?
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by fmw42 »

You may be running out of resources or your JPG file may have limits on the width.

What do you get from

convert -list resource

If it does not show large enough resources to match your RAM and disk, then you should increase your limits by editing the policy.xml file.

I am not sure if JPG has a width limit.

Will your total file size for both the input images and the final output image fit into your RAM? If not, then IM will try to use disk space. So do you have enough free disk space?
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by magick »

Your command works without complaint for us. We're using ImageMagick 6.8.10-8, the current release. We saved all the image file names to a file "filelist" to reduce the command line arguments size, then ran this command:

Code: Select all

$ montage -define registry:temporary-path=/data/tmp -limit memory 16mb -verbose -depth 8 \
  -geometry 256x256+0+0 -mode concatenate -tile 104x @filelist tiff64:stitchX.tiff
inMotion
Posts: 16
Joined: 2018-07-26T12:37:42-07:00
Authentication code: 1152

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by inMotion »

Hmm. would limiting it to just 16mb exponentially increase the time it takes to process? it already takes hours. why not like 20GB?
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by magick »

Sure, use 20GB. We wanted to post a command that is known to work as a baseline.
inMotion
Posts: 16
Joined: 2018-07-26T12:37:42-07:00
Authentication code: 1152

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by inMotion »

D:\gigapan\Jerusalem_Gigapan\Python27\143766>montage -define registry:temporary-
path=data/tmp limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode
concatenate -tile 107x @filelist tiff64:stitchX.tiff
montage.exe: unable to open image `limit': No such file or directory @ error/blo
b.c/OpenBlob/2701.
montage.exe: no decode delegate for this image format `' @ error/constitute.c/Re
adImage/504.
montage.exe: unable to open image `memory': No such file or directory @ error/bl
ob.c/OpenBlob/2701.
montage.exe: no decode delegate for this image format `' @ error/constitute.c/Re
adImage/504.
montage.exe: unable to open image `8000mb': No such file or directory @ error/bl
ob.c/OpenBlob/2701.
montage.exe: no decode delegate for this image format `' @ error/constitute.c/Re
adImage/504.
montage.exe: unable to open image `@filelist': No such file or directory @ error
/blob.c/OpenBlob/2701.
montage.exe: no decode delegate for this image format `' @ error/constitute.c/Re
adImage/504.
montage.exe: missing an image filename `tiff64:stitchX.tiff' @ error/montage.c/M
ontageImageCommand/1793.

Am feel like I'm making an obvious and simple mistake....
inMotion
Posts: 16
Joined: 2018-07-26T12:37:42-07:00
Authentication code: 1152

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by inMotion »

magick wrote: 2018-07-26T17:50:50-07:00 Your command works without complaint for us. We're using ImageMagick 6.8.10-9, the current release. We saved all the image file names to a file "filelist" to reduce the command line arguments size, then ran this command:

Code: Select all

$ montage -define registry:temporary-path=/data/tmp -limit memory 16mb -verbose -depth 8 \
  -geometry 256x256+0+0 -mode concatenate -tile 104x @filelist tiff64:stitchX.tiff
Isn't 7.0.8-8-Q16-x64 the current/latest release?
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by fmw42 »

6.8.10.9 is the current beta. 6.8.10.8 and 7.0.8.8 are the current releases.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by snibgo »

inMotion wrote:...montage -define registry:temporary-path=data/tmp limit memory ...
"-limit" is the option you want, with a "-" at the front. As you have no "-", IM thinks you want to read an image called "limit", and it can't find it.
snibgo's IM pages: im.snibgo.com
inMotion
Posts: 16
Joined: 2018-07-26T12:37:42-07:00
Authentication code: 1152

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by inMotion »

Well, I installed ImageMagick-6.9.9-37-Q16-HDRI-x64-dll

Tried this:
D:\gigapan\Jerusalem_Gigapan\Python27\143766>montage -define registry:temporary-path=data/tmp -limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode concatenate -tile 127x @filelist tiff64:stitchX.tiff

got this:
montage: unable to open image `@filelist': No such file or directory @ error/blob.c/OpenBlob/2761.montage: no decode delegate for this image format `' @ error/constitute.c/ReadImage/504.montage: missing an image filename `tiff64:stitchX.tiff' @ error/montage.c/MontageImageCommand/1795.
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by fmw42 »

Try enclosing @filelist in double quotes as "@filelist.txt" with a suffix on the actual file list.
inMotion
Posts: 16
Joined: 2018-07-26T12:37:42-07:00
Authentication code: 1152

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by inMotion »

That gives me this:

montage: unable to open image `@filelist.txt': No such file or directory @ error/blob.c/OpenBlob/2761.montage: missing an image filename `tiff64:stitchX.tiff' @ error/montage.c/MontageImageCommand/1795.

Thank you so much for your help!
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by fmw42 »

Your actual file needs to be named filelist.txt. Perhaps you should post a link to your txt file and we can test with it.

Also try putting the @filelist.txt right after montage.

If on IM 7, then use magick montage and not montage.

Another issue might be that your resources have been restricted to prevent the use of @. See your policy.xml file. https://www.imagemagick.org/script/resources.php

Is this line commented out or not?

<!-- <policy domain="path" rights="none" pattern="@*" /> -->

If uncommented, then you are restricted from using it.

If that is the case or even not, then try

Code: Select all

cat path\to\filelist.txt | montage - -define registry:temporary-path=data/tmp -limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode concatenate -tile 127x tiff64:stitchX.tiff
inMotion
Posts: 16
Joined: 2018-07-26T12:37:42-07:00
Authentication code: 1152

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by inMotion »

Well...i think we're getting somewhere. I don't know what you mean by filelist.txt, so i used *.jpg and it seemed to start to work. I had 150GB free when I started this. Do I just need to free up space or is there another issue with the cache I should know about?

This is what I used:

Code: Select all

montage -define registry:temporary-path=/data/tmp -limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode concatenate -tile 127x *.jpg tiff64:stitchX.tiff
And after many hours, I got this:

tiff64:stitchX.tiff=>stitchX.tiff JPEG 32512x204032 8-bit sRGB 76.453u 1:18.385
montage: UnableToExtendCache `tiff64:stitchX.tiff': No space left on device @ er
ror/cache.c/OpenPixelCache/4101.
inMotion
Posts: 16
Joined: 2018-07-26T12:37:42-07:00
Authentication code: 1152

Re: resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

Post by inMotion »

After 8 hours of processing, now i'm getting this error: https://www.dropbox.com/s/ix69yh5qq3aoh ... 1.jpg?dl=0

A ton of these:
montage: unable to open file `magick-16188shKu7TTUf-rE': Too many open files @ error/cache.c/WritePixelCachePixels/5942.
then this:
montage: too many exceptions (exception processing is suspended) @ error/exception.c/ThrowException/969.
Post Reply