Page 1 of 1

Out of memory when converting multipage Tiff

Posted: 2016-07-06T02:49:50-07:00
by DP-LT
Hi,

I'm having a problem with merging and converting multi page Tiffs. I am using the following code to merge multiple multipage Tiffs and apply adaptive thresholding to them:

Code: Select all

public void applyAdaptiveThresholdingAndMergeTiffs(List<string> inputFiles, string outputFile)
        {
            using (MagickImageCollection images = new MagickImageCollection()) {
                foreach (string inputFile in inputFiles) images.AddRange(inputFile);
                for (int c = 0; c < images.Count; c++) images[c].AdaptiveThreshold(25, 25, -10);
                images[0].CompressionMethod = CompressionMethod.Group4;
                images.Write(outputFile);
            }
        }
It works fine for Tiffs with a small amount of pages, however when converting Tiffs with 50 pages or more images.Write() throws the following exception:

Memory allocation failed `C:\ImageMagickTest\ImageMagickOutput\merged.tiff' @ error/quantize.c/QuantizeImage/2705
at ImageMagick.MagickExceptionHelper.Check(IntPtr exception)
at ImageMagick.NativeHelper.CheckException(IntPtr exception)
at ImageMagick.MagickImageCollection.NativeMagickImageCollection.WriteFile(MagickImage image, MagickSettings settings)
at ImageMagick.MagickImageCollection.Write(String fileName)
at ImageMagickTest.ImageMagickController.applyAdaptiveThresholdingAndMergeTiffs(List`1 inputFiles, String outputFile)

When images.Write is called memory usage starts to drop, the exception is thrown when memory usage is only 120MB:

Image

I have tried using

Code: Select all

ResourceLimits.Memory = 900000000; //900000000 bytes or 900MB, just to be on the safe side of the ~1300MB limit of 32-bit .NET

to limit memory usage, without succes.

I have created a test solution to simulate and recreate the problem that occcurs in my main application. To mimic the memory overhead of my main application I use an empty byte array of 100MB in the test.

Am I doing something wrong in my code, or is it not possible to convert multipage Tiffs with 50 pages or more?

Re: Out of memory when converting multipage Tiff

Posted: 2016-07-06T05:32:50-07:00
by snibgo
I'm not familiar with the functions you use, but you seem to loop through all the images in all the files. I can't see where images are closed, releasing memory.


Number of images isn't a problem. I regularly have hundreds or thousands in one command. The usual limit is total number of pixels in all the concurrently open images.

Re: Out of memory when converting multipage Tiff

Posted: 2016-07-06T23:56:37-07:00
by dlemstra
Is it possible to create a small console application that I can use to reproduce the problem and share it on dropbox? Feel free to contact me if you don't want to share your files publicly. The out of memory message might not be a memory issue but another quantize (reduce colors) error. We should probably change this to avoid confusion in the future.

Re: Out of memory when converting multipage Tiff

Posted: 2016-07-07T00:40:17-07:00
by DP-LT
I have created an application that reproduces the problem:
https://www.dropbox.com/s/ubuhrylid29fz ... t.zip?dl=0

The tiffs I use to test are here:
https://www.dropbox.com/s/d9y3k2cfsbu85 ... s.zip?dl=0

Re: Out of memory when converting multipage Tiff

Posted: 2016-07-07T12:55:21-07:00
by dlemstra
Thanks for the quick response, I will try to take a look at it this weekend.

Re: Out of memory when converting multipage Tiff

Posted: 2016-07-17T14:47:15-07:00
by dlemstra
I am unable to reproduce the issue with your example. I takes a while but finished without any problems and creates a tiff document with 44 pages.

Re: Out of memory when converting multipage Tiff

Posted: 2016-07-19T03:14:54-07:00
by DP-LT
Could the problem have something to do with my system specifications?

Operating System
Windows 10 Pro 64-bit

CPU
Intel Core i5 2540M @ 2.60GHz
Sandy Bridge 32nm Technology

RAM

8,00GB Dual-Channel DDR3 @ 665MHz (9-9-9-24)

Motherboard
Dell Inc. 0NVF5K (CPU 1)

Graphics
DELL P2213 (1680x1050@59Hz)
DELL P2213 (1680x1050@59Hz)
Intel HD Graphics 3000 (Dell)

Storage
232GB Samsung SSD 840 EVO 250GB (SSD)

Optical Drives
MATSHITA DVD+-RW UJ8B2

Audio
High Definition Audio-apparaat

Re: Out of memory when converting multipage Tiff

Posted: 2016-07-21T13:03:25-07:00
by dlemstra
Your hardware looks okay. Mine is a bit better but that should not matter. Not sure what is causing this issue. Are you using the latest version btw?

Re: Out of memory when converting multipage Tiff

Posted: 2016-07-22T01:01:04-07:00
by DP-LT
I am using Magick.NET-Q8-AnyCPU version 7.0.2.400. I did notice that when I uncheck "Prefer 32-bit" (thus prefering 64-bit) the exception is not thrown.