Out of memory when converting multipage Tiff

Magick.NET is an object-oriented C# interface to ImageMagick. Use this forum to discuss, make suggestions about, or report bugs concerning Magick.NET
Post Reply
DP-LT
Posts: 4
Joined: 2016-07-06T00:56:07-07:00
Authentication code: 1151

Out of memory when converting multipage Tiff

Post by DP-LT » 2016-07-06T02:49:50-07:00

Hi,

I'm having a problem with merging and converting multi page Tiffs. I am using the following code to merge multiple multipage Tiffs and apply adaptive thresholding to them:

Code: Select all

public void applyAdaptiveThresholdingAndMergeTiffs(List<string> inputFiles, string outputFile)
        {
            using (MagickImageCollection images = new MagickImageCollection()) {
                foreach (string inputFile in inputFiles) images.AddRange(inputFile);
                for (int c = 0; c < images.Count; c++) images[c].AdaptiveThreshold(25, 25, -10);
                images[0].CompressionMethod = CompressionMethod.Group4;
                images.Write(outputFile);
            }
        }
It works fine for Tiffs with a small amount of pages, however when converting Tiffs with 50 pages or more images.Write() throws the following exception:

Memory allocation failed `C:\ImageMagickTest\ImageMagickOutput\merged.tiff' @ error/quantize.c/QuantizeImage/2705
at ImageMagick.MagickExceptionHelper.Check(IntPtr exception)
at ImageMagick.NativeHelper.CheckException(IntPtr exception)
at ImageMagick.MagickImageCollection.NativeMagickImageCollection.WriteFile(MagickImage image, MagickSettings settings)
at ImageMagick.MagickImageCollection.Write(String fileName)
at ImageMagickTest.ImageMagickController.applyAdaptiveThresholdingAndMergeTiffs(List`1 inputFiles, String outputFile)

When images.Write is called memory usage starts to drop, the exception is thrown when memory usage is only 120MB:

Image

I have tried using

Code: Select all

ResourceLimits.Memory = 900000000; //900000000 bytes or 900MB, just to be on the safe side of the ~1300MB limit of 32-bit .NET

to limit memory usage, without succes.

I have created a test solution to simulate and recreate the problem that occcurs in my main application. To mimic the memory overhead of my main application I use an empty byte array of 100MB in the test.

Am I doing something wrong in my code, or is it not possible to convert multipage Tiffs with 50 pages or more?
Last edited by DP-LT on 2016-07-06T05:39:33-07:00, edited 2 times in total.

snibgo
Posts: 12152
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Out of memory when converting multipage Tiff

Post by snibgo » 2016-07-06T05:32:50-07:00

I'm not familiar with the functions you use, but you seem to loop through all the images in all the files. I can't see where images are closed, releasing memory.


Number of images isn't a problem. I regularly have hundreds or thousands in one command. The usual limit is total number of pixels in all the concurrently open images.
snibgo's IM pages: im.snibgo.com

User avatar
dlemstra
Posts: 1569
Joined: 2013-05-04T15:28:54-07:00
Authentication code: 6789
Contact:

Re: Out of memory when converting multipage Tiff

Post by dlemstra » 2016-07-06T23:56:37-07:00

Is it possible to create a small console application that I can use to reproduce the problem and share it on dropbox? Feel free to contact me if you don't want to share your files publicly. The out of memory message might not be a memory issue but another quantize (reduce colors) error. We should probably change this to avoid confusion in the future.
.NET + ImageMagick = Magick.NET https://github.com/dlemstra/Magick.NET, @MagickNET, Donate

DP-LT
Posts: 4
Joined: 2016-07-06T00:56:07-07:00
Authentication code: 1151

Re: Out of memory when converting multipage Tiff

Post by DP-LT » 2016-07-07T00:40:17-07:00

I have created an application that reproduces the problem:
https://www.dropbox.com/s/ubuhrylid29fz ... t.zip?dl=0

The tiffs I use to test are here:
https://www.dropbox.com/s/d9y3k2cfsbu85 ... s.zip?dl=0

User avatar
dlemstra
Posts: 1569
Joined: 2013-05-04T15:28:54-07:00
Authentication code: 6789
Contact:

Re: Out of memory when converting multipage Tiff

Post by dlemstra » 2016-07-07T12:55:21-07:00

Thanks for the quick response, I will try to take a look at it this weekend.
.NET + ImageMagick = Magick.NET https://github.com/dlemstra/Magick.NET, @MagickNET, Donate

User avatar
dlemstra
Posts: 1569
Joined: 2013-05-04T15:28:54-07:00
Authentication code: 6789
Contact:

Re: Out of memory when converting multipage Tiff

Post by dlemstra » 2016-07-17T14:47:15-07:00

I am unable to reproduce the issue with your example. I takes a while but finished without any problems and creates a tiff document with 44 pages.
.NET + ImageMagick = Magick.NET https://github.com/dlemstra/Magick.NET, @MagickNET, Donate

DP-LT
Posts: 4
Joined: 2016-07-06T00:56:07-07:00
Authentication code: 1151

Re: Out of memory when converting multipage Tiff

Post by DP-LT » 2016-07-19T03:14:54-07:00

Could the problem have something to do with my system specifications?

Operating System
Windows 10 Pro 64-bit

CPU
Intel Core i5 2540M @ 2.60GHz
Sandy Bridge 32nm Technology

RAM

8,00GB Dual-Channel DDR3 @ 665MHz (9-9-9-24)

Motherboard
Dell Inc. 0NVF5K (CPU 1)

Graphics
DELL P2213 (1680x1050@59Hz)
DELL P2213 (1680x1050@59Hz)
Intel HD Graphics 3000 (Dell)

Storage
232GB Samsung SSD 840 EVO 250GB (SSD)

Optical Drives
MATSHITA DVD+-RW UJ8B2

Audio
High Definition Audio-apparaat

User avatar
dlemstra
Posts: 1569
Joined: 2013-05-04T15:28:54-07:00
Authentication code: 6789
Contact:

Re: Out of memory when converting multipage Tiff

Post by dlemstra » 2016-07-21T13:03:25-07:00

Your hardware looks okay. Mine is a bit better but that should not matter. Not sure what is causing this issue. Are you using the latest version btw?
.NET + ImageMagick = Magick.NET https://github.com/dlemstra/Magick.NET, @MagickNET, Donate

DP-LT
Posts: 4
Joined: 2016-07-06T00:56:07-07:00
Authentication code: 1151

Re: Out of memory when converting multipage Tiff

Post by DP-LT » 2016-07-22T01:01:04-07:00

I am using Magick.NET-Q8-AnyCPU version 7.0.2.400. I did notice that when I uncheck "Prefer 32-bit" (thus prefering 64-bit) the exception is not thrown.

Post Reply