Exception thrown No space for output buffer

Magick.NET is an object-oriented C# interface to ImageMagick. Use this forum to discuss, make suggestions about, or report bugs concerning Magick.NET
Keith.Vinson
Posts: 22
Joined: 2017-08-30T12:59:12-07:00
Authentication code: 1151

Exception thrown No space for output buffer

Post by Keith.Vinson » 2018-02-15T08:58:29-07:00

Hello All,

When processing some fairly large TIFF images 400MB+ I randomly get this error (which doesn't seem to be related to any specific image from the block of images I'm working with), which seem to me to indicate some sort of an out of memory error. However at the time of the error the process's memory utilization is fairly low typically less than 200 MB on a machine showing about 8GB free & plenty of free disk space. Any ideas as to how I should fix or address this situation? Are there some config parameters I should be looking at?

Thanks,
Keith

snibgo
Posts: 10271
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Exception thrown No space for output buffer

Post by snibgo » 2018-02-15T09:34:39-07:00

What version of ImageMagick? On what platform?

How many pixels are in memory at any one time?

What does "magick -list resource" say? On my computer, this is:

Code: Select all

Resource limits:
  Width: 214.7MP
  Height: 214.7MP
  Area: 25.627GB
  Memory: 11.934GiB
  Map: 23.867GiB
  Disk: unlimited
  File: 1536
  Thread: 8
  Throttle: 0
  Time: unlimited
If you have a small limit in there, that may be the problem. Fix it by editing delegates.xml.
snibgo's IM pages: im.snibgo.com

Keith.Vinson
Posts: 22
Joined: 2017-08-30T12:59:12-07:00
Authentication code: 1151

Re: Exception thrown No space for output buffer

Post by Keith.Vinson » 2018-02-15T09:49:35-07:00

I'm using Magick.NET-Q8-AnyCPU" version="7.4.0"
Let me look for how to get the equivalent of magick -list from the DotNet library.
I looked at the delegates.xml file but I didn't see any tiff config params listed

snibgo
Posts: 10271
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Exception thrown No space for output buffer

Post by snibgo » 2018-02-15T10:10:50-07:00

Can you paste the exact error message here? One possibility is that pixels have consumed all the available physical memory, leaving insufficient for the output processing. If so, then putting low limits on memory and map in delegates.xml will store pixels on disk, leaving room for output processing.

This will slow the program, so may not be the final solution, but is a good test.
snibgo's IM pages: im.snibgo.com

User avatar
magick
Site Admin
Posts: 10807
Joined: 2003-05-31T11:32:55-07:00

Re: Exception thrown No space for output buffer

Post by magick » 2018-02-15T10:25:31-07:00

ImageMagick has resource limits to help push large images to disk to conserve memory, however, some TIFF images are decoded by the libTIFF delegate library rather than ImageMagick. If so, libTIFF may try to allocate enough memory for the entire image and fail. It all depends on the image type. Many TIFF images are processed by iterating over a single scanline at a time which typically does not consume much memory.

Keith.Vinson
Posts: 22
Joined: 2017-08-30T12:59:12-07:00
Authentication code: 1151

Re: Exception thrown No space for output buffer

Post by Keith.Vinson » 2018-02-15T10:42:02-07:00

Here's one of the errors as captured in the debugger...
ImageMagick.MagickCoderErrorException: 'No space for data buffer at scanline 4294967295. `TIFFFillStrip' @ error/tiff.c/TIFFErrors/650'

at ImageMagick.NativeInstance.CheckException(IntPtr exception, IntPtr result)\r\n
at ImageMagick.MagickImage.NativeMagickImage.ReadFile(MagickSettings settings)\r\n
at ImageMagick.MagickImage.Read(String fileName, MagickReadSettings readSettings, Boolean ping)\r\n
at ImageMagick.MagickImage..ctor(String fileName)\r\n

<the rest is just the stack trace of my crap...>

Keith.Vinson
Posts: 22
Joined: 2017-08-30T12:59:12-07:00
Authentication code: 1151

Re: Exception thrown No space for output buffer

Post by Keith.Vinson » 2018-02-15T10:48:51-07:00

The image in question is 14644 x 9346 as a 24 bit RGB uncompressed TIFF, its disk size is roughly 391 MB...

Keith.Vinson
Posts: 22
Joined: 2017-08-30T12:59:12-07:00
Authentication code: 1151

Re: Exception thrown No space for output buffer

Post by Keith.Vinson » 2018-02-15T13:03:32-07:00

IS thee a way I can control the delegate config params in code? If not I'm guessing I can by adding parameters into the delegates.xml file. Are the parameters documented somewhere?

Thanks,
Keith

User avatar
magick
Site Admin
Posts: 10807
Joined: 2003-05-31T11:32:55-07:00

Re: Exception thrown No space for output buffer

Post by magick » 2018-02-15T13:05:25-07:00

It looks like our speculation was correct. The libTIFF delegate library is attempting to allocate enough space to host the entire image in memory. We did not write nor do we support libTIFF so we can't help. If you post a URL to your image, we will download and inspect it. We might be able to read the image a scanline at a time which would resolve the problem and conversely the image might be in a format only libTIFF can handle. We can't tell without inspecting the image.

Keith.Vinson
Posts: 22
Joined: 2017-08-30T12:59:12-07:00
Authentication code: 1151

Re: Exception thrown No space for output buffer

Post by Keith.Vinson » 2018-02-15T13:08:20-07:00

Thanks, I'll post the image and send you a URL as soon as I can get it uploaded.


Keith.Vinson
Posts: 22
Joined: 2017-08-30T12:59:12-07:00
Authentication code: 1151

Re: Exception thrown No space for output buffer

Post by Keith.Vinson » 2018-02-15T17:01:24-07:00

Just an FYI: while playing with it, I received a new class of error while importing image 4.tif.
"2018-02-15T18:00:17-06:00 4:24.578 70.188u 7.0.7 Exception OpusFreeFlow.exe[8204]: constitute.c/ReadImage/572/Exception\n Coder (TIFF) generated an image despite an error (450), notify the developers"

User avatar
magick
Site Admin
Posts: 10807
Joined: 2003-05-31T11:32:55-07:00

Re: Exception thrown No space for output buffer

Post by magick » 2018-02-15T18:38:29-07:00

The conversion worked for us with this command:

Code: Select all

convert -limit memory 0 -limit map 0 1.tif 1.png
We're using ImageMagick 7.0.7-22 under Linux. Try the command line as above. Does that work? If so, you need to set the memory and map resource limits in your Magick.NET script.

Keith.Vinson
Posts: 22
Joined: 2017-08-30T12:59:12-07:00
Authentication code: 1151

Re: Exception thrown No space for output buffer

Post by Keith.Vinson » 2018-02-16T11:40:20-07:00

Inside the Magick.NET library I tried setting the memory and map sizes to 0, it still throws the exception: "2018-02-16T12:29:22-06:00 0:16.702 1.703u 7.0.7 Exception OpusFreeFlow.exe[19276]: tiff.c/TIFFErrors/650/Exception\n No space for data buffer at scanline 4294967295. `TIFFFillStrip'"

I used the policy.xml file to set them like this:
<policy domain="resource" name="memory" value="0MiB"/>
<policy domain="resource" name="map" value="0MiB"/>

I haven't tried the command line version yet, I'll have to download a copy of it as the Dot.Net library doesn't expose the underlying DLL and EXEs for me to try...

Keith.Vinson
Posts: 22
Joined: 2017-08-30T12:59:12-07:00
Authentication code: 1151

Re: Exception thrown No space for output buffer

Post by Keith.Vinson » 2018-02-16T13:36:50-07:00

I'm guessing under windows this is the equivalent cmd line "magick -limit memory 0 -limit map 0 1.tif 1.png" which worked, but then again this cmd line also worked: "magick 1.tif 1.png". To me it seems that this issue seems to be more related to large object heap fragmentation issues. In that, sometimes the conversion works and sometimes it doesn't even when processing the same image in a loop. I've had the test fail on its first attempt, and then sometimes it may loop 3 or 4 times before the failure occurs. But sooner or later it will fail with the same exception "No space for data buffer "

Keith

Post Reply