Page 1 of 1

Resizing very large 16-bit TIFF messes up the grayscale.

Posted: 2020-02-06T14:28:35-07:00
by Maluminas
Hi! I'm very new to ImageMagick so I'm posting this here, since it is way more likely to be user error than a bug.

I tried ImageMagick because I need to resize a very large TIFF to a more manageable size, so I needed software that could do it without loading the whole thing into RAM. The TIFF in question is a 5 gigapixel 16-bit grayscale heightmap of Mars (can be downloaded here, 11 GB). The only other size they offer a much too small 1024x512 px sample.

The resizing works really smoothly and reasonably fast on my average machine, but the resulting TIFF is not what I expect: the levels are weirdly corrupted, as if the histogram got rotated 180 degrees along the brightness axis:

The command I used is very simple, and is probably missing something I'm unaware of:

Code: Select all

magick Mars_HRSC_MOLA_BlendDEM_Global_200mp_v2.tif -resize 1024x512 resized.tiff
Here is the output of -identify on the 11GB file, if you don't wish to download it. The errors at the end happen anytime a command involves that file.

Code: Select all

Image: Mars_HRSC_MOLA_BlendDEM_Global_200mp_v2.tif
  Format: TIFF64 (Tagged Image File Format (64-bit))
  Mime type: image/tiff
  Class: DirectClass
  Geometry: 106694x53347+0+0
  Units: PixelsPerInch
  Colorspace: Gray
  Type: Grayscale
  Endianess: LSB
  Depth: 16-bit
  Channel depth:
    Gray: 16-bit
  Channel statistics:
    Pixels: 5691804818
      min: 0  (0)
      max: 65535 (1)
      mean: 33032.8 (0.504048)
      standard deviation: 30193.6 (0.460724)
      kurtosis: -1.98759
      skewness: -0.0596043
      entropy: 0.889647
  Rendering intent: Undefined
  Gamma: 0.454545
  Matte color: grey74
  Background color: white
  Border color: srgb(223,223,223)
  Transparent color: none
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 106694x53347+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: None
  Orientation: TopLeft
    date:create: 2020-02-05T16:50:41+00:00
    date:modify: 2020-02-05T20:26:48+00:00
    signature: ca278a2b0a0071404f759b9511a2246b7987d124f28dbb369f569a5c01659276
    tiff:alpha: unspecified
    tiff:endian: lsb
    tiff:photometric: min-is-black
    tiff:rows-per-strip: 1
    verbose: true
  Tainted: False
  Filesize: 10.6026GiB
  Number pixels: 5.6918G
  Pixels per second: 64.2424MP
  User time: 40.594u
  Elapsed time: 1:29.598
  Version: ImageMagick 7.0.9-21 Q16 x64 2020-02-01
identify: Unknown field with tag 33550 (0x830e) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/1025.
identify: Unknown field with tag 33922 (0x8482) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/1025.
identify: Unknown field with tag 34735 (0x87af) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/1025.
identify: Unknown field with tag 34736 (0x87b0) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/1025.
identify: Unknown field with tag 34737 (0x87b1) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/1025.
identify: Unknown field with tag 42113 (0xa481) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/1025.
This seems like a very characteristic issue but I couldn't find similar posts by searching. Please let me know what I'm doing wrong... Thanks!

Re: Resizing very large 16-bit TIFF messes up the grayscale.

Posted: 2020-02-06T15:52:36-07:00
by snibgo
Sadly, 11 GB is too large for me to download.

IM does try to load the entire image into RAM. If that fails, it should use disk.

I have seen DEM files where a nominal "sea level" (standard ellipsoidal surface, or something) is represented as zero in the image files, with elevation above that as positive numbers, and numbers below that as negative numbers. But IM always reads integers as positive numbers. Hence elevation just below "sea level" comes out as white. Could that be happening here?

If it is, then "-evaluate AddModulus 50%" should fix it. Well, it helps, but doesn't seem to be the total solution.

Re: Resizing very large 16-bit TIFF messes up the grayscale.

Posted: 2020-02-06T16:18:50-07:00
by magick
@snibgo is correct. Try this command:

Code: Select all

magick Mars_HRSC_MOLA_BlendDEM_Global_200mp_v2.tif -evaluate AddModulus 50% -normalize -resize 1024x512 resized.tiff
You will likely get the expected image results. However, your image includes a number of private tags. They are ignored by ImageMagick but are perhaps clues on how the image should be post processed. They include:

Code: Select all

  Tag 33550: 0.003374,0.003374,0.000000
  Tag 33922: 0.000000,0.000000,0.000000,-180.000000,90.000000,0.000000
  Tag 34735: 1,1,0,11,1024,0,1,2,1025,0,1,1,2048,0,1,32767,2049,34737,123,0,2050,0,1,32767,2054,0,1,9102,2056,0,1,32767,2057,34736,1,0,2058,34736,1,1,2061,34736,1,2,2062,34736,3,3,0,0,0,0
  Tag 34736: 3396190.000000,3396190.000000,0.000000,0.000000,0.000000,0.000000
  Tag 34737: GCS Name = GCS_Mars_2000_Sphere|Datum = Mars_2000_Sphere|Ellipsoid = Mars_2000_Sphere_IAU_IAG|Primem = Reference_Meridian||
  Tag 42113: -32768

Re: Resizing very large 16-bit TIFF messes up the grayscale.

Posted: 2020-02-06T16:44:49-07:00
by snibgo
I tested "-evaluate AddModulus 50%"on the OP's downsized image, so it was bound to give weird results at the black/white boundary that was resized to give various shades of gray. As @magick suggests, this should be done before downsizing.

Re: Resizing very large 16-bit TIFF messes up the grayscale.

Posted: 2020-02-06T17:05:41-07:00
by magick
In the mean-time, we'll add a patch to return expected results without the need for the -evaluate option. The patch will land in the next release of ImageMagick, likely within a week.

Re: Resizing very large 16-bit TIFF messes up the grayscale.

Posted: 2020-02-06T21:33:18-07:00
by Maluminas
Thanks everyone for the help!

magick, your command worked almost perfectly, I just replaced -normalized with -contrast-stretch 0 to avoid losing data at the extremes. The result is exactly what I wanted.

snibgo, your idea about some kind of DEM encoding is probably what is going on. The description of the data (on the download page) seems to suggest the elevation is encoded against the areoid, which is the Mars equivalent of Earth's geoid. The transition line between the light and dark regions in my image probably lies along where the terrain elevation crosses the areoid, or something of the sort. I have little experience with this kind of data, so I'm glad magick's command effectively decoded that.

I'm very happy I found ImageMagick. It seems very powerful and has a very responsive community. Thanks again!

Re: Resizing very large 16-bit TIFF messes up the grayscale.

Posted: 2020-02-06T21:36:01-07:00
by fmw42
-contrast-stretch 0 may still lose a little data as it works from histogram bins. Use -auto-level in its place.

Re: Resizing very large 16-bit TIFF messes up the grayscale.

Posted: 2020-02-06T22:07:37-07:00
by snibgo
I've learnt a new word, thanks!
Areoid: The surface that provides the datum line (the equivalent for sea level) for Mars.
Beware of "-normalize", "-contrast-stretch" and "-auto-level", as they will all shift what was at 50% (the areoid) to some other value. But that may not matter for your purposes.

It is possible to expand the range of values so either the bottom reaches zero or the top reaches 100%, without shifting the 50% mark, if you want. I can't remember how, but it involves finding the minimum and maximum, then calculating parameters for "-level".

Re: Resizing very large 16-bit TIFF messes up the grayscale.

Posted: 2020-02-07T11:30:39-07:00
by Maluminas
I'm a stickler for accuracy, so I'd love to be able to process the data to end up with a reasonably accurate height map of the actual shape of Mars.

I'll describe my purpose here, though I don't expect any help since this is not really related to ImageMagick! I want to 3D print a topologically accurate globe of Mars. At a 1:10,000,000 scale, the globe would be about 339 mm in diameter. I've estimated that on average, not factoring-in the areoid shape (which is not accounted for in the 11 GB DEM image) the deviation of the surface of my globe compared to an accurate model would be on the order of 2 mm, which is quite a lot. I expect it would be even worse at the top and bottom of the elevation range.

Unfortunately, this seems to be way over my head. I'm a biochemist, so I know my way around scientific literature, but the actual papers I find are way out of my knowledge set. I'm hoping to find a dumb heightmap of the areoid, so that I can factor it into the DEM heightmap, but so far no luck. The areoid associated with the DEM file I'm working with is called GMM-2B (paper), if anyone skilled in the geodetic magicks is looking for a challenge.

As a technical note, I plan to re-project the final elevation model so that it can be wrapped around a heavily tesselated icosphere in Blender, and used to displace the surface to generate the actual 3D model to be printed (in 20 pieces because my printer can't fit the whole thing). I'm still looking for a perfect solution for re-projecting the map. So far in my experiments I've used Wilbur (by Joe Slayton) but it only saves the output with 8-bit depth, and I'd like to preserve the 16-bit depth.

However, it would be nice if I could avoid introducing new errors during my processing with ImageMagick, so I'll explore what you suggested snibgo, with -level parameters.

Re: Resizing very large 16-bit TIFF messes up the grayscale.

Posted: 2020-02-07T13:12:25-07:00
by snibgo
For an accurate globe, you need to know the scale of the DEM, eg a difference of one in pixel values represents a height of 1 metre or one foot or whatever. I can't see that info in the NASA page. Perhaps an email to someone will answer that.

Assuming that 50% of QuantumRange reprepresents the areoid, and we have values above and below that, we can stretch the scale so we have either 0 or 100% in the image, without shifting any pixels that were exactly 50%.

My Windows BAT script at Slopes does the job. I wrote it for IM v6, and it's a bit clunky.

For v7, the code is simpler:

Code: Select all

magick ^
  in.tiff ^
  -set option:FACT "%%[fx:maxima-0.5>0.5-minima?0.5/(maxima-0.5):0.5/(0.5-minima)]" ^
  -set option:OFFS "%%[fx:maxima-0.5>0.5-minima?0.5-0.25/(maxima-0.5):0.5-0.25/(0.5-minima)]" ^
  -function polynomial %%[FACT],%%[OFFS] ^
For bash, change each line-continuation from ^ to \, and change each double %% to single %. You may need to escape parentheses \( and \)

The transformation is:
v' = FACT*v + OFFS
where FACT is either 0.5/(maxima-0.5) or 0.5/(0.5-minima),
and OFFS is 0.5 - 0.5*FACT