Invalid files DPX -> TIF

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
friolator
Posts: 23
Joined: 2014-02-18T15:57:30-07:00
Authentication code: 6789

Invalid files DPX -> TIF

Post by friolator » 2014-02-18T16:30:49-07:00

We need to convert some 2k film scans (DPX, Linear) to TIF, for processing in another command line tool that only accepts TIF. This tool came with a shell script that batch processes all the DPX files in a folder, using ImageMagick, and outputs the grayscale TIF files it wants. However, the resulting TIF files don't work in that application, nor can we open them in Photoshop or other image viewing tools.

The shell script does the following:

Code: Select all

for filename in *
do
 convert -set reference-black 0 -set reference-white 1023 -depth 16 -colorspace Gray -gamma 1.0 $filename ../convertedTIFs/${filename%.dpx}_gl.tif
done
This correctly results in the creation of files in the 'convertedTIFs' folder. But these files are no good, at least according to Photoshop and the target application for the converted files.

I have uploaded a source file (DPX) and the ouput file (TIF) here:

https://www.dropbox.com/sh/p4bg3ao7dorxf9u/sIhiU_216R

Even if I strip that convert command down to the bare minimum: "convert source.dpx target.tif", the result is no good.

I am able to make a grayscale TIF in Photoshop, save it out and run it through the target command line app, so I've ruled that application out as the problem.

The system this is on:

Mac OS X 10.6.8
ImageMagick 6.8.8-7 (compiled from source this afternoon, default settings)
LibTIFF 3.8.2 (which the target app requires, not sure if it's relevant here)

The source files were scanned today on our film scanner, and we've tried with 10 and 16 bit DPX files. Because it's coming from a positive, our scanner automatically creates Linear files, so no log->lin conversion should be necessary.

Any thoughts? Suggestions?

What's the best way to compare the TIF file created by ImageMagick (which doesn't work) with the TIF created by Photoshop from the same DPX (this one does work)? Maybe that will shed some light on what's happening.

Thanks!

User avatar
fmw42
Posts: 25577
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Invalid files DPX -> TIF

Post by fmw42 » 2014-02-18T17:19:57-07:00

I am on Mac OSX Snow Leopard but temporarily running IM 6.8.0.5 and LIBTIFF, Version 3.9.5. I can process your image with the following commands and I can open the file in PS CS. But the result is very grainy. I am not an expert on DPX. Perhaps you need to ugrade your libtif

Code: Select all

convert BaseUp_DPX_COLOR_10Bit_016.dpx -set reference-black 0 -set reference-white 1023 -set gamma 1.0 -colorspace Gray -depth 16  tmp.tif
It looks fine with png output or -depth 8 tiff output. So perhaps there is a bug in libtiff or it cannot handle 16-bit grayscale properly or a newer version of libtiff is needed.

Can you try converting using libtiff standalone? If that works, then there is a bug in IM?

friolator
Posts: 23
Joined: 2014-02-18T15:57:30-07:00
Authentication code: 6789

Re: Invalid files DPX -> TIF

Post by friolator » 2014-02-18T21:13:32-07:00

Would you mind posting the TIF file you were able to make? This film is a bit unusual - it's a scan of a home movie made using a short-lived Kodak lenticular color process, so it's not going to look super sharp. There should be lots of vertical lines, and because it's from 16mm it's pretty grainy.

I don't see any tools in LibTIFF to convert from DPX, but there is a Color->BW converter (tiff2bw), so I may be able to scan direct to RGB 16bit TIFF files and then convert those using tiff2bw instead of ImageMagick. Looks like it also has a tool for comparing TIFFs, so I'll run both the IM and the Photoshop TIFFs through that tomorrow to see if I can find differences.

Thanks!

User avatar
fmw42
Posts: 25577
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Invalid files DPX -> TIF

Post by fmw42 » 2014-02-18T21:28:17-07:00

http://www.fmwconcepts.com/misc_tests/d ... _16bit.tif

http://www.fmwconcepts.com/misc_tests/d ... t_8bit.tif

http://www.fmwconcepts.com/misc_tests/d ... _16bit.png

I used libtiff 3.9.5, but the latest is 4.0.3. Perhaps it will work with that version.

Perhaps repost this to the bugs or developers forum.

I think the IM developers should look into this and verify if it is an IM or libtiff issue with generating 16-bit grayscale tiff from dpx. It looks so bad compared to the 8-bit tiff or 16-bit png.


P.S. I think it is a libtiff issue, since going directly to PNG works, but this does not:

Code: Select all

convert BaseUp_DPX_COLOR_10Bit_016.dpx -set reference-black 0 -set reference-white 1023 -set gamma 1.0 -colorspace Gray -depth 16  PNG:- | convert - tmp.tiff

snibgo
Posts: 12165
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Invalid files DPX -> TIF

Post by snibgo » 2014-02-18T22:07:51-07:00

With the current version of IM, on Windows 8.1, the command:

Code: Select all

convert BaseUp_DPX_COLOR_10Bit_016.dpx t.tiff
... succeeds but the result seems bad. Windows Photo Viewer says t.tiff is corrupt. Gimp opens it without objection, but shows a blank (transparent black) image. I get the same result from older versions of IM.

The following is more successful:

Code: Select all

convert BaseUp_DPX_COLOR_10Bit_016.dpx t.png
convert t.png t.tiff
According to "identify -verbose", t.png has gamma=1.

(Note that the operation "-gamma 1" should do nothing at all to any image, as the parameter is the required change, not an absolute value. To set an absolute value, you can use "-set gamma 1".)

"convert t.png t.tiff" copies the pixels without changing their values. "identify -verbose t.tiff" thinks t.tiff is sRGB, but I think tiff has no flags that could tell it otherwise.
snibgo's IM pages: im.snibgo.com

User avatar
fmw42
Posts: 25577
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Invalid files DPX -> TIF

Post by fmw42 » 2014-02-18T22:13:42-07:00

snibgo,

It is odd that the following does not work for me. The png file is good, but the tif file is not very good. Note in my command and above I used -set gamma (though the OP used -gamma, which as you say is a no-op and leaves the tiff gamma undefined)

Code: Select all

convert BaseUp_DPX_COLOR_10Bit_016.dpx -set reference-black 0 -set reference-white 1023 -set gamma 1.0 -colorspace Gray -depth 16  tmp.png

convert tmp.png tmp.tif
What version libtiff are you using?

snibgo
Posts: 12165
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Invalid files DPX -> TIF

Post by snibgo » 2014-02-18T22:35:54-07:00

fmw42 wrote:The png file is good, but the tif file is not very good.
That massive graininess is characteristic of swapped bytes. We can test this theory:

Code: Select all

convert result_16bit.tif r2.png

convert -define png:swap-bytes r2.png r2swap.tiff
r2swap.tiff looks good (to me).
fmw42 wrote:What version libtiff are you using?
Whatever the pre-compiled Windows binary uses. Is there a command I can use to tell me the libtiff version?
snibgo's IM pages: im.snibgo.com

User avatar
fmw42
Posts: 25577
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Invalid files DPX -> TIF

Post by fmw42 » 2014-02-18T22:39:30-07:00

on unix, tiffinfo, tells the version. don't know if that works on windows.

I tried earlier to swap the bytes using -set tiff:endian lsb, but that did not help. If that is the issue, then libtiff has a bug, since converting to png works fine.

P.S. Your swap bytes does not work for me, but I am on an older IM 6.8.0.5 and that may not have been available back then. But that may be a good workaround for now.

User avatar
dlemstra
Posts: 1571
Joined: 2013-05-04T15:28:54-07:00
Authentication code: 6789
Contact:

Re: Invalid files DPX -> TIF

Post by dlemstra » 2014-02-19T02:53:35-07:00

snibgo wrote:Whatever the pre-compiled Windows binary uses. Is there a command I can use to tell me the libtiff version?
There is no command to get this information. You can view the version on the properties of the file 'CORE_RL_tiff_.dll'. A new version of ImageMagick on Windows will be using 4.0.3.
.NET + ImageMagick = Magick.NET https://github.com/dlemstra/Magick.NET, @MagickNET, Donate

snibgo
Posts: 12165
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Invalid files DPX -> TIF

Post by snibgo » 2014-02-19T03:23:28-07:00

I've reported my problem as a bug: viewtopic.php?f=3&t=25029

I think the workaround for the OP is simply to have two convert commands. The first creates a PNG. The second reads that PNG and creates a TIFF.

fmw42 wrote:P.S. Your swap bytes does not work for me, but I am on an older IM 6.8.0.5 and that may not have been available back then.
Yes, png:swap-bytes is a very recent addition.
dlemstra wrote:You can view the version on the properties of the file 'CORE_RL_tiff_.dll'.
Thanks. I use the static version, which doesn't have dll files.
snibgo's IM pages: im.snibgo.com

friolator
Posts: 23
Joined: 2014-02-18T15:57:30-07:00
Authentication code: 6789

Re: Invalid files DPX -> TIF

Post by friolator » 2014-02-19T06:22:32-07:00

Thanks for the suggestions. I'll give the DPX->PNG->TIFF path a try today, though I'm not nuts about having to create another set of images. There will be well over 20,000 of them in the final film, once we're done with the tests, so that's a lot of extra 2k data to generate and a lot more processing time. Ideally I'd like to find a way that avoids having to create too many intermediate files, to speed things up, so the first test will be to scan direct to 16 bit TIFFs and see if we can just convert them to greyscale using the tools provided by LibTIFF. If that doesn't work, then it's back to IM.

Thanks for all the suggestions!

snibgo
Posts: 12165
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Invalid files DPX -> TIF

Post by snibgo » 2014-02-19T10:00:02-07:00

You don't have to keep the png file.

Over on the bug thread, magick points out that IM creates a 10-bit tiff, and that's why other software can't read it. I have confirmed that putting "-depth 16" before the output tiff name cures that problem. I see you have "-depth 16", but not just before the output name. I suggest you try that.
snibgo's IM pages: im.snibgo.com

friolator
Posts: 23
Joined: 2014-02-18T15:57:30-07:00
Authentication code: 6789

Re: Invalid files DPX -> TIF

Post by friolator » 2014-02-19T10:07:40-07:00

Thanks, I'll give the '-depth 16' flag a try when I sit down to work on this after lunch.

The drive space really isn't an issue - this is all happening on a machine with 21TB of fast storage. But it's the processing time I'm worried about. As it is, once scanned it will take quite a bit of time to convert the files to TIFF, and then at least 12-14 hours to process the final images through the software that these are ultimately destined for. So, I'm looking more at speed than space here.

Thanks!

User avatar
fmw42
Posts: 25577
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Invalid files DPX -> TIF

Post by fmw42 » 2014-02-19T10:27:29-07:00

This should work without saving the png.
convert BaseUp_DPX_COLOR_10Bit_016.dpx -set reference-black 0 -set reference-white 1023 -set gamma 1.0 -colorspace Gray -depth 16 PNG:- | convert -define png:swap-bytes - tmp.tiff

friolator
Posts: 23
Joined: 2014-02-18T15:57:30-07:00
Authentication code: 6789

Re: Invalid files DPX -> TIF

Post by friolator » 2014-02-19T12:14:52-07:00

I've tried running this directly through LibTIFF's tiff2bw tool from a 16 bit TIFF file fresh off the scanner, but I get an error that it only handles 8 bit files. I'm able to get it to work with an 8bit scan, but of course it only outputs 8 bit files, even though I need 16bit files for the target application later on. So that's out. I'm not sure if that's a LibTIFF limitation or a limitation of the tiff2bw too. My guess is that it's LibTIFF.
convert BaseUp_DPX_COLOR_10Bit_016.dpx -set reference-black 0 -set reference-white 1023 -set gamma 1.0 -colorspace Gray -depth 16 PNG:- | convert -define png:swap-bytes - tmp.tiff
This doesn't work for me. I get the following when I run this. Maybe there's a syntax error somewhere?

Code: Select all

$ ./convertCommand.sh
/Volumes/RAID/Lenticular/DPX_COLOR_10Bit
BaseUp_DPX_COLOR_10Bit_016.dpx to be processed...
convert: no decode delegate for this image format `/var/tmp/magick-6323FLOd5w4MLLzA' @ error/constitute.c/ReadImage/555.
convert: no images defined `../convertedTIFs/BaseUp_DPX_COLOR_10Bit_016_gl.tif' @ error/convert.c/ConvertImageCommand/3147.
The line in convertCommand.sh that's calling IM is as follows (a slightly modified version of the above, to fit into the loop that's iterating through files in this folder:

Code: Select all

convert $filename -set reference-black 0 -set reference-white 1023 -set gamma 1.0 -colorspace Gray -depth 16 PNG:- | convert -define png:swap-bytes - ../convertedTIFs/${filename%.dpx}_gl.tif
I like the idea of writing to a temp PNG file, and I think that's probably a viable solution for now. Hopefully there's just a dumb syntax error on my part that I'm not seeing.

thanks!

Post Reply