batch processing / response file support

Questions and postings pertaining to the development of ImageMagick, feature enhancements, and ImageMagick internals. ImageMagick source code and algorithms are discussed here. Usage questions which are too arcane for the normal user list should also be posted here.
Post Reply
00bins
Posts: 10
Joined: 2019-04-18T15:11:07-07:00
Authentication code: 1152

batch processing / response file support

Post by 00bins » 2019-04-18T15:15:53-07:00

Hi

We use imagemagick to process 1000's of images as part of our build process.

As a test I've timed processing a subset of these files - around 1000.

Invoking convert for every file takes around 3 minutes, where each operation is like this

convert -resize 64x64! Character_packs.bmp -type truecolor tmp/Character_packs.bmp

I then have a modified version of convert that I can pass a file with the same operations like below

-resize 64x64! Character_packs.bmp -type truecolor tmp/Character_packs.bmp

and which loops over the file 'creating' argc & argv from the file without having the overhead of invoking convert for everyfile

timing both methods shows that the second method is way faster

$ time sh callperfile.sh

real 2m59.130s
user 2m36.229s
sys 0m19.217s

$ time convert @tmp.txt

real 0m22.214s
user 0m20.685s
sys 0m1.508s

I ran these tests on a mac but the same speed up applies to windows.

I'd like to contribute this change back to the source for imagemagick so that we don't have to keep our own version,
and so that anyone else can make use of this.

How do I go about proposing this change, and are there any docs for coding standards I should adhere to?


cheers
Rob

User avatar
fmw42
Posts: 25664
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: batch processing / response file support

Post by fmw42 » 2019-04-18T15:23:56-07:00

I cannot address the issue of contribution.

But I would point out that correct IM 6 and IM 7 processing should have the input should come before the -resize operation. IM syntax is not like Unix command syntax. See https://imagemagick.org/Usage/basics/#why. This is even more important for IM 7, which is much more strict in syntax usage. In IM 6 it may not matter, but it will in IM 7.

snibgo
Posts: 12306
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: batch processing / response file support

Post by snibgo » 2019-04-18T17:30:28-07:00

00bins wrote:I then have a modified version of convert ...
You can do the job in single command, eg:

Code: Select all

convert \
in1.bmp -resize 64x64! type truecolor -write out1.bmp +delete \
in2.bmp -resize 64x64! type truecolor -write out2.bmp +delete \
in3.bmp -resize 64x64! type truecolor -write out3.bmp +delete \
:
in999.bmp -resize 64x64! type truecolor -write out999.bmp +delete \
xc: NULL:
Or you can use IM's script facility.

Code: Select all

convert @myscript.txt xc: NULL:
(Note: IM v6 scripting is different to v7.)
snibgo's IM pages: im.snibgo.com

00bins
Posts: 10
Joined: 2019-04-18T15:11:07-07:00
Authentication code: 1152

Re: batch processing / response file support

Post by 00bins » 2019-04-18T22:48:58-07:00

Thanks for the replies

re syntax - I'll fix that!

From what I can tell IM scripts don't really address my issue - they are shell/bat/perl/etc.. scripts that still end up invoking magick once per file - unless I'm missing something.

On windows there is a command line length limit of 8k chars. My test with only 1000 files is already 256k chars so I can't use that.

also the example

convert @myscript.txt xc: NULL:

just prints out the usage - and afaict there is nothing in the existing code that knows how to process an argument preceeded by @ (because I had to add that myself) - or am I missing something here too?

thanks
Rob

00bins
Posts: 10
Joined: 2019-04-18T15:11:07-07:00
Authentication code: 1152

Re: batch processing / response file support

Post by 00bins » 2019-04-18T23:35:59-07:00

aha - I've just noticed that magick has a -script option, which would appear to be what I need after all

thanks
Rob

00bins
Posts: 10
Joined: 2019-04-18T15:11:07-07:00
Authentication code: 1152

Re: batch processing / response file support

Post by 00bins » 2019-04-19T00:06:04-07:00

ugh - it's early but I can't seem to find anything on the forums relating to the -script option. Searching for "-script" returns no hits - not even my post above, and no amount of shell-escaping or quoting seems to help.

I've made a small file with what I think should work, but it doesn't do what I want, and I probably should be posting in the users section now anyway

cheers
Rob

snibgo
Posts: 12306
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: batch processing / response file support

Post by snibgo » 2019-04-19T05:32:14-07:00

00bins wrote:On windows there is a command line length limit of 8k chars.
Yes. The IM script facility has no limit, as far as I can see. I have tested it with long scripts, eg 100KB.
snibgo's IM pages: im.snibgo.com

Post Reply