Feature Request: Command line pipes other than stdin/stdout

Questions and postings pertaining to the development of ImageMagick, feature enhancements, and ImageMagick internals. ImageMagick source code and algorithms are discussed here. Usage questions which are too arcane for the normal user list should also be posted here.
dmhost

Feature Request: Command line pipes other than stdin/stdout

Post by dmhost »

It would be really cool if the command line tools of imagemagick supported more than just the default pipes (using the "-" argument) much like command line tools such as openssl and gpg

This would be super handy to pass in many input images files using numbered pipes aka file descriptors such as 3, 4, 5 etc.

For example something like this (using a suggested notation):

Code: Select all

convert - -fd3 -fd4 -fd5 -fd6 -append output.png
I am requesting that such a command would append input from file descriptor 0, 3, 4, 5 and 6 and write it to the file output.png

What do you think? Right now to do something like this (especially with more complicated commands) we need to use temporary files.

Thanks!
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: Feature Request: Command line pipes other than stdin/stdout

Post by magick »

Easy enough to do. From you we need a simple test environment we can use to verify our implementation. Which command sequence can we use to read an image from file descriptor 3, for example?
dmhost

Re: Feature Request: Command line pipes other than stdin/stdout

Post by dmhost »

Wow that was a fast reply!

Here is an example using a suggested notation of -fdN where N is a number greater than 2 corresponding to the pipe.

The code demonstrating theoretical custom file descriptors/pipes is written in PHP (simply due to my ignorance of shell scripting although I'm sure somebody with a little experience could help out). Let me know if you would like any extra info.

Code: Select all

<?php

// To run this script type: php SCRIPTNAME.php

// this script should output image-output.png which in theory is the following 5 images appended using convet -append
// in a real life situation these blobs would be coming from some other source such as another process's output descriptos or maybe even a database
$inputImageBlogA = file_get_contents('image-a.png');
$inputImageBlogB = file_get_contents('image-b.png');
$inputImageBlogC = file_get_contents('image-c.png');
$inputImageBlogD = file_get_contents('image-d.png');
$inputImageBlogE = file_get_contents('image-e.png');

// set up the descriptors
$descriptors = array(
  0 => array("pipe", "r"), // this is standard input
  1 => array("pipe", "w"), // this is standard output
  2 => array("pipe", "w"), // this is standard error
  3 => array("pipe", "r"), // this our custom input pipe to feed images
  4 => array("pipe", "r"), // this our custom input pipe to feed images
  5 => array("pipe", "r"), // this our custom input pipe to feed images
  6 => array("pipe", "r")  // this our custom input pipe to feed images
);

// build the command line
$commandLine = 'convert - -fd3 -fd4 -fd5 -fd6 -append image-output.png';
$processHandle = proc_open($commandLine, $descriptors, $pipes);

if(is_resource($processHandle)) {
  // push image E to pipe 6
  fwrite($pipes[6], $inputImageBlogE);
  fclose($pipes[6]);

  // push image D to pipe 5
  fwrite($pipes[5], $inputImageBlogD);
  fclose($pipes[5]);

  // push image C to pipe 4
  fwrite($pipes[4], $inputImageBlogC);
  fclose($pipes[4]);

  // push image B to pipe 3
  fwrite($pipes[3], $inputImageBlogB);
  fclose($pipes[3]);

  // push image A to pipe 0 (standard input)
  fwrite($pipes[0], $inputImageBlogA);
  fclose($pipes[0]);

  // read standardOut
  $standardOut = '';
  while(!feof($pipes[1])) {
    $standardOut .= fgets($pipes[1], 1024);
  }
  fclose($pipes[1]);

  // read standardError
  $standardError = '';
  while(!feof($pipes[2]))    {
    $standardError .= fgets($pipes[2], 1024);
  }
  fclose($pipes[2]);

  // close the process
  $returnCode = proc_close($processHandle);
} else {
  throw new Exception("Cannot run process");
}

echo "RETURNED CODE: $returnCode\n";
echo "STANDARD OUT: $standardOut\n";
echo "STANDARD ERR: $standardError\n";

?>
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: Feature Request: Command line pipes other than stdin/stdout

Post by magick »

Give us a few days to apply a patch to the ImageMagick Subversion trunk. Your command will end up looking like this:
  • convert - fd:3 fd:4 fd:5 fd:6 -append image-output.png
User avatar
anthony
Posts: 8883
Joined: 2004-05-31T19:27:03-07:00
Authentication code: 8675308
Location: Brisbane, Australia

Re: Feature Request: Command line pipes other than stdin/stdout

Post by anthony »

NOTE: You can do that already -- using BASH!!!!!

And as PHP does not provide a nice way of running your convert command without the use of a shell (arrrggghhh...) you may as well use it.

Code: Select all

   convert  /dev/fd3  /dev/fd/4  /dev/fd/5  -append  image-output.png
To find out if your 'PHP shell' is bash run

Code: Select all

<?php
  header('Content-Type: text/plain');
  system("exec 2>&1; echo shell is $SHELL");
?>

So if you are using BASH to generate the extra file descriptors than you
already have it.

Of course this does not help if you are avoiding the use of a SHELL
and in a perl script, or C programming. Also as this is probably
more to do with secure programing, it is probably a good idea.


However I would expand that idea to OUTPUT as well.

Code: Select all

   convert  rose:   -write fd:5   -swirl 90   fd:6
But it may need some method of specifing the OUTPUT format

For example

Code: Select all

   convert  rose:   -write jpeg:fd:5   -swirl 90   png:fd:6

WARNING WARNING WARNING.

While the large UNIX buffers will prevent a lot of problems with I/O deadlocks handling multiple streams of data can cause problems, unless each stream come from or goes to a completely separate parallel process. You need to be certain that IM can read all the data for each image, in the order given.

Your own script for example relies on the UNIX buffer being large enough to hold all the image data, as you write to the pipelines in the WRONG order!
Anthony Thyssen -- Webmaster for ImageMagick Example Pages
https://imagemagick.org/Usage/
User avatar
anthony
Posts: 8883
Joined: 2004-05-31T19:27:03-07:00
Authentication code: 8675308
Location: Brisbane, Australia

Re: Feature Request: Command line pipes other than stdin/stdout

Post by anthony »

NOTE some image file formats allow multiple images

MIFF: for example is especially good for this as you can just concatenate the MIFF image files together to pass it to IM. this makes it perfect to have a shell loop or sub-process that just generates each image and sends it down the pipeline one-after-the-other, for the main command to process.

For example in shell

for i in image1.gif image2.jpg image3.png
do
convert "$i" miff:-
done |
convert miff:- +append all_images_appened.png

Another example of this is in IM Examples, laying images
http://www.imagemagick.org/Usage/layers/#example

I also use this in the IM shell script "segment-image"
where I am splitting up an image into multiple sub-images
which still needs some extra processing to finish off the job.
http://www.imagemagick.org/Usage/scripts/segment_image
Anthony Thyssen -- Webmaster for ImageMagick Example Pages
https://imagemagick.org/Usage/
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: Feature Request: Command line pipes other than stdin/stdout

Post by magick »

We considered the /dev/fd devices, however, ImageMagick is cross-platform and this method does not work under Windows. Instead we use the solution stated earlier, for example, fd:3.
User avatar
anthony
Posts: 8883
Joined: 2004-05-31T19:27:03-07:00
Authentication code: 8675308
Location: Brisbane, Australia

Re: Feature Request: Command line pipes other than stdin/stdout

Post by anthony »

I agree... the /dev/fd devices are really for use by BASH and bash scripts, which makes it useless for more direct execution without a shell, from things like perl and C.

Though PHP does not have a simple direct execution of a command, without having that command parsed by a shell. A situation I thought silly considering that PHP is used in such a security conscious environment!

No the fd:# special format is a good general solution, and one I would also like to see used for output as well, though as mentioned the output format will still need to be settable.
Anthony Thyssen -- Webmaster for ImageMagick Example Pages
https://imagemagick.org/Usage/
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: Feature Request: Command line pipes other than stdin/stdout

Post by magick »

You can fd:# for output as well:
  • convert fd:3 gif:fd:4
User avatar
anthony
Posts: 8883
Joined: 2004-05-31T19:27:03-07:00
Authentication code: 8675308
Location: Brisbane, Australia

Re: Feature Request: Command line pipes other than stdin/stdout

Post by anthony »

Great
Anthony Thyssen -- Webmaster for ImageMagick Example Pages
https://imagemagick.org/Usage/
dmhost

Re: Feature Request: Command line pipes other than stdin/stdout

Post by dmhost »

Wow I am happy to see this discussion evolve into what it is!
What a great idea to add this with outputs as well: convert fd:3 gif:fd:4

Best!
dmhost

Re: Feature Request: Command line pipes other than stdin/stdout

Post by dmhost »

Aside: anthony did you know with in windows with php's proc_open you can bypass the shell by setting a flag on the $other_options parameter?
User avatar
anthony
Posts: 8883
Joined: 2004-05-31T19:27:03-07:00
Authentication code: 8675308
Location: Brisbane, Australia

Re: Feature Request: Command line pipes other than stdin/std

Post by anthony »

I took a look at the manual and could not see that flag, However it looks like a great function for starting a background process with multiple file descriptor channels. Something that is hard to do in a shell without resorting to multiple named pipes.

However the command remands 'shell parsed', in that command and its arguments are still given as a single string, which requires some shell to divide into the argument array of the command. And that is a security hazard.

There is a 'bypass shell' but that is for windows only.

See Perl system() function, for an example. if multiple arguments are given it calls the command direct!
Anthony Thyssen -- Webmaster for ImageMagick Example Pages
https://imagemagick.org/Usage/
Trevor
Posts: 2
Joined: 2015-11-11T22:55:49-07:00
Authentication code: 1151

Re: Feature Request: Command line pipes other than stdin/stdout

Post by Trevor »

Hi

I know this is somewhat of an old thread but I thought it would be appropriate to post my question on it.
I am calling ImageMagick from a node js environment as a child-process and trying to pass 2+ buffers as arguments into the convert process.
How can I know what, if any fd is assigned to a buffer?
Please see my original question http://stackoverflow.com/questions/3365 ... ld-process
My code below uses 1 buffer through stdin but I want all the input "files" which in fact are not files rather buffers to be passed as buffers without the need to save them as files. Over there I was suggested to use the fd:3 fd:4 but I don't know in practice how to get the real fd numbers, I wouldn't have thought that a "buffer" has an fd as it's not a file.
The code I currently have is below, I would really appreciate an example of how to get it to work without using any input files.
Thanks in Advance

Trevor

Code: Select all

var arrow1JpgBase64, arrow2JpgBase64, arrowBuffer1, arrowBuffer2, magicCommands, imagic;

spawn = require('child_process').spawn;
exec = require('child_process').exec;

arrow1JpgBase64 = "/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAUDBAQEAwUEBAQFBQUGBwwIBwcHBw8LCwkMEQ8SEhEPERETFhwXExQaFRERGCEYGh0dHx8fExciJCIeJBweHx7/2wBDAQUFBQcGBw4ICA4eFBEUHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh7/wAARCAAFAAkDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD1/wCFngjxJba78QbWT4reLTLH4pZ2mjt9PLTCSxs5VL+bbSYZUkWPCFUxGu1EHFdz/wAIb4j/AOiseM//AAF0n/5BoooA/9k=";
arrow2JpgBase64 = "/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAUDBAQEAwUEBAQFBQUGBwwIBwcHBw8LCwkMEQ8SEhEPERETFhwXExQaFRERGCEYGh0dHx8fExciJCIeJBweHx7/2wBDAQUFBQcGBw4ICA4eFBEUHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh7/wAARCAAFAAkDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD1/wCOvgjxJc/Dt7NPit4tMt1q+k20Ty2+nqsLyajbIkoMFtFJuRmDjbIpyo5xmu5/4Q3xH/0Vjxn/AOAuk/8AyDRRQB//2Q==";
arrowBuffer1 = new Buffer(arrow1JpgBase64, 'base64');
arrowBuffer2 = new Buffer(arrow2JpgBase64, 'base64');
magicCommands = ["jpg:",
                 "in_2.jpg",
                 "+append",
                 "out.jpg"];

imagic = spawn("convert", magicCommands);
imagic.stdin.write(arrowBuffer1);
imagic.stdin.end();

imagic.on('exit', function (code) {
    if (code === 0) {
        exec("open out.jpg");
    } else {
        console.log("error code: " + code);
    }
}); // end of on exit
User avatar
anthony
Posts: 8883
Joined: 2004-05-31T19:27:03-07:00
Authentication code: 8675308
Location: Brisbane, Australia

Re: Feature Request: Command line pipes other than stdin/stdout

Post by anthony »

all streams have a file number.
stdin is 0 stdout is 1 stderr is 2
After that streams are opened with later numbers.
you can convert file descriptors (Suchas what "stdin" really means) to there fileno

Shells use file numbers more directly for example

exec 3<some image
convert fd:3 show:
exec 3<&-

which is equivalent to the shell doing the open and close of the file, and giving the already opened file streams to the command.

As the file are opened by the shell, the shell itself could read or write extra information to those streams in addition to the convert command itself. For example dealing with multi-image file streams, or network streams that is also sending and receiving embedded images in the stream.

The original request is to get IM to not close such streams itself when it reaches the end of an image it is reading.
Of course that only works for images that have a well defined end, without relying on a actual EOF condition.
Not that in reading an concatenated multi-image streams, will need things to tell IM how many images to read before stopping!
Anthony Thyssen -- Webmaster for ImageMagick Example Pages
https://imagemagick.org/Usage/
Post Reply