Page 1 of 1

Imagemagick wand use only one single CPU

Posted: 2018-03-13T14:38:35-07:00
by loki5100
I have a multi thread app (8 threads) on a multi core machine (8 cpu). each thread is doing in parallel
NewMagickWand
MagickReadImage
MagickResizeImage
DestroyMagickWand
but imageMagick use only ONE CPU instead of 8 ! Even if I setup each thread to use a dedicated CPU with SetThreadAffinityMask ImageMagick still use always the same single CPU

How to make magickWand to use all the available CPU ? I m on windows and using imagemagick 7.
Is their any way to optimize magickwand to work in a multithread environment (inside a web server app) ?

Re: Imagemagick wand use only one single CPU

Posted: 2018-03-13T14:58:56-07:00
by snibgo
I'm not sure what you are saying. You are doing the four operations in parallel with each other? I doubt it. The image must be entirely read before it can be resized, etc.

How large is the image? Parallelism won't start below a certain size, which depends on how many threads are available.

It might help if you paste a minimal program that shows the problem, and link to a sample input image.

Re: Imagemagick wand use only one single CPU

Posted: 2018-03-14T00:56:39-07:00
by loki5100
Thanks snibgo, i m doing the "block" of theses operations in parallel like below :
THREAD 1
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
THREAD 2
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
THREAD 3
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
etc..

The image is 3390x2160 i guess it's enough to start the parallelism
I try with any image, alway the same result

I use delphi so the code will be in pascal but I think you will understand it :

Code: Select all

uses ALFiles,
     ALImageMagick;

procedure _resizeImage;
var aWand: PMagickWand;
begin
  aWand := ALImageMagickLib.NewMagickWand;
  try
    if ALImageMagickLib.MagickReadImage(aWand, pansiChar(ALGetModulePath + 'sample.jpg')) <> MagickTrue then RaiseLastWandError(aWand);
    if (ALImageMagickLib.MagickResizeImage(aWand, 640, 480, LanczosFilter) <> MagickTrue) then RaiseLastWandError(aWand);
  finally
    ALImageMagickLib.DestroyMagickWand(aWand);
  end;
end;

procedure TForm1.Button1Click(Sender: TObject);
var i: integer;
begin

  for i := 0 to 8 do begin
    TThread.CreateAnonymousThread(
      procedure
      begin
        while True do
          _resizeImage
      end).Start;
  end;

end;

procedure TForm1.FormCreate(Sender: TObject);
begin
  alCreateImageMagickLibrary(string(ALGetModulePath) + 'imagemagick');
end;
And this is the CPU utilization :

Image

Re: Imagemagick wand use only one single CPU

Posted: 2018-03-14T01:37:06-07:00
by loki5100
@snibgo I think i found what i will call a bug in ImageMagick

On my computer I have 4 core and 8 Logical processors

in the default policy.xml the row
<!-- <policy domain="resource" name="thread" value="4"/> -->
is commented, but when i do
convert -list resource
Resource limits:
Width: 214.748MP
Height: 214.748MP
List length: 18.4467EP
Area: 17.1373GP
Memory: 7.98018GiB
Map: 15.9604GiB
Disk: unlimited
File: 1536
Thread: 8
Throttle: 0
Time: unlimited
we saw that imageMagick is configured to use Thread: 8 that look like the number of logical processor in my computer

But it's where the bug seam to be :

When imagemagick is configured to use 8 Threads (default) => only one core will be used!

Image

When imagemagick is configured to use 4 Threads (updading policy.xml) => ALL the core will be used!

Image

do you have any idea why ?

Re: Imagemagick wand use only one single CPU

Posted: 2018-03-14T10:27:32-07:00
by snibgo
You have the same processor as me, with 8 CPUs.

You are multithreading a multithreaded application. That is, IM's resize will use multiple threads, if multiple CPUs are available. But you are doing this in multiple threads of your own. If you have 8 threads running then IM's resize will find that no CPUs are available for its own multithreading.

Your 8 threads seem to be reading the same file. I don't suppose the lock is exclusive, but this might reduce ability to multithread.

I'm no expert on multithreading, but for maximum throughput I think you should choose one multithreading method OR the other, and not attempt both methods.

Re: Imagemagick wand use only one single CPU

Posted: 2018-03-14T11:30:04-07:00
by loki5100
thanks snibgo

I try also with only one thread (i m inside an web ISAPI app)
so instead of doing :
THREAD 1
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
THREAD 2
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
THREAD 3
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
I simply do
THREAD 1
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
so only on Thread in my app is doing the block (in loop).

but i have same kind of problem, when i set MAGICK_THREAD_LIMIT from 1 to 8 only one (more or less) CPU is used and the operation is very slow.
The maximun speed is reach when i set MAGICK_THREAD_LIMIT=2
In average i have
8.5 seconds with MAGICK_THREAD_LIMIT=2,
19.1 seconds with MAGICK_THREAD_LIMIT = 8
13.5 seconds with MAGICK_THREAD_LIMIT=1

I quite complicated to understand how to configure Thread_limit :(

Re: Imagemagick wand use only one single CPU

Posted: 2018-03-14T12:01:26-07:00
by snibgo
loki5100 wrote:when i set MAGICK_THREAD_LIMIT from 1 to 8 only one (more or less) CPU is used and the operation is very slow.
The maximun speed is reach when i set MAGICK_THREAD_LIMIT=2
That is a common situation: increasing the number of threads has a "sweet spot" with maximum throughput. Beyond that, adding more threads decreases performance.

The optimum MAGICK_THREAD_LIMIT depends on many factors: the compiler, the CPU, the code within the thread, and so on. I often find the best performance comes from setting this low (1 or 2), and using other CPU cores by multithreading outside the internals of IM.

Remember that aiming for 100% CPU utilization is a bad goal. It measures the wrong thing. A better goal is throughput, eg the time required to process 1000 images.

Re: Imagemagick wand use only one single CPU

Posted: 2018-03-14T12:44:57-07:00
by loki5100
@snibgo but why in this way IM choose by default MAGICK_THREAD_LIMIT=8 :(

Re: Imagemagick wand use only one single CPU

Posted: 2018-03-14T13:59:12-07:00
by snibgo
loki5100 wrote:@snibgo but why in this way IM choose by default MAGICK_THREAD_LIMIT=8
IM is greedy. It will use all resources available (CPU, memory, disk etc) unless you tell it not to (with "-limit" or policy.xml).

Re: Imagemagick wand use only one single CPU

Posted: 2018-09-01T00:00:59-07:00
by loki5100
problem is that if you don't do fine tuning you can easily fall in situation where imagemagick will work very slowly, using only one CPU, or worse swaping between all cpu making operation very slow