Imagemagick wand use only one single CPU

The MagickWand interface is a new high-level C API interface to ImageMagick core methods. We discourage the use of the core methods and encourage the use of this API instead. Post MagickWand questions, bug reports, and suggestions to this forum.
Post Reply
loki5100
Posts: 22
Joined: 2018-02-24T14:51:41-07:00
Authentication code: 1152

Imagemagick wand use only one single CPU

Post by loki5100 » 2018-03-13T14:38:35-07:00

I have a multi thread app (8 threads) on a multi core machine (8 cpu). each thread is doing in parallel
NewMagickWand
MagickReadImage
MagickResizeImage
DestroyMagickWand
but imageMagick use only ONE CPU instead of 8 ! Even if I setup each thread to use a dedicated CPU with SetThreadAffinityMask ImageMagick still use always the same single CPU

How to make magickWand to use all the available CPU ? I m on windows and using imagemagick 7.
Is their any way to optimize magickwand to work in a multithread environment (inside a web server app) ?

snibgo
Posts: 10563
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Imagemagick wand use only one single CPU

Post by snibgo » 2018-03-13T14:58:56-07:00

I'm not sure what you are saying. You are doing the four operations in parallel with each other? I doubt it. The image must be entirely read before it can be resized, etc.

How large is the image? Parallelism won't start below a certain size, which depends on how many threads are available.

It might help if you paste a minimal program that shows the problem, and link to a sample input image.
snibgo's IM pages: im.snibgo.com

loki5100
Posts: 22
Joined: 2018-02-24T14:51:41-07:00
Authentication code: 1152

Re: Imagemagick wand use only one single CPU

Post by loki5100 » 2018-03-14T00:56:39-07:00

Thanks snibgo, i m doing the "block" of theses operations in parallel like below :
THREAD 1
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
THREAD 2
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
THREAD 3
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
etc..

The image is 3390x2160 i guess it's enough to start the parallelism
I try with any image, alway the same result

I use delphi so the code will be in pascal but I think you will understand it :

Code: Select all

uses ALFiles,
     ALImageMagick;

procedure _resizeImage;
var aWand: PMagickWand;
begin
  aWand := ALImageMagickLib.NewMagickWand;
  try
    if ALImageMagickLib.MagickReadImage(aWand, pansiChar(ALGetModulePath + 'sample.jpg')) <> MagickTrue then RaiseLastWandError(aWand);
    if (ALImageMagickLib.MagickResizeImage(aWand, 640, 480, LanczosFilter) <> MagickTrue) then RaiseLastWandError(aWand);
  finally
    ALImageMagickLib.DestroyMagickWand(aWand);
  end;
end;

procedure TForm1.Button1Click(Sender: TObject);
var i: integer;
begin

  for i := 0 to 8 do begin
    TThread.CreateAnonymousThread(
      procedure
      begin
        while True do
          _resizeImage
      end).Start;
  end;

end;

procedure TForm1.FormCreate(Sender: TObject);
begin
  alCreateImageMagickLibrary(string(ALGetModulePath) + 'imagemagick');
end;
And this is the CPU utilization :

Image

loki5100
Posts: 22
Joined: 2018-02-24T14:51:41-07:00
Authentication code: 1152

Re: Imagemagick wand use only one single CPU

Post by loki5100 » 2018-03-14T01:37:06-07:00

@snibgo I think i found what i will call a bug in ImageMagick

On my computer I have 4 core and 8 Logical processors

in the default policy.xml the row
<!-- <policy domain="resource" name="thread" value="4"/> -->
is commented, but when i do
convert -list resource
Resource limits:
Width: 214.748MP
Height: 214.748MP
List length: 18.4467EP
Area: 17.1373GP
Memory: 7.98018GiB
Map: 15.9604GiB
Disk: unlimited
File: 1536
Thread: 8
Throttle: 0
Time: unlimited
we saw that imageMagick is configured to use Thread: 8 that look like the number of logical processor in my computer

But it's where the bug seam to be :

When imagemagick is configured to use 8 Threads (default) => only one core will be used!

Image

When imagemagick is configured to use 4 Threads (updading policy.xml) => ALL the core will be used!

Image

do you have any idea why ?

snibgo
Posts: 10563
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Imagemagick wand use only one single CPU

Post by snibgo » 2018-03-14T10:27:32-07:00

You have the same processor as me, with 8 CPUs.

You are multithreading a multithreaded application. That is, IM's resize will use multiple threads, if multiple CPUs are available. But you are doing this in multiple threads of your own. If you have 8 threads running then IM's resize will find that no CPUs are available for its own multithreading.

Your 8 threads seem to be reading the same file. I don't suppose the lock is exclusive, but this might reduce ability to multithread.

I'm no expert on multithreading, but for maximum throughput I think you should choose one multithreading method OR the other, and not attempt both methods.
snibgo's IM pages: im.snibgo.com

loki5100
Posts: 22
Joined: 2018-02-24T14:51:41-07:00
Authentication code: 1152

Re: Imagemagick wand use only one single CPU

Post by loki5100 » 2018-03-14T11:30:04-07:00

thanks snibgo

I try also with only one thread (i m inside an web ISAPI app)
so instead of doing :
THREAD 1
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
THREAD 2
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
THREAD 3
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
I simply do
THREAD 1
while true do
_ NewMagickWand
_ MagickReadImage
_ MagickResizeImage
_ DestroyMagickWand
end
so only on Thread in my app is doing the block (in loop).

but i have same kind of problem, when i set MAGICK_THREAD_LIMIT from 1 to 8 only one (more or less) CPU is used and the operation is very slow.
The maximun speed is reach when i set MAGICK_THREAD_LIMIT=2
In average i have
8.5 seconds with MAGICK_THREAD_LIMIT=2,
19.1 seconds with MAGICK_THREAD_LIMIT = 8
13.5 seconds with MAGICK_THREAD_LIMIT=1

I quite complicated to understand how to configure Thread_limit :(

snibgo
Posts: 10563
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Imagemagick wand use only one single CPU

Post by snibgo » 2018-03-14T12:01:26-07:00

loki5100 wrote:when i set MAGICK_THREAD_LIMIT from 1 to 8 only one (more or less) CPU is used and the operation is very slow.
The maximun speed is reach when i set MAGICK_THREAD_LIMIT=2
That is a common situation: increasing the number of threads has a "sweet spot" with maximum throughput. Beyond that, adding more threads decreases performance.

The optimum MAGICK_THREAD_LIMIT depends on many factors: the compiler, the CPU, the code within the thread, and so on. I often find the best performance comes from setting this low (1 or 2), and using other CPU cores by multithreading outside the internals of IM.

Remember that aiming for 100% CPU utilization is a bad goal. It measures the wrong thing. A better goal is throughput, eg the time required to process 1000 images.
snibgo's IM pages: im.snibgo.com

loki5100
Posts: 22
Joined: 2018-02-24T14:51:41-07:00
Authentication code: 1152

Re: Imagemagick wand use only one single CPU

Post by loki5100 » 2018-03-14T12:44:57-07:00

@snibgo but why in this way IM choose by default MAGICK_THREAD_LIMIT=8 :(

snibgo
Posts: 10563
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Imagemagick wand use only one single CPU

Post by snibgo » 2018-03-14T13:59:12-07:00

loki5100 wrote:@snibgo but why in this way IM choose by default MAGICK_THREAD_LIMIT=8
IM is greedy. It will use all resources available (CPU, memory, disk etc) unless you tell it not to (with "-limit" or policy.xml).
snibgo's IM pages: im.snibgo.com

loki5100
Posts: 22
Joined: 2018-02-24T14:51:41-07:00
Authentication code: 1152

Re: Imagemagick wand use only one single CPU

Post by loki5100 » 2018-09-01T00:00:59-07:00

problem is that if you don't do fine tuning you can easily fall in situation where imagemagick will work very slowly, using only one CPU, or worse swaping between all cpu making operation very slow

Post Reply