High Load after server update.. CentOS 5.2 to 5.3

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
dognose
Posts: 265
Joined: 2005-03-08T22:16:37-07:00

High Load after server update.. CentOS 5.2 to 5.3

Post by dognose »

Hi all,

I'm seeing very high load on my server after server update (Centos 5 x86_64) today.. I haven't updated IM (ImageMagick 6.4.8-7 2009-01-22 Q16 OpenMP)
(I'm running on a multi core machine)

From the initial tests I've done, it seems that it has something to do with multiple convert operations running at the same time, although, I haven't figured out specific examples..

if I run
convert ... &
convert ... &
wait

it takes MUCH longer than if I run:

convert ...
convert ...

Does anyone know what might be going on? I'm thinking it's something with OpenMP or weird memory locking? I really don't know the internals of what's going on, but I don't know that this is a SIGNIFICANT degradation in performance.

an example of the operation I'm performing, but it really seems to be happening with loads of different processes.

convert imagein.png -bordercolor white -border 10 -bordercolor grey60 -border 1 -background none -rotate 5 -background black \( +clone -shadow 60x4+4+4 \) +swap -background transparent -flatten imageout.png
Last edited by dognose on 2009-06-23T09:38:27-07:00, edited 1 time in total.
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: High Load after server update..

Post by magick »

Try disabling OpenMP and see if that makes a difference (e.g. ./configure --disable-openmp). Recent releases of ImageMagick have a number of performance improvements so one would expect your commands to run faster, not slower.
dognose
Posts: 265
Joined: 2005-03-08T22:16:37-07:00

Re: High Load after server update..

Post by dognose »

Well, I'd love to, but, this was my stable IM version. I always have trouble compiling the new versions.

Just wondering if anyone else is having problems with concurrency issues or whatnot with os updates.

I do appreciate the OpenMP.. so, I hope that's not fallen by the way side.
dognose
Posts: 265
Joined: 2005-03-08T22:16:37-07:00

Re: High Load after server update..

Post by dognose »

Found a log of what was updated... Does anything here look like it could cause trouble like this?

Jun 22 11:14:26 Updated: libgcc - 4.1.2-44.el5.x86_64
Jun 22 11:14:26 Updated: libgcc - 4.1.2-44.el5.i386
Jun 22 11:14:30 Updated: tzdata - 2009i-2.el5.noarch
Jun 22 11:14:46 Updated: glibc-common - 2.5-34.x86_64
Jun 22 11:14:48 Updated: glibc - 2.5-34.i686
Jun 22 11:14:53 Updated: glibc - 2.5-34.x86_64
Jun 22 11:14:54 Updated: bash - 3.2-24.el5.x86_64
Jun 22 11:14:54 Updated: audit-libs - 1.7.7-6.el5_3.3.x86_64
Jun 22 11:14:57 Updated: perl - 4:5.8.8-18.el5_3.1.x86_64
Jun 22 11:14:58 Updated: glib2 - 2.12.3-4.el5_3.1.i386
Jun 22 11:14:58 Updated: glib2 - 2.12.3-4.el5_3.1.x86_64
Jun 22 11:14:58 Updated: popt - 1.10.2.3-9.el5.x86_64
Jun 22 11:14:58 Updated: tcp_wrappers - 7.6-40.6.el5.x86_64
Jun 22 11:14:59 Updated: freetype - 2.2.1-21.el5_3.x86_64
Jun 22 11:14:59 Updated: libpng - 2:1.2.10-7.1.el5_3.2.x86_64
Jun 22 11:14:59 Updated: elfutils-libelf - 0.137-3.el5.x86_64
Jun 22 11:14:59 Updated: file - 4.17-15.el5_3.1.x86_64
Jun 22 11:14:59 Updated: libpng - 2:1.2.10-7.1.el5_3.2.i386
Jun 22 11:15:00 Updated: libstdc++ - 4.1.2-44.el5.x86_64
Jun 22 11:15:00 Updated: libgcrypt - 1.2.4-1.el5.x86_64
Jun 22 11:15:00 Updated: php-common - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:15:00 Updated: procps - 3.2.7-11.1.el5.x86_64
Jun 22 11:15:00 Updated: libvolume_id - 095-14.20.el5_3.x86_64
Jun 22 11:15:01 Updated: freetype - 2.2.1-21.el5_3.i386
Jun 22 11:15:01 Updated: libgcrypt - 1.2.4-1.el5.i386
Jun 22 11:15:01 Updated: audit-libs - 1.7.7-6.el5_3.3.i386
Jun 22 11:15:02 Updated: iproute - 2.6.18-9.el5.x86_64
Jun 22 11:15:02 Updated: popt - 1.10.2.3-9.el5.i386
Jun 22 11:15:02 Updated: perl-DBI - 1.52-2.el5.x86_64
Jun 22 11:15:03 Updated: lm_sensors - 2.10.7-4.el5.x86_64
Jun 22 11:15:03 Updated: libstdc++ - 4.1.2-44.el5.i386
Jun 22 11:15:03 Updated: libvolume_id - 095-14.20.el5_3.i386
Jun 22 11:15:03 Updated: php-pdo - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:15:04 Updated: iputils - 20020927-45.el5.x86_64
Jun 22 11:15:05 Updated: binutils - 2.17.50.0.6-9.el5.x86_64
Jun 22 11:15:05 Updated: ethtool - 6-2.el5.x86_64
Jun 22 11:15:05 Updated: tcp_wrappers - 7.6-40.6.el5.i386
Jun 22 11:15:05 Updated: nash - 5.1.19.6-44.1.x86_64
Jun 22 11:15:05 Updated: centos-release-notes - 5.3-3.x86_64
Jun 22 11:15:05 Updated: centos-release - 10:5-3.el5.centos.1.x86_64
Jun 22 11:15:05 Updated: setup - 2.5.58-4.el5.noarch
Jun 22 11:15:06 Updated: libwmf - 0.2.8.4-10.2.x86_64
Jun 22 11:15:06 Updated: pango - 1.14.9-5.el5.centos.x86_64
Jun 22 11:15:07 Updated: ORBit2 - 2.14.3-5.el5.x86_64
Jun 22 11:15:07 Updated: numactl - 0.9.8-7.el5.x86_64
Jun 22 11:15:07 Updated: libgomp - 4.3.2-7.el5.x86_64
Jun 22 11:15:07 Updated: lcms - 1.18-0.1.beta1.el5_3.2.x86_64
Jun 22 11:15:07 Installed: avahi-compat-libdns_sd - 0.6.16-1.el5_2.1.x86_64
Jun 22 11:15:08 Updated: crash - 4.0-7.2.3.el5.centos.1.x86_64
Jun 22 11:15:08 Updated: php-gd - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:15:08 Updated: php-xml - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:15:08 Updated: ypbind - 3:1.19-11.el5.x86_64
Jun 22 11:15:08 Updated: unzip - 5.52-3.el5.x86_64
Jun 22 11:15:09 Updated: cpuspeed - 1:1.2.1-5.el5.x86_64
Jun 22 11:15:09 Updated: tmpwatch - 2.9.7-1.1.el5.2.x86_64
Jun 22 11:15:09 Updated: traceroute - 3:2.0.1-5.el5.x86_64
Jun 22 11:15:09 Updated: libhugetlbfs - 1.3-3.el5.x86_64
Jun 22 11:15:09 Installed: giflib - 4.1.3-7.1.el5_3.1.x86_64
Jun 22 11:15:09 Updated: dosfstools - 2.11-7.el5.x86_64
Jun 22 11:15:09 Updated: rdate - 1.4-8.el5.x86_64
Jun 22 11:15:09 Updated: zip - 2.31-2.el5.x86_64
Jun 22 11:15:09 Updated: patch - 2.5.4-29.2.3.el5.x86_64
Jun 22 11:15:10 Updated: ftp - 0.17-35.el5.x86_64
Jun 22 11:15:10 Updated: libwmf - 0.2.8.4-10.2.i386
Jun 22 11:15:12 Updated: pango - 1.14.9-5.el5.centos.i386
Jun 22 11:15:19 Updated: ORBit2 - 2.14.3-5.el5.i386
Jun 22 11:15:20 Updated: numactl - 0.9.8-7.el5.i386
Jun 22 11:15:24 Updated: libgomp - 4.3.2-7.el5.i386
Jun 22 11:15:24 Updated: lcms - 1.18-0.1.beta1.el5_3.2.i386
Jun 22 11:15:25 Updated: libhugetlbfs - 1.3-3.el5.i386
Jun 22 11:15:26 Updated: filesystem - 2.4.0-2.el5.centos.x86_64
Jun 22 11:15:27 Updated: redhat-menus - 6.7.8-3.el5.noarch
Jun 22 11:15:28 Updated: redhat-logos - 4.9.99-11.el5.centos.noarch
Jun 22 11:15:31 Updated: man-pages - 2.39-12.el5.noarch
Jun 22 11:15:31 Updated: words - 3.0-9.1.noarch
Jun 22 11:15:32 Updated: libselinux - 1.33.4-5.1.el5.i386
Jun 22 11:15:34 Updated: device-mapper - 1.02.28-2.el5.i386
Jun 22 11:15:36 Updated: e2fsprogs-libs - 1.39-20.el5.i386
Jun 22 11:15:37 Updated: cryptsetup-luks - 1.0.3-4.el5.i386
Jun 22 11:15:39 Updated: parted - 1.8.1-23.el5.i386
Jun 22 11:15:44 Updated: mesa-libGL - 6.5.1-7.7.el5.i386
Jun 22 11:15:51 Updated: libselinux - 1.33.4-5.1.el5.x86_64
Jun 22 11:15:51 Updated: shadow-utils - 2:4.0.17-14.el5.x86_64
Jun 22 11:15:52 Updated: device-mapper - 1.02.28-2.el5.x86_64
Jun 22 11:15:52 Updated: e2fsprogs-libs - 1.39-20.el5.x86_64
Jun 22 11:15:52 Updated: findutils - 1:4.2.27-5.el5.x86_64
Jun 22 11:15:55 Updated: coreutils - 5.97-19.el5.x86_64
Jun 22 11:15:56 Updated: krb5-libs - 1.6.1-31.el5_3.3.x86_64
Jun 22 11:15:56 Updated: openssl - 0.9.8e-7.el5.x86_64
Jun 22 11:15:59 Updated: python - 2.4.3-24.el5.x86_64
Jun 22 11:16:13 Updated: pam - 0.99.6.2-4.el5.x86_64
Jun 22 11:16:13 Updated: openldap - 2.3.43-3.el5.x86_64
Jun 22 11:16:13 Updated: module-init-tools - 3.3-0.pre3.1.42.el5.x86_64
Jun 22 11:16:17 Updated: curl - 7.15.5-2.1.el5_3.4.x86_64
Jun 22 11:16:17 Updated: kpartx - 0.4.7-23.el5_3.4.x86_64
Jun 22 11:16:17 Updated: e2fsprogs - 1.39-20.el5.x86_64
Jun 22 11:16:21 Updated: newt - 0.52.2-12.el5.x86_64
Jun 22 11:16:21 Updated: cups-libs - 1:1.3.7-8.el5_3.6.x86_64
Jun 22 11:16:24 Updated: cryptsetup-luks - 1.0.3-4.el5.x86_64
Jun 22 11:16:24 Updated: dmraid - 1.0.0.rc13-33.el5.x86_64
Jun 22 11:16:25 Updated: device-mapper-multipath - 0.4.7-23.el5_3.4.x86_64
Jun 22 11:16:27 Updated: nscd - 2.5-34.x86_64
Jun 22 11:16:27 Updated: apr-util - 1.2.7-7.el5_3.1.x86_64
Jun 22 11:16:30 Updated: bind-libs - 30:9.3.4-10.P1.el5.x86_64
Jun 22 11:16:34 Updated: libpcap - 14:0.9.4-14.el5.x86_64
Jun 22 11:16:42 Updated: net-snmp-libs - 1:5.3.2.2-5.el5_3.1.x86_64
Jun 22 11:16:42 Updated: logrotate - 3.7.4-9.x86_64
Jun 22 11:16:42 Updated: audit-libs-python - 1.7.7-6.el5_3.3.x86_64
Jun 22 11:16:42 Updated: libselinux-python - 1.33.4-5.1.el5.x86_64
Jun 22 11:16:43 Updated: m2crypto - 0.16-6.el5.3.x86_64
Jun 22 11:16:43 Updated: device-mapper-event - 1.02.28-2.el5.x86_64
Jun 22 11:17:35 Updated: lvm2 - 2.02.40-6.el5.x86_64
Jun 22 11:17:37 Updated: ghostscript - 8.15.2-9.4.el5_3.7.x86_64
Jun 22 11:17:37 Updated: php-cli - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:17:38 Updated: pam_krb5 - 2.2.14-10.x86_64
Jun 22 11:17:39 Updated: ecryptfs-utils - 56-8.el5.x86_64
Jun 22 11:17:39 Updated: perl-DBD-MySQL - 3.0007-2.el5.x86_64
Jun 22 11:17:45 Installed: fipscheck - 1.0.3-1.el5.x86_64
Jun 22 11:17:47 Updated: nss_ldap - 253-17.el5.x86_64
Jun 22 11:17:47 Updated: neon - 0.25.5-10.el5.x86_64
Jun 22 11:17:48 Updated: alsa-lib - 1.0.17-1.el5.x86_64
Jun 22 11:17:48 Updated: parted - 1.8.1-23.el5.x86_64
Jun 22 11:17:48 Updated: libutempter - 1.1.4-4.el5.x86_64
Jun 22 11:17:50 Updated: mesa-libGL - 6.5.1-7.7.el5.x86_64
Jun 22 11:17:50 Installed: libselinux-utils - 1.33.4-5.1.el5.x86_64
Jun 22 11:17:50 Updated: bind-utils - 30:9.3.4-10.P1.el5.x86_64
Jun 22 11:17:51 Updated: gnupg - 1.4.5-14.x86_64
Jun 22 11:17:52 Updated: autofs - 1:5.0.1-0.rc2.102.x86_64
Jun 22 11:17:52 Updated: stunnel - 4.15-2.el5.1.x86_64
Jun 22 11:17:52 Updated: php-mysql - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:17:53 Updated: tcpdump - 14:3.9.4-14.el5.x86_64
Jun 22 11:17:53 Updated: pkinit-nss - 0.7.6-1.el5.x86_64
Jun 22 11:17:53 Updated: ksh - 20080202-2.el5.x86_64
Jun 22 11:17:54 Updated: audit - 1.7.7-6.el5_3.3.x86_64
Jun 22 11:17:54 Updated: psacct - 6.3.2-44.el5.x86_64
Jun 22 11:17:54 Updated: tcsh - 6.14-14.el5.x86_64
Jun 22 11:17:54 Updated: mlocate - 0.15-1.el5.1.x86_64
Jun 22 11:17:56 Updated: pam - 0.99.6.2-4.el5.i386
Jun 22 11:17:57 Updated: udev - 095-14.20.el5_3.x86_64
Jun 22 11:17:58 Updated: util-linux - 2.13-0.50.el5.x86_64
Jun 22 11:17:58 Updated: usermode - 1.88-3.el5.2.x86_64
Jun 22 11:17:59 Updated: krb5-libs - 1.6.1-31.el5_3.3.i386
Jun 22 11:17:59 Updated: authconfig - 5.3.21-5.el5.x86_64
Jun 22 11:18:00 Updated: SysVinit - 2.86-15.el5.x86_64
Jun 22 11:18:01 Updated: initscripts - 8.45.25-1.el5.centos.x86_64
Jun 22 11:18:01 Updated: mcstrans - 0.2.11-3.el5.x86_64
Jun 22 11:18:02 Updated: dhclient - 12:3.0.5-18.el5.x86_64
Jun 22 11:18:02 Updated: openssh - 4.3p2-29.el5.x86_64
Jun 22 11:18:03 Updated: httpd - 2.2.3-22.el5.centos.1.x86_64
Jun 22 11:18:04 Updated: kbd - 1.12-21.el5.x86_64
Jun 22 11:18:04 Updated: mkinitrd - 5.1.19.6-44.1.x86_64
Jun 22 11:18:04 Installed: dnsmasq - 2.45-1.el5_2.1.x86_64
Jun 22 11:18:05 Updated: ppp - 2.4.4-2.el5.x86_64
Jun 22 11:18:05 Updated: openssl - 0.9.8e-7.el5.i686
Jun 22 11:18:05 Updated: cups-libs - 1:1.3.7-8.el5_3.6.i386
Jun 22 11:18:06 Updated: openldap - 2.3.43-3.el5.i386
Jun 22 11:18:07 Updated: ghostscript - 8.15.2-9.4.el5_3.7.i386
Jun 22 11:18:07 Installed: fipscheck - 1.0.3-1.el5.i386
Jun 22 11:18:08 Updated: nss_ldap - 253-17.el5.i386
Jun 22 11:18:08 Updated: ecryptfs-utils - 56-8.el5.i386
Jun 22 11:18:08 Updated: pam_krb5 - 2.2.14-10.i386
Jun 22 11:18:09 Updated: libutempter - 1.1.4-4.el5.i386
Jun 22 11:18:09 Updated: python-urlgrabber - 3.1.0-5.el5.noarch
Jun 22 11:18:09 Updated: hwdata - 0.213.11-1.el5.noarch
Jun 22 11:18:09 Updated: php - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:18:09 Updated: openssh-clients - 4.3p2-29.el5.x86_64
Jun 22 11:18:10 Updated: openssh-server - 4.3p2-29.el5.x86_64
Jun 22 11:18:10 Updated: ipsec-tools - 0.6.5-13.el5_3.1.x86_64
Jun 22 11:18:10 Updated: ccid - 1.3.8-1.el5.x86_64
Jun 22 11:18:11 Updated: vixie-cron - 4:4.1-76.el5.x86_64
Jun 22 11:18:11 Updated: acpid - 1.0.4-7.el5_3.1.x86_64
Jun 22 11:18:11 Updated: smartmontools - 1:5.38-2.el5.x86_64
Jun 22 11:18:12 Updated: dhcpv6-client - 1.0.10-16.el5.x86_64
Jun 22 11:18:12 Updated: quota - 1:3.13-1.2.5.el5.x86_64
Jun 22 11:18:13 Updated: krb5-workstation - 1.6.1-31.el5_3.3.x86_64
Jun 22 11:18:15 Updated: nfs-utils - 1:1.0.9-40.el5.x86_64
Jun 22 11:18:17 Updated: prelink - 0.4.0-2.el5.x86_64
Jun 22 11:18:17 Updated: sudo - 1.6.9p17-3.el5_3.1.x86_64
Jun 22 11:18:18 Updated: mkinitrd - 5.1.19.6-44.1.i386
Jun 22 11:20:30 Installed: kernel - 2.6.18-128.1.14.el5.x86_64
Jun 22 11:20:31 Updated: sos - 1.7-9.16.el5_3.5.noarch
Jun 22 11:20:31 Updated: rpm-libs - 4.4.2.3-9.el5.x86_64
Jun 22 11:20:32 Updated: rpm - 4.4.2.3-9.el5.x86_64
Jun 22 11:20:32 Updated: rpm-python - 4.4.2.3-9.el5.x86_64
Jun 22 11:20:34 Updated: policycoreutils - 1.33.12-14.2.el5.x86_64
Jun 22 11:20:35 Updated: pm-utils - 0.99.3-10.el5.centos.x86_64
Jun 22 11:20:35 Updated: dbus - 1.1.2-12.el5.i386
Jun 22 11:20:35 Installed: dbus-libs - 1.1.2-12.el5.x86_64
Jun 22 11:20:36 Updated: dbus-glib - 0.73-8.el5.x86_64
Jun 22 11:20:37 Updated: hal - 0.5.8.1-38.el5.x86_64
Jun 22 11:20:37 Installed: dbus-libs - 1.1.2-12.el5.i386
Jun 22 11:20:37 Updated: kudzu - 1.2.57.1.21-1.el5.centos.x86_64
Jun 22 11:20:37 Updated: wpa_supplicant - 1:0.5.10-8.el5.x86_64
Jun 22 11:20:38 Updated: dhcdbd - 2.2-2.el5.x86_64
Jun 22 11:20:38 Updated: dbus - 1.1.2-12.el5.x86_64
Jun 22 11:20:40 Updated: cups - 1:1.3.7-8.el5_3.6.x86_64
Jun 22 11:20:41 Updated: net-snmp - 1:5.3.2.2-5.el5_3.1.x86_64
Jun 22 11:20:41 Updated: dbus-glib - 0.73-8.el5.i386
Jun 22 11:20:42 Updated: system-config-network-tui - 1.3.99.12-1.el5.noarch
Jun 22 11:20:43 Updated: hal - 0.5.8.1-38.el5.i386
Jun 22 11:20:44 Updated: selinux-policy - 2.4.6-203.el5.noarch
Jun 22 11:22:34 Updated: firstboot-tui - 1.4.27.7-1.el5.centos.x86_64
Jun 22 11:22:35 Updated: NetworkManager - 1:0.7.0-4.el5_3.x86_64
Jun 22 11:22:35 Installed: NetworkManager-glib - 1:0.7.0-4.el5_3.x86_64
Jun 22 11:22:35 Updated: yum - 3.2.19-18.el5.centos.noarch
Jun 22 11:22:36 Updated: yum-fastestmirror - 1.1.16-13.el5.centos.noarch
dognose
Posts: 265
Joined: 2005-03-08T22:16:37-07:00

Re: High Load after server update.. CentOS 5.2 to 5.3

Post by dognose »

Yes, it was openMP causing the problem.. how can that be fixed?
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: High Load after server update.. CentOS 5.2 to 5.3

Post by magick »

We've had really good success with OpenMP except on the one CentOS 5.3 server where it slows the process down considerably. We have several other CentOS 5.3 servers where ImageMagick built with OpenMP support works just fine (i.e. processes speed-up rather than slow-down). Right now we do not know under what conditions the slow-down occurs. The one CentOS 5.3 slow-down is on a particular virtual server. Are you on a virtual server as well?
dognose
Posts: 265
Joined: 2005-03-08T22:16:37-07:00

Re: High Load after server update.. CentOS 5.2 to 5.3

Post by dognose »

I've now seen the problem on 3 different servers. Two were core i7's, which generally run very fast, with 4 cores and 8 'threads' and the other was as q9400

I've also tried downgrading to centos5.2 and just updating the IM dependencies, and I still have the problem. My only theory is that it's something in the update list above.
Did one of them go from openmp safe to not?
dognose
Posts: 265
Joined: 2005-03-08T22:16:37-07:00

Re: High Load after server update.. CentOS 5.2 to 5.3

Post by dognose »

Hmm, this thread indicated problems with gcc-4.1.2 and openmp

http://www.nabble.com/GCC-and-OpenMP-td17220268.html

it's not easy to update the compiler though.
User avatar
balamm
Posts: 4
Joined: 2009-08-22T11:02:08-07:00
Authentication code: 8675309

Re: High Load after server update.. CentOS 5.2 to 5.3

Post by balamm »

So is there a solution for this yet?

Sure, I can recompile IM and disable openMP, which I just did on my own VPS and it solved the deadlock problems, but what about end users of the applications I develop, who are on shared hosting?

Telling them to tell their hosts to recompile IM isn't going to fly.

And you know how often most shared hosts update their imagemagick....
Even if you disable openMP by defaut, It could be 2 years before any new version fixes filter down to these shared hosts.

Is there no command line switch that might be used to address this for any IM version at the application script level?
SEO

Re: High Load after server update.. CentOS 5.2 to 5.3

Post by SEO »

Hi!

I may be experiencing the same problem, the load of my server sometimes jumps to over 200 while normally it is at 0,5/1,0 (low traffic).

I watched TOP and noticed mass convert and montage commands.

How could I test if OpenMP is increasing the server load per command?

Best Regards,
Jan Jaap
Post Reply