Page 1 of 1
High Load after server update.. CentOS 5.2 to 5.3
Posted: 2009-06-22T14:38:20-07:00
by dognose
Hi all,
I'm seeing very high load on my server after server update (Centos 5 x86_64) today.. I haven't updated IM (ImageMagick 6.4.8-7 2009-01-22 Q16 OpenMP)
(I'm running on a multi core machine)
From the initial tests I've done, it seems that it has something to do with multiple convert operations running at the same time, although, I haven't figured out specific examples..
if I run
convert ... &
convert ... &
wait
it takes MUCH longer than if I run:
convert ...
convert ...
Does anyone know what might be going on? I'm thinking it's something with OpenMP or weird memory locking? I really don't know the internals of what's going on, but I don't know that this is a SIGNIFICANT degradation in performance.
an example of the operation I'm performing, but it really seems to be happening with loads of different processes.
convert imagein.png -bordercolor white -border 10 -bordercolor grey60 -border 1 -background none -rotate 5 -background black \( +clone -shadow 60x4+4+4 \) +swap -background transparent -flatten imageout.png
Re: High Load after server update..
Posted: 2009-06-22T16:15:48-07:00
by magick
Try disabling OpenMP and see if that makes a difference (e.g. ./configure --disable-openmp). Recent releases of ImageMagick have a number of performance improvements so one would expect your commands to run faster, not slower.
Re: High Load after server update..
Posted: 2009-06-22T19:32:11-07:00
by dognose
Well, I'd love to, but, this was my stable IM version. I always have trouble compiling the new versions.
Just wondering if anyone else is having problems with concurrency issues or whatnot with os updates.
I do appreciate the OpenMP.. so, I hope that's not fallen by the way side.
Re: High Load after server update..
Posted: 2009-06-23T08:38:59-07:00
by dognose
Found a log of what was updated... Does anything here look like it could cause trouble like this?
Jun 22 11:14:26 Updated: libgcc - 4.1.2-44.el5.x86_64
Jun 22 11:14:26 Updated: libgcc - 4.1.2-44.el5.i386
Jun 22 11:14:30 Updated: tzdata - 2009i-2.el5.noarch
Jun 22 11:14:46 Updated: glibc-common - 2.5-34.x86_64
Jun 22 11:14:48 Updated: glibc - 2.5-34.i686
Jun 22 11:14:53 Updated: glibc - 2.5-34.x86_64
Jun 22 11:14:54 Updated: bash - 3.2-24.el5.x86_64
Jun 22 11:14:54 Updated: audit-libs - 1.7.7-6.el5_3.3.x86_64
Jun 22 11:14:57 Updated: perl - 4:5.8.8-18.el5_3.1.x86_64
Jun 22 11:14:58 Updated: glib2 - 2.12.3-4.el5_3.1.i386
Jun 22 11:14:58 Updated: glib2 - 2.12.3-4.el5_3.1.x86_64
Jun 22 11:14:58 Updated: popt - 1.10.2.3-9.el5.x86_64
Jun 22 11:14:58 Updated: tcp_wrappers - 7.6-40.6.el5.x86_64
Jun 22 11:14:59 Updated: freetype - 2.2.1-21.el5_3.x86_64
Jun 22 11:14:59 Updated: libpng - 2:1.2.10-7.1.el5_3.2.x86_64
Jun 22 11:14:59 Updated: elfutils-libelf - 0.137-3.el5.x86_64
Jun 22 11:14:59 Updated: file - 4.17-15.el5_3.1.x86_64
Jun 22 11:14:59 Updated: libpng - 2:1.2.10-7.1.el5_3.2.i386
Jun 22 11:15:00 Updated: libstdc++ - 4.1.2-44.el5.x86_64
Jun 22 11:15:00 Updated: libgcrypt - 1.2.4-1.el5.x86_64
Jun 22 11:15:00 Updated: php-common - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:15:00 Updated: procps - 3.2.7-11.1.el5.x86_64
Jun 22 11:15:00 Updated: libvolume_id - 095-14.20.el5_3.x86_64
Jun 22 11:15:01 Updated: freetype - 2.2.1-21.el5_3.i386
Jun 22 11:15:01 Updated: libgcrypt - 1.2.4-1.el5.i386
Jun 22 11:15:01 Updated: audit-libs - 1.7.7-6.el5_3.3.i386
Jun 22 11:15:02 Updated: iproute - 2.6.18-9.el5.x86_64
Jun 22 11:15:02 Updated: popt - 1.10.2.3-9.el5.i386
Jun 22 11:15:02 Updated: perl-DBI - 1.52-2.el5.x86_64
Jun 22 11:15:03 Updated: lm_sensors - 2.10.7-4.el5.x86_64
Jun 22 11:15:03 Updated: libstdc++ - 4.1.2-44.el5.i386
Jun 22 11:15:03 Updated: libvolume_id - 095-14.20.el5_3.i386
Jun 22 11:15:03 Updated: php-pdo - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:15:04 Updated: iputils - 20020927-45.el5.x86_64
Jun 22 11:15:05 Updated: binutils - 2.17.50.0.6-9.el5.x86_64
Jun 22 11:15:05 Updated: ethtool - 6-2.el5.x86_64
Jun 22 11:15:05 Updated: tcp_wrappers - 7.6-40.6.el5.i386
Jun 22 11:15:05 Updated: nash - 5.1.19.6-44.1.x86_64
Jun 22 11:15:05 Updated: centos-release-notes - 5.3-3.x86_64
Jun 22 11:15:05 Updated: centos-release - 10:5-3.el5.centos.1.x86_64
Jun 22 11:15:05 Updated: setup - 2.5.58-4.el5.noarch
Jun 22 11:15:06 Updated: libwmf - 0.2.8.4-10.2.x86_64
Jun 22 11:15:06 Updated: pango - 1.14.9-5.el5.centos.x86_64
Jun 22 11:15:07 Updated: ORBit2 - 2.14.3-5.el5.x86_64
Jun 22 11:15:07 Updated: numactl - 0.9.8-7.el5.x86_64
Jun 22 11:15:07 Updated: libgomp - 4.3.2-7.el5.x86_64
Jun 22 11:15:07 Updated: lcms - 1.18-0.1.beta1.el5_3.2.x86_64
Jun 22 11:15:07 Installed: avahi-compat-libdns_sd - 0.6.16-1.el5_2.1.x86_64
Jun 22 11:15:08 Updated: crash - 4.0-7.2.3.el5.centos.1.x86_64
Jun 22 11:15:08 Updated: php-gd - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:15:08 Updated: php-xml - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:15:08 Updated: ypbind - 3:1.19-11.el5.x86_64
Jun 22 11:15:08 Updated: unzip - 5.52-3.el5.x86_64
Jun 22 11:15:09 Updated: cpuspeed - 1:1.2.1-5.el5.x86_64
Jun 22 11:15:09 Updated: tmpwatch - 2.9.7-1.1.el5.2.x86_64
Jun 22 11:15:09 Updated: traceroute - 3:2.0.1-5.el5.x86_64
Jun 22 11:15:09 Updated: libhugetlbfs - 1.3-3.el5.x86_64
Jun 22 11:15:09 Installed: giflib - 4.1.3-7.1.el5_3.1.x86_64
Jun 22 11:15:09 Updated: dosfstools - 2.11-7.el5.x86_64
Jun 22 11:15:09 Updated: rdate - 1.4-8.el5.x86_64
Jun 22 11:15:09 Updated: zip - 2.31-2.el5.x86_64
Jun 22 11:15:09 Updated: patch - 2.5.4-29.2.3.el5.x86_64
Jun 22 11:15:10 Updated: ftp - 0.17-35.el5.x86_64
Jun 22 11:15:10 Updated: libwmf - 0.2.8.4-10.2.i386
Jun 22 11:15:12 Updated: pango - 1.14.9-5.el5.centos.i386
Jun 22 11:15:19 Updated: ORBit2 - 2.14.3-5.el5.i386
Jun 22 11:15:20 Updated: numactl - 0.9.8-7.el5.i386
Jun 22 11:15:24 Updated: libgomp - 4.3.2-7.el5.i386
Jun 22 11:15:24 Updated: lcms - 1.18-0.1.beta1.el5_3.2.i386
Jun 22 11:15:25 Updated: libhugetlbfs - 1.3-3.el5.i386
Jun 22 11:15:26 Updated: filesystem - 2.4.0-2.el5.centos.x86_64
Jun 22 11:15:27 Updated: redhat-menus - 6.7.8-3.el5.noarch
Jun 22 11:15:28 Updated: redhat-logos - 4.9.99-11.el5.centos.noarch
Jun 22 11:15:31 Updated: man-pages - 2.39-12.el5.noarch
Jun 22 11:15:31 Updated: words - 3.0-9.1.noarch
Jun 22 11:15:32 Updated: libselinux - 1.33.4-5.1.el5.i386
Jun 22 11:15:34 Updated: device-mapper - 1.02.28-2.el5.i386
Jun 22 11:15:36 Updated: e2fsprogs-libs - 1.39-20.el5.i386
Jun 22 11:15:37 Updated: cryptsetup-luks - 1.0.3-4.el5.i386
Jun 22 11:15:39 Updated: parted - 1.8.1-23.el5.i386
Jun 22 11:15:44 Updated: mesa-libGL - 6.5.1-7.7.el5.i386
Jun 22 11:15:51 Updated: libselinux - 1.33.4-5.1.el5.x86_64
Jun 22 11:15:51 Updated: shadow-utils - 2:4.0.17-14.el5.x86_64
Jun 22 11:15:52 Updated: device-mapper - 1.02.28-2.el5.x86_64
Jun 22 11:15:52 Updated: e2fsprogs-libs - 1.39-20.el5.x86_64
Jun 22 11:15:52 Updated: findutils - 1:4.2.27-5.el5.x86_64
Jun 22 11:15:55 Updated: coreutils - 5.97-19.el5.x86_64
Jun 22 11:15:56 Updated: krb5-libs - 1.6.1-31.el5_3.3.x86_64
Jun 22 11:15:56 Updated: openssl - 0.9.8e-7.el5.x86_64
Jun 22 11:15:59 Updated: python - 2.4.3-24.el5.x86_64
Jun 22 11:16:13 Updated: pam - 0.99.6.2-4.el5.x86_64
Jun 22 11:16:13 Updated: openldap - 2.3.43-3.el5.x86_64
Jun 22 11:16:13 Updated: module-init-tools - 3.3-0.pre3.1.42.el5.x86_64
Jun 22 11:16:17 Updated: curl - 7.15.5-2.1.el5_3.4.x86_64
Jun 22 11:16:17 Updated: kpartx - 0.4.7-23.el5_3.4.x86_64
Jun 22 11:16:17 Updated: e2fsprogs - 1.39-20.el5.x86_64
Jun 22 11:16:21 Updated: newt - 0.52.2-12.el5.x86_64
Jun 22 11:16:21 Updated: cups-libs - 1:1.3.7-8.el5_3.6.x86_64
Jun 22 11:16:24 Updated: cryptsetup-luks - 1.0.3-4.el5.x86_64
Jun 22 11:16:24 Updated: dmraid - 1.0.0.rc13-33.el5.x86_64
Jun 22 11:16:25 Updated: device-mapper-multipath - 0.4.7-23.el5_3.4.x86_64
Jun 22 11:16:27 Updated: nscd - 2.5-34.x86_64
Jun 22 11:16:27 Updated: apr-util - 1.2.7-7.el5_3.1.x86_64
Jun 22 11:16:30 Updated: bind-libs - 30:9.3.4-10.P1.el5.x86_64
Jun 22 11:16:34 Updated: libpcap - 14:0.9.4-14.el5.x86_64
Jun 22 11:16:42 Updated: net-snmp-libs - 1:5.3.2.2-5.el5_3.1.x86_64
Jun 22 11:16:42 Updated: logrotate - 3.7.4-9.x86_64
Jun 22 11:16:42 Updated: audit-libs-python - 1.7.7-6.el5_3.3.x86_64
Jun 22 11:16:42 Updated: libselinux-python - 1.33.4-5.1.el5.x86_64
Jun 22 11:16:43 Updated: m2crypto - 0.16-6.el5.3.x86_64
Jun 22 11:16:43 Updated: device-mapper-event - 1.02.28-2.el5.x86_64
Jun 22 11:17:35 Updated: lvm2 - 2.02.40-6.el5.x86_64
Jun 22 11:17:37 Updated: ghostscript - 8.15.2-9.4.el5_3.7.x86_64
Jun 22 11:17:37 Updated: php-cli - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:17:38 Updated: pam_krb5 - 2.2.14-10.x86_64
Jun 22 11:17:39 Updated: ecryptfs-utils - 56-8.el5.x86_64
Jun 22 11:17:39 Updated: perl-DBD-MySQL - 3.0007-2.el5.x86_64
Jun 22 11:17:45 Installed: fipscheck - 1.0.3-1.el5.x86_64
Jun 22 11:17:47 Updated: nss_ldap - 253-17.el5.x86_64
Jun 22 11:17:47 Updated: neon - 0.25.5-10.el5.x86_64
Jun 22 11:17:48 Updated: alsa-lib - 1.0.17-1.el5.x86_64
Jun 22 11:17:48 Updated: parted - 1.8.1-23.el5.x86_64
Jun 22 11:17:48 Updated: libutempter - 1.1.4-4.el5.x86_64
Jun 22 11:17:50 Updated: mesa-libGL - 6.5.1-7.7.el5.x86_64
Jun 22 11:17:50 Installed: libselinux-utils - 1.33.4-5.1.el5.x86_64
Jun 22 11:17:50 Updated: bind-utils - 30:9.3.4-10.P1.el5.x86_64
Jun 22 11:17:51 Updated: gnupg - 1.4.5-14.x86_64
Jun 22 11:17:52 Updated: autofs - 1:5.0.1-0.rc2.102.x86_64
Jun 22 11:17:52 Updated: stunnel - 4.15-2.el5.1.x86_64
Jun 22 11:17:52 Updated: php-mysql - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:17:53 Updated: tcpdump - 14:3.9.4-14.el5.x86_64
Jun 22 11:17:53 Updated: pkinit-nss - 0.7.6-1.el5.x86_64
Jun 22 11:17:53 Updated: ksh - 20080202-2.el5.x86_64
Jun 22 11:17:54 Updated: audit - 1.7.7-6.el5_3.3.x86_64
Jun 22 11:17:54 Updated: psacct - 6.3.2-44.el5.x86_64
Jun 22 11:17:54 Updated: tcsh - 6.14-14.el5.x86_64
Jun 22 11:17:54 Updated: mlocate - 0.15-1.el5.1.x86_64
Jun 22 11:17:56 Updated: pam - 0.99.6.2-4.el5.i386
Jun 22 11:17:57 Updated: udev - 095-14.20.el5_3.x86_64
Jun 22 11:17:58 Updated: util-linux - 2.13-0.50.el5.x86_64
Jun 22 11:17:58 Updated: usermode - 1.88-3.el5.2.x86_64
Jun 22 11:17:59 Updated: krb5-libs - 1.6.1-31.el5_3.3.i386
Jun 22 11:17:59 Updated: authconfig - 5.3.21-5.el5.x86_64
Jun 22 11:18:00 Updated: SysVinit - 2.86-15.el5.x86_64
Jun 22 11:18:01 Updated: initscripts - 8.45.25-1.el5.centos.x86_64
Jun 22 11:18:01 Updated: mcstrans - 0.2.11-3.el5.x86_64
Jun 22 11:18:02 Updated: dhclient - 12:3.0.5-18.el5.x86_64
Jun 22 11:18:02 Updated: openssh - 4.3p2-29.el5.x86_64
Jun 22 11:18:03 Updated: httpd - 2.2.3-22.el5.centos.1.x86_64
Jun 22 11:18:04 Updated: kbd - 1.12-21.el5.x86_64
Jun 22 11:18:04 Updated: mkinitrd - 5.1.19.6-44.1.x86_64
Jun 22 11:18:04 Installed: dnsmasq - 2.45-1.el5_2.1.x86_64
Jun 22 11:18:05 Updated: ppp - 2.4.4-2.el5.x86_64
Jun 22 11:18:05 Updated: openssl - 0.9.8e-7.el5.i686
Jun 22 11:18:05 Updated: cups-libs - 1:1.3.7-8.el5_3.6.i386
Jun 22 11:18:06 Updated: openldap - 2.3.43-3.el5.i386
Jun 22 11:18:07 Updated: ghostscript - 8.15.2-9.4.el5_3.7.i386
Jun 22 11:18:07 Installed: fipscheck - 1.0.3-1.el5.i386
Jun 22 11:18:08 Updated: nss_ldap - 253-17.el5.i386
Jun 22 11:18:08 Updated: ecryptfs-utils - 56-8.el5.i386
Jun 22 11:18:08 Updated: pam_krb5 - 2.2.14-10.i386
Jun 22 11:18:09 Updated: libutempter - 1.1.4-4.el5.i386
Jun 22 11:18:09 Updated: python-urlgrabber - 3.1.0-5.el5.noarch
Jun 22 11:18:09 Updated: hwdata - 0.213.11-1.el5.noarch
Jun 22 11:18:09 Updated: php - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:18:09 Updated: openssh-clients - 4.3p2-29.el5.x86_64
Jun 22 11:18:10 Updated: openssh-server - 4.3p2-29.el5.x86_64
Jun 22 11:18:10 Updated: ipsec-tools - 0.6.5-13.el5_3.1.x86_64
Jun 22 11:18:10 Updated: ccid - 1.3.8-1.el5.x86_64
Jun 22 11:18:11 Updated: vixie-cron - 4:4.1-76.el5.x86_64
Jun 22 11:18:11 Updated: acpid - 1.0.4-7.el5_3.1.x86_64
Jun 22 11:18:11 Updated: smartmontools - 1:5.38-2.el5.x86_64
Jun 22 11:18:12 Updated: dhcpv6-client - 1.0.10-16.el5.x86_64
Jun 22 11:18:12 Updated: quota - 1:3.13-1.2.5.el5.x86_64
Jun 22 11:18:13 Updated: krb5-workstation - 1.6.1-31.el5_3.3.x86_64
Jun 22 11:18:15 Updated: nfs-utils - 1:1.0.9-40.el5.x86_64
Jun 22 11:18:17 Updated: prelink - 0.4.0-2.el5.x86_64
Jun 22 11:18:17 Updated: sudo - 1.6.9p17-3.el5_3.1.x86_64
Jun 22 11:18:18 Updated: mkinitrd - 5.1.19.6-44.1.i386
Jun 22 11:20:30 Installed: kernel - 2.6.18-128.1.14.el5.x86_64
Jun 22 11:20:31 Updated: sos - 1.7-9.16.el5_3.5.noarch
Jun 22 11:20:31 Updated: rpm-libs - 4.4.2.3-9.el5.x86_64
Jun 22 11:20:32 Updated: rpm - 4.4.2.3-9.el5.x86_64
Jun 22 11:20:32 Updated: rpm-python - 4.4.2.3-9.el5.x86_64
Jun 22 11:20:34 Updated: policycoreutils - 1.33.12-14.2.el5.x86_64
Jun 22 11:20:35 Updated: pm-utils - 0.99.3-10.el5.centos.x86_64
Jun 22 11:20:35 Updated: dbus - 1.1.2-12.el5.i386
Jun 22 11:20:35 Installed: dbus-libs - 1.1.2-12.el5.x86_64
Jun 22 11:20:36 Updated: dbus-glib - 0.73-8.el5.x86_64
Jun 22 11:20:37 Updated: hal - 0.5.8.1-38.el5.x86_64
Jun 22 11:20:37 Installed: dbus-libs - 1.1.2-12.el5.i386
Jun 22 11:20:37 Updated: kudzu - 1.2.57.1.21-1.el5.centos.x86_64
Jun 22 11:20:37 Updated: wpa_supplicant - 1:0.5.10-8.el5.x86_64
Jun 22 11:20:38 Updated: dhcdbd - 2.2-2.el5.x86_64
Jun 22 11:20:38 Updated: dbus - 1.1.2-12.el5.x86_64
Jun 22 11:20:40 Updated: cups - 1:1.3.7-8.el5_3.6.x86_64
Jun 22 11:20:41 Updated: net-snmp - 1:5.3.2.2-5.el5_3.1.x86_64
Jun 22 11:20:41 Updated: dbus-glib - 0.73-8.el5.i386
Jun 22 11:20:42 Updated: system-config-network-tui - 1.3.99.12-1.el5.noarch
Jun 22 11:20:43 Updated: hal - 0.5.8.1-38.el5.i386
Jun 22 11:20:44 Updated: selinux-policy - 2.4.6-203.el5.noarch
Jun 22 11:22:34 Updated: firstboot-tui - 1.4.27.7-1.el5.centos.x86_64
Jun 22 11:22:35 Updated: NetworkManager - 1:0.7.0-4.el5_3.x86_64
Jun 22 11:22:35 Installed: NetworkManager-glib - 1:0.7.0-4.el5_3.x86_64
Jun 22 11:22:35 Updated: yum - 3.2.19-18.el5.centos.noarch
Jun 22 11:22:36 Updated: yum-fastestmirror - 1.1.16-13.el5.centos.noarch
Re: High Load after server update.. CentOS 5.2 to 5.3
Posted: 2009-06-24T11:08:52-07:00
by dognose
Yes, it was openMP causing the problem.. how can that be fixed?
Re: High Load after server update.. CentOS 5.2 to 5.3
Posted: 2009-06-24T11:26:41-07:00
by magick
We've had really good success with OpenMP except on the one CentOS 5.3 server where it slows the process down considerably. We have several other CentOS 5.3 servers where ImageMagick built with OpenMP support works just fine (i.e. processes speed-up rather than slow-down). Right now we do not know under what conditions the slow-down occurs. The one CentOS 5.3 slow-down is on a particular virtual server. Are you on a virtual server as well?
Re: High Load after server update.. CentOS 5.2 to 5.3
Posted: 2009-06-24T12:02:17-07:00
by dognose
I've now seen the problem on 3 different servers. Two were core i7's, which generally run very fast, with 4 cores and 8 'threads' and the other was as q9400
I've also tried downgrading to centos5.2 and just updating the IM dependencies, and I still have the problem. My only theory is that it's something in the update list above.
Did one of them go from openmp safe to not?
Re: High Load after server update.. CentOS 5.2 to 5.3
Posted: 2009-07-04T11:23:59-07:00
by dognose
Hmm, this thread indicated problems with gcc-4.1.2 and openmp
http://www.nabble.com/GCC-and-OpenMP-td17220268.html
it's not easy to update the compiler though.
Re: High Load after server update.. CentOS 5.2 to 5.3
Posted: 2009-10-12T23:06:39-07:00
by balamm
So is there a solution for this yet?
Sure, I can recompile IM and disable openMP, which I just did on my own VPS and it solved the deadlock problems, but what about end users of the applications I develop, who are on shared hosting?
Telling them to tell their hosts to recompile IM isn't going to fly.
And you know how often most shared hosts update their imagemagick....
Even if you disable openMP by defaut, It could be 2 years before any new version fixes filter down to these shared hosts.
Is there no command line switch that might be used to address this for any IM version at the application script level?
Re: High Load after server update.. CentOS 5.2 to 5.3
Posted: 2009-11-18T03:33:00-07:00
by SEO
Hi!
I may be experiencing the same problem, the load of my server sometimes jumps to over 200 while normally it is at 0,5/1,0 (low traffic).
I watched TOP and noticed mass convert and montage commands.
How could I test if OpenMP is increasing the server load per command?
Best Regards,
Jan Jaap