Package: util-linux Version: 2.20.1-5.8 Severity: important LAVA has systems which prepare and manipulate images for multiple test jobs in parallel - some of these calls involve using sfdisk to manipulate partitions when generating a new or updated image.
We are regularly seeing deadlocks involving sfdisk with umount operations waiting indefinitely for a call to sfdisk to complete. Although these operations are on the same filesystem, each operation is on a different mountpoint and a different filesystem device. (Typically, /var/lib/lava/dispatcher/tmp/tmpQeJXVm/foo.) There are also NFS mounts on some affected machines. LAVA has to manage development devices which do not always behave in standard ways, so there are times when a misbehaving USB device can cause errors in dmesg and problems umounting for those devices. However, the presence of calls to 'sync' complicates the resolution of these errors by forcing *all* filesystem buffers to be sync'd (local and remote) not just the filesystem being manipulated. This therefore causes a single problematic device to be able to stall sfdisk operations on multiple different devices. This causes test jobs to wait unnecessarily for admin intervention, leaving devices waiting instead of running more jobs from the queue. I've attached a short log showing the ps output related to this issue for one specific instance - currently, this is coming up a few times a week. Please remove the unnecessary (and unwarranted) calls to 'sync' inside the sfdisk binary in the reread_disk_partition() and do_fdisk() functions: I'm going to test with this change, commenting out the unnecessary calls, with a local NMU build. --- orig/util-linux-2.20.1/fdisk/sfdisk.c 2014-07-23 14:32:38.000000000 +0100 +++ util-linux-2.20.1/fdisk/sfdisk.c 2014-07-23 14:18:05.158740017 +0100 @@ -817,7 +817,7 @@ reread_disk_partition(char *dev, int fd) { printf(_("Re-reading the partition table ...\n")); fflush(stdout); - sync(); +// sync(); if (reread_ioctl(fd) && is_blockdev(fd)) { do_warn(_("The command to re-read the partition table failed.\n" @@ -3257,6 +3257,6 @@ "to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1\n" "(See fdisk(8).)\n")); - sync(); /* superstition */ + //sync(); /* superstition */ exit(exit_status); } -- System Information: Debian Release: jessie/sid APT prefers unstable APT policy: (500, 'unstable'), (500, 'stable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 armhf Kernel: Linux 3.14-1-amd64 (SMP w/4 CPU cores) Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_GB.UTF-8) Shell: /bin/sh linked to /bin/dash Versions of packages util-linux depends on: ii debconf [debconf-2.0] 1.5.53 ii dpkg 1.17.10 ii initscripts 2.88dsf-53.2 ii install-info 5.2.0.dfsg.1-4 ii libblkid1 2.20.1-5.8 ii libc6 2.19-7 ii libncurses5 5.9+20140712-2 ii libselinux1 2.3-1 ii libslang2 2.2.4-17 ii libtinfo5 5.9+20140712-2 ii libuuid1 2.20.1-5.8 ii lsb-base 4.1+Debian13 ii tzdata 2014e-1 ii zlib1g 1:1.2.8.dfsg-1 util-linux recommends no packages. Versions of packages util-linux suggests: ii dosfstools 3.0.26-2 ii kbd 1.15.5-1 ii util-linux-locales 2.20.1-5.8 -- debconf information excluded
admin@dispatcher01:~$ ps auxf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 2 0.0 0.0 0 0 ? S Jul18 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S Jul18 0:29 \_ [ksoftirqd/0] root 5 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/0:0H] root 7 0.0 0.0 0 0 ? S Jul18 1:13 \_ [rcu_sched] root 8 0.0 0.0 0 0 ? S Jul18 0:00 \_ [rcu_bh] root 9 0.0 0.0 0 0 ? S Jul18 0:00 \_ [migration/0] root 10 0.0 0.0 0 0 ? S Jul18 0:01 \_ [watchdog/0] root 11 0.0 0.0 0 0 ? S Jul18 0:01 \_ [watchdog/1] root 12 0.0 0.0 0 0 ? S Jul18 0:00 \_ [migration/1] root 13 0.0 0.0 0 0 ? S Jul18 0:10 \_ [ksoftirqd/1] root 15 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/1:0H] root 16 0.0 0.0 0 0 ? S Jul18 0:00 \_ [watchdog/2] root 17 0.0 0.0 0 0 ? S Jul18 0:00 \_ [migration/2] root 18 0.0 0.0 0 0 ? S Jul18 0:08 \_ [ksoftirqd/2] root 20 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/2:0H] root 21 0.0 0.0 0 0 ? S Jul18 0:00 \_ [watchdog/3] root 22 0.0 0.0 0 0 ? S Jul18 0:00 \_ [migration/3] root 23 0.0 0.0 0 0 ? S Jul18 0:09 \_ [ksoftirqd/3] root 25 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/3:0H] root 26 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [khelper] root 27 0.0 0.0 0 0 ? S Jul18 0:00 \_ [kdevtmpfs] root 28 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [netns] root 29 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [writeback] root 30 0.0 0.0 0 0 ? SN Jul18 0:00 \_ [ksmd] root 31 0.0 0.0 0 0 ? SN Jul18 0:00 \_ [khugepaged] root 32 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kintegrityd] root 33 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [bioset] root 34 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [crypto] root 35 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kblockd] root 37 0.0 0.0 0 0 ? S Jul18 0:00 \_ [khungtaskd] root 38 0.0 0.0 0 0 ? S Jul18 0:01 \_ [kswapd0] root 39 0.0 0.0 0 0 ? S Jul18 0:00 \_ [fsnotify_mark] root 44 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kthrotld] root 45 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [ipv6_addrconf] root 47 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [deferwq] root 124 0.0 0.0 0 0 ? S Jul18 0:30 \_ [khubd] root 133 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [ata_sff] root 179 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_0] root 180 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_0] root 181 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_1] root 182 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_1] root 187 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_2] root 188 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_2] root 189 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_3] root 190 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_3] root 208 0.0 0.0 0 0 ? S< Jul18 0:08 \_ [kworker/0:1H] root 217 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [md] root 223 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [bioset] root 224 0.0 0.0 0 0 ? S Jul18 0:21 \_ [md0_raid1] root 247 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [bioset] root 256 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_4] root 257 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_4] root 258 0.0 0.0 0 0 ? S Jul18 0:00 \_ [usb-storage] root 272 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_6] root 273 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_6] root 274 0.0 0.0 0 0 ? S Jul18 0:00 \_ [usb-storage] root 277 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_7] root 278 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_7] root 279 0.0 0.0 0 0 ? S Jul18 0:00 \_ [usb-storage] root 282 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/1:1H] root 283 0.0 0.0 0 0 ? S Jul18 0:15 \_ [jbd2/md0-8] root 284 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [ext4-rsv-conver] root 285 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/2:1H] root 429 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/3:1H] root 558 0.0 0.0 0 0 ? SN Jul18 0:12 \_ [kipmi0] root 559 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kpsmoused] root 578 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kvm-irqfd-clean] root 1899 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [rpciod] root 1901 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [nfsiod] root 2501 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [nfsd4] root 2502 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [nfsd4_callbacks] root 2503 0.0 0.0 0 0 ? S Jul18 0:00 \_ [lockd] root 2506 0.0 0.0 0 0 ? S Jul18 0:05 \_ [nfsd] root 2507 0.0 0.0 0 0 ? S Jul18 0:02 \_ [nfsd] root 2508 0.0 0.0 0 0 ? S Jul18 0:03 \_ [nfsd] root 2509 0.0 0.0 0 0 ? S Jul18 0:02 \_ [nfsd] root 2510 0.0 0.0 0 0 ? S Jul18 0:07 \_ [nfsd] root 2511 0.0 0.0 0 0 ? S Jul18 0:03 \_ [nfsd] root 2512 0.0 0.0 0 0 ? S Jul18 0:05 \_ [nfsd] root 2513 0.0 0.0 0 0 ? S Jul18 0:04 \_ [nfsd] root 3289 0.0 0.0 0 0 ? S Jul18 0:00 \_ [kauditd] root 30791 0.0 0.0 0 0 ? S Jul22 0:00 \_ [scsi_eh_259] root 30792 0.0 0.0 0 0 ? S< Jul22 0:00 \_ [scsi_tmf_259] root 30793 0.0 0.0 0 0 ? S Jul22 0:00 \_ [usb-storage] root 23489 0.0 0.0 0 0 ? D Jul22 0:00 \_ [scsi_eh_370] root 23490 0.0 0.0 0 0 ? S< Jul22 0:00 \_ [scsi_tmf_370] root 23491 0.0 0.0 0 0 ? S Jul22 0:00 \_ [usb-storage] root 2327 0.0 0.0 0 0 ? S 01:55 0:00 \_ [kworker/u128:0] root 14405 0.0 0.0 0 0 ? S 02:57 0:00 \_ [kworker/u128:1] root 28395 0.0 0.0 0 0 ? S 08:55 0:00 \_ [kworker/0:0] root 29538 0.0 0.0 0 0 ? S 09:01 0:00 \_ [kworker/3:3] root 29798 0.0 0.0 0 0 ? S 09:02 0:00 \_ [kworker/1:2] root 30317 0.0 0.0 0 0 ? S 09:05 0:00 \_ [kworker/2:1] root 31573 0.0 0.0 0 0 ? S 09:12 0:00 \_ [kworker/2:2] root 31862 0.0 0.0 0 0 ? S 09:13 0:00 \_ [kworker/1:1] root 31876 0.0 0.0 0 0 ? S 09:13 0:00 \_ [kworker/0:1] root 32244 0.0 0.0 0 0 ? S 09:15 0:00 \_ [kworker/3:2] root 32824 0.0 0.0 0 0 ? S 09:18 0:00 \_ [kworker/2:0] root 32949 0.0 0.0 0 0 ? S 09:19 0:00 \_ [kworker/1:0] root 32999 0.0 0.0 0 0 ? S 09:19 0:00 \_ [kworker/0:2] root 33226 0.0 0.0 0 0 ? S 09:20 0:00 \_ [kworker/3:0] root 1 0.0 0.0 13224 816 ? Ss Jul18 0:03 init [2] root 442 0.0 0.0 42240 1420 ? Ss Jul18 0:02 udevd --daemon root 1866 0.0 0.0 23272 1152 ? Ss Jul18 0:00 /sbin/rpcbind -w statd 1894 0.0 0.0 23476 1328 ? Ss Jul18 0:00 /sbin/rpc.statd root 1908 0.0 0.0 25428 424 ? Ss Jul18 0:00 /usr/sbin/rpc.idmapd root 2128 0.0 0.3 308044 37180 ? Ssl Jul18 0:25 /sbin/zfs-fuse --pidfile /var/run/zfs-fuse.pid root 2395 0.0 0.0 6448 368 ? S Jul18 0:00 /usr/sbin/ser2net -c /etc/ser2net.conf -P /run/ser2net.pid root 2477 0.0 0.0 258976 8456 ? Ssl Jul18 0:10 /usr/sbin/rsyslogd root 2520 0.0 0.0 28356 1948 ? Ss Jul18 0:00 /usr/sbin/rpc.mountd --manage-gids root 2535 0.0 0.0 4240 644 ? Ss Jul18 0:00 /usr/sbin/acpid root 2557 0.0 0.0 39600 7744 ? S Jul18 0:00 /usr/bin/python /usr/bin/lava-coordinator --loglevel=DEBUG root 2598 0.0 0.0 88380 3992 ? Ss Jul18 0:07 /usr/sbin/apache2 -k start lavaser+ 14425 0.0 0.0 213496 5688 ? Sl Jul22 0:01 \_ /usr/sbin/apache2 -k start www-data 14426 0.0 0.0 379716 4088 ? Sl Jul22 0:08 \_ /usr/sbin/apache2 -k start www-data 14427 0.0 0.0 379780 4208 ? Sl Jul22 0:11 \_ /usr/sbin/apache2 -k start root 2705 0.0 0.0 13264 828 ? Ss Jul18 0:00 /sbin/mdadm --monitor --pid-file /run/mdadm/monitor.pid --daemonise --scan --syslog root 2795 0.0 0.0 14984 316 ? Ss Jul18 0:00 /usr/sbin/in.tftpd --listen --user tftp --address 0.0.0.0:69 --secure /var/lib/lava/dispatcher/tmp/ root 2821 0.0 0.0 25564 1420 ? S Jul18 0:00 /usr/sbin/vsftpd ntp 2842 0.0 0.0 33308 2216 ? Ss Jul18 0:12 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 107:111 root 2865 0.0 0.0 25240 1072 ? Ss Jul18 0:00 /usr/sbin/cron daemon 2889 0.0 0.0 19000 164 ? Ss Jul18 0:00 /usr/sbin/atd root 2907 0.0 0.0 54976 1288 ? Ss Jul18 0:06 /usr/sbin/sshd root 33251 0.0 0.0 97256 4224 ? Ss 09:20 0:00 \_ sshd: admin [priv] matthew+ 33256 0.0 0.0 97256 1968 ? S 09:20 0:00 \_ sshd: admin@pts/13 matthew+ 33257 0.9 0.0 21288 3844 pts/13 Ss 09:20 0:00 \_ -bash matthew+ 33397 0.0 0.0 19244 1444 pts/13 R+ 09:20 0:00 \_ ps auxf message+ 2926 0.0 0.0 41620 1256 ? Ss Jul18 0:24 /usr/bin/dbus-daemon --system root 2945 0.0 0.3 496988 46144 ? Sl Jul18 1:08 /usr/bin/python /usr/bin/salt-minion -d root 2948 0.0 0.1 378376 12512 ? Sl Jul18 0:01 /usr/sbin/libvirtd -d postgres 2990 0.0 0.1 220668 15760 ? S Jul18 0:02 /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf postgres 3028 0.0 0.0 220668 1844 ? Ss Jul18 0:00 \_ postgres: checkpointer process postgres 3029 0.0 0.0 220668 2640 ? Ss Jul18 0:01 \_ postgres: writer process postgres 3030 0.0 0.0 220668 1632 ? Ss Jul18 0:01 \_ postgres: wal writer process postgres 3031 0.0 0.0 221476 3004 ? Ss Jul18 0:02 \_ postgres: autovacuum launcher process postgres 3032 0.0 0.0 76344 1740 ? Ss Jul18 0:06 \_ postgres: stats collector process lavaser+ 3346 0.0 0.0 46436 3364 ? S Jul18 2:00 ssh -x -a -oClearAllForwardings=yes -oPasswordAuthentication=no -oStrictHostKeyChecking=no -2 lavaser...@validation.linaro.org -s sftp Debian-+ 3349 0.0 0.0 52804 1092 ? Ss Jul18 0:00 /usr/sbin/exim4 -bd -q30m lavaser+ 3371 0.0 0.0 387828 912 ? Ssl Jul18 1:49 sshfs -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o allow_root -o reconnect lavaser...@validation.linaro.org:/var/lib/lava-server//default/media /var/lib/lava- root 3425 0.0 0.0 16452 944 tty1 Ss+ Jul18 0:00 /sbin/getty 38400 tty1 root 3426 0.0 0.0 16452 940 tty2 Ss+ Jul18 0:00 /sbin/getty 38400 tty2 root 3427 0.0 0.0 16452 940 tty3 Ss+ Jul18 0:00 /sbin/getty 38400 tty3 root 3428 0.0 0.0 16452 936 tty4 Ss+ Jul18 0:00 /sbin/getty 38400 tty4 root 3429 0.0 0.0 16452 932 tty5 Ss+ Jul18 0:00 /sbin/getty 38400 tty5 root 3430 0.0 0.0 16452 944 tty6 Ss+ Jul18 0:00 /sbin/getty 38400 tty6 root 3507 0.0 0.0 37492 2160 ? S Jul18 0:18 /lib/systemd/systemd-logind codehelp 5119 0.0 0.0 25816 1644 ? Ss Jul18 0:00 SCREEN codehelp 5120 0.0 0.0 21384 4052 pts/3 Ss+ Jul18 0:00 \_ /bin/bash root 5428 0.0 0.0 498200 2488 ? SNl Jul18 4:45 adb -P 5037 fork-server server root 30360 0.0 0.5 366504 62540 ? Ssl Jul22 0:27 /usr/bin/python /usr/bin/lava-server manage schedulermonitor lava-dispatch wg-02 /tmp/tmpty846N -l info -f /var/log/lava-server/lava-scheduler.log root 30366 0.0 0.4 243260 50136 ? Sl Jul22 0:13 \_ /usr/bin/python /usr/bin/lava dispatch /tmp/tmpBdJJBO --output-dir /var/lib/lava-server/default/media/job-output/job-142527 root 30371 0.0 0.0 4220 352 ? S Jul22 0:01 \_ tee /var/lib/lava-server/default/media/job-output/job-142527/output.txt root 30372 0.0 0.0 25424 1484 pts/7 Ss+ Jul22 0:00 \_ /usr/bin/telnet serial01 7021 root 55936 0.0 0.0 8488 636 ? S Jul22 0:00 \_ timeout 72000s lava-android-test run cts -s 10.254.30.129:5555 -o /tmp/tmp5GdUAl/cts.2014-07-22T14:44:40Z.bundle -O --cts-pkg android-cts-armv7-master.zip --plan CTS root 55937 0.0 0.2 340112 28472 ? Sl Jul22 0:12 \_ /usr/bin/python /usr/bin/lava-android-test run cts -s 10.254.30.129:5555 -o /tmp/tmp5GdUAl/cts.2014-07-22T14:44:40Z.bundle -O --cts-pkg android-cts-armv7-master. root 56118 0.0 0.0 4312 640 ? S Jul22 0:00 \_ /bin/sh -c python /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/cts/cts_wrapper.py 10.254.30.129:5555 --cts-pkg android-cts-armv7-maste root 56120 0.3 0.1 274024 14968 ? S Jul22 4:13 \_ python /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/cts/cts_wrapper.py 10.254.30.129:5555 --cts-pkg android-cts-armv7-master.zip - root 56249 0.0 0.0 9412 1348 pts/6 Ss+ Jul22 0:00 \_ /bin/bash ./android-cts/tools/cts-tradefed run cts --serial 10.254.30.129:5555 --plan CTS --disable-reboot root 56268 0.7 0.6 5039960 79472 pts/6 Sl+ Jul22 7:26 \_ java -cp :./android-cts/tools/../../android-cts/tools/ddmlib-prebuilt.jar:./android-cts/tools/../../android-cts/tools/tradefed-prebuilt.jar:./and root 56128 0.0 0.0 9396 648 ? S Jul22 0:00 bash /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/cts/cts_redirect.sh kmsg.log adb -s 10.254.30.129:5555 shell cat /proc/kmsg root 56129 0.0 0.0 14700 1280 ? S Jul22 0:00 \_ adb -s 10.254.30.129:5555 shell cat /proc/kmsg root 56135 0.0 0.0 9400 800 ? S Jul22 0:00 bash /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/cts/cts_redirect.sh logcat.log adb -s 10.254.30.129:5555 logcat -c; adb -s 10.254.30.129:5555 logcat root 56140 0.0 0.0 14832 1280 ? S Jul22 0:00 \_ adb -s 10.254.30.129:5555 logcat -v time root 14523 0.0 0.0 36720 7136 ? S Jul22 0:00 /usr/bin/python /usr/bin/lava-daemon manage --instance-template=/etc/lava-server/{{filename}}.conf --instance=production scheduler --logfile /var/log/lava-server/lava-schedu root 14524 5.4 7.1 1180884 879940 ? Sl Jul22 52:50 \_ /usr/bin/python /usr/bin/lava-server manage --instance-template=/etc/lava-server/{{filename}}.conf --instance=production scheduler --logfile /var/log/lava-server/lava-sc root 23343 0.0 0.5 366576 62988 ? Ssl Jul22 0:17 \_ /usr/bin/python /usr/bin/lava-server manage schedulermonitor lava-dispatch vexpress-tc2-benchmark-01 /tmp/tmpPAujAu -l info -f /var/log/lava-server/lava-scheduler.lo root 23358 0.0 0.4 169400 49556 ? S Jul22 0:01 | \_ /usr/bin/python /usr/bin/lava dispatch /tmp/tmpO_beXH --output-dir /var/lib/lava-server/default/media/job-output/job-142570 root 23403 0.0 0.0 4220 348 ? S Jul22 0:00 | \_ tee /var/lib/lava-server/default/media/job-output/job-142570/output.txt root 23407 0.0 0.0 25424 1484 pts/1 Ss+ Jul22 0:00 | \_ /usr/bin/telnet serial01 7002 root 23541 0.0 0.0 4312 636 ? SN Jul22 0:00 | \_ sh -c umount /var/lib/lava/dispatcher/tmp/tmpq7WLmG/vexpress-usb root 23542 0.0 0.0 19744 1000 ? DN Jul22 0:00 | \_ umount /var/lib/lava/dispatcher/tmp/tmpq7WLmG/vexpress-usb root 47223 0.0 0.5 366580 62484 ? Ssl 00:10 0:10 \_ /usr/bin/python /usr/bin/lava-server manage schedulermonitor lava-dispatch arndale03 /tmp/tmp85NZjU -l info -f /var/log/lava-server/lava-scheduler.log root 47235 0.0 0.4 169524 49636 ? S 00:10 0:03 | \_ /usr/bin/python /usr/bin/lava dispatch /tmp/tmp0e1irr --output-dir /var/lib/lava-server/default/media/job-output/job-142697 root 47240 0.0 0.0 4220 352 ? S 00:10 0:00 | \_ tee /var/lib/lava-server/default/media/job-output/job-142697/output.txt root 47244 0.0 0.0 25424 1484 pts/0 Ss+ 00:10 0:00 | \_ /usr/bin/telnet serial1 7016 root 47565 0.0 0.0 54648 1868 pts/5 Ss+ 00:11 0:00 | \_ /usr/bin/sudo flock /var/lock/lava-lmc.lck linaro-media-create --hwpack-force-yes --dev arndale --image-file /var/lib/lava/dispatcher/tmp/tmpJ9_9mo/lava.img root 47566 0.0 0.0 4200 344 pts/5 S+ 00:11 0:00 | \_ flock /var/lock/lava-lmc.lck linaro-media-create --hwpack-force-yes --dev arndale --image-file /var/lib/lava/dispatcher/tmp/tmpJ9_9mo/lava.img --binary / root 46062 0.0 0.5 292852 62576 ? Ssl 05:40 0:06 \_ /usr/bin/python /usr/bin/lava-server manage schedulermonitor lava-dispatch panda01 /tmp/tmpVxp2IY -l info -f /var/log/lava-server/lava-scheduler.log root 46078 0.0 0.4 243396 49704 ? Sl 05:40 0:11 | \_ /usr/bin/python /usr/bin/lava dispatch /tmp/tmpFDnq1w --output-dir /var/lib/lava-server/default/media/job-output/job-142750 root 46091 0.0 0.0 4220 348 ? S 05:40 0:00 | \_ tee /var/lib/lava-server/default/media/job-output/job-142750/output.txt root 46092 0.0 0.0 25424 1480 pts/2 Ss+ 05:40 0:01 | \_ /usr/bin/telnet serial1 7004 root 11710 0.0 0.0 8488 632 ? S 07:30 0:00 | \_ timeout 18000s lava-android-test run methanol -s 10.254.30.165:5555 -o /tmp/tmp12w6cY/methanol.2014-07-23T06:30:45Z.bundle -O -t canvas root 11711 0.0 0.1 331156 19004 ? Sl 07:30 0:00 | \_ /usr/bin/python /usr/bin/lava-android-test run methanol -s 10.254.30.165:5555 -o /tmp/tmp12w6cY/methanol.2014-07-23T06:30:45Z.bundle -O -t canvas root 11895 0.0 0.0 4312 644 ? S 07:30 0:00 | \_ /bin/sh -c bash /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/methanol/methanol.sh 10.254.30.165:5555 -t canvas root 11897 0.0 0.0 9652 1640 ? S 07:30 0:00 | \_ bash /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/methanol/methanol.sh 10.254.30.165:5555 -t canvas root 11926 0.0 0.0 43076 9316 ? S 07:30 0:00 | \_ python /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/methanol/start_server.py 10.2.1.1 /tmp/lava-android-test/tmphOOj3h root 33388 0.0 0.0 4216 352 ? S 09:20 0:00 | \_ sleep 300 root 2709 0.0 0.5 292848 62456 ? Ssl 06:58 0:03 \_ /usr/bin/python /usr/bin/lava-server manage schedulermonitor lava-dispatch panda03 /tmp/tmp1CDH6F -l info -f /var/log/lava-server/lava-scheduler.log root 2718 0.0 0.4 169532 49652 ? S 06:58 0:02 | \_ /usr/bin/python /usr/bin/lava dispatch /tmp/tmppMHFeK --output-dir /var/lib/lava-server/default/media/job-output/job-142759 root 2723 0.0 0.0 4220 348 ? S 06:58 0:00 | \_ tee /var/lib/lava-server/default/media/job-output/job-142759/output.txt root 2727 0.0 0.0 25424 1488 pts/9 Ss+ 06:58 0:00 | \_ /usr/bin/telnet serial1 7006 root 3081 0.0 0.0 54648 1872 pts/11 Ss+ 07:00 0:00 | \_ /usr/bin/sudo flock /var/lock/lava-lmc.lck linaro-media-create --hwpack-force-yes --dev panda --image-file /var/lib/lava/dispatcher/tmp/tmpdTrY73/lava.img -- root 3082 0.0 0.0 4200 348 pts/11 S+ 07:00 0:00 | \_ flock /var/lock/lava-lmc.lck linaro-media-create --hwpack-force-yes --dev panda --image-file /var/lib/lava/dispatcher/tmp/tmpdTrY73/lava.img --binary /va root 20393 0.0 0.5 292844 62428 ? Ssl 08:14 0:02 \_ /usr/bin/python /usr/bin/lava-server manage schedulermonitor lava-dispatch arndale01 /tmp/tmp28WIBB -l info -f /var/log/lava-server/lava-scheduler.log root 20402 0.1 0.4 169528 49644 ? S 08:14 0:05 \_ /usr/bin/python /usr/bin/lava dispatch /tmp/tmpbcrNHZ --output-dir /var/lib/lava-server/default/media/job-output/job-142779 root 20410 0.0 0.0 4220 348 ? S 08:14 0:00 \_ tee /var/lib/lava-server/default/media/job-output/job-142779/output.txt root 20411 0.0 0.0 25424 1484 pts/8 Ss+ 08:14 0:00 \_ /usr/bin/telnet serial1 7012 root 20727 0.0 0.0 54648 1872 pts/12 Ss+ 08:15 0:00 \_ /usr/bin/sudo flock /var/lock/lava-lmc.lck linaro-media-create --hwpack-force-yes --dev arndale --image-file /var/lib/lava/dispatcher/tmp/tmpN6Q8pi/lava.img root 20728 0.0 0.0 4200 348 pts/12 S+ 08:15 0:00 \_ flock /var/lock/lava-lmc.lck linaro-media-create --hwpack-force-yes --dev arndale --image-file /var/lib/lava/dispatcher/tmp/tmpN6Q8pi/lava.img --binary / root 23548 0.0 0.4 117604 59364 ? S Jul22 0:06 /usr/bin/python /usr/bin/linaro-media-create --hwpack-force-yes --dev arndale --image-file /var/lib/lava/dispatcher/tmp/tmpQeJXVm/lava.img --binary /var/lib/lava/dispatcher/ root 25841 0.0 0.0 4324 608 ? D Jul22 0:00 \_ sfdisk --force -D -uS -H 128 -S 32 -C 1536 /var/lib/lava/dispatcher/tmp/tmpQeJXVm/lava.img admin@dispatcher01:~$ ls /var/lib/lava/dispatcher/tmp/tmpQeJXVm/lava.img ls: cannot access /var/lib/lava/dispatcher/tmp/tmpQeJXVm/lava.img: No such file or directory admin@dispatcher01:~$ mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=1533833,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1228960k,mode=755) /dev/disk/by-uuid/f39f4d00-5adb-40b7-80cd-06e9380067ab on / type ext4 (rw,relatime,errors=remount-ro,data=ordered) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) pstore on /sys/fs/pstore type pstore (rw,relatime) tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6384940k) fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime) rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw,relatime) none on /sys/kernel/config type configfs (rw,relatime) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime) nfsd on /proc/fs/nfsd type nfsd (rw,relatime) lavaser...@validation.linaro.org:/var/lib/lava-server//default/media on /var/lib/lava-server/default/media type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=111,group_id=116,allow_other) none on /sys/fs/cgroup type tmpfs (rw,relatime,size=4k,mode=755) systemd on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,name=systemd) proc on /tmp/tmpAFpoZw/rootfs/binary/proc type proc (rw,relatime) admin@dispatcher01:~$ ls -al /var/lock/lava-lmc.lck -rw-r--r-- 1 root root 0 Jul 18 15:15 /var/lock/lava-lmc.lck admin@dispatcher01:~$ cat /var/lock/lava-lmc.lck admin@dispatcher01:~$ sudo lsof | grep lmc [sudo] password for admin: Sorry, try again. [sudo] password for admin: flock 20728 root 3u REG 0,16 0 32286 /run/lock/lava-lmc.lck linaro-me 36314 root 3u REG 0,16 0 32286 /run/lock/lava-lmc.lck linaro-hw 36483 root 3u REG 0,16 0 32286 /run/lock/lava-lmc.lck apt-get 36690 root 3u REG 0,16 0 32286 /run/lock/lava-lmc.lck flock 47566 root 3uW REG 0,16 0 32286 /run/lock/lava-lmc.lck admin@dispatcher01:~$ ls -al /run/lock/lava-lmc.lck -rw-r--r-- 1 root root 0 Jul 18 15:15 /run/lock/lava-lmc.lck admin@dispatcher01:~$ ls -alh /run/lock/lava-lmc.lck -rw-r--r-- 1 root root 0 Jul 18 15:15 /run/lock/lava-lmc.lck admin@dispatcher01:~$ ls -alh /var/lock/lava-lmc.lck -rw-r--r-- 1 root root 0 Jul 18 15:15 /var/lock/lava-lmc.lck admin@dispatcher01:~$ sudo losetup -a /dev/loop0: [0900]:14287000 (/var/lib/lava/dispatcher/tmp/tmpJ9_9mo/lava.img), offset 58720256 admin@dispatcher01:~$ ls /var/lib/lava/dispatcher/tmp/tmpQeJXVm/ ls: cannot access /var/lib/lava/dispatcher/tmp/tmpQeJXVm/: No such file or directory admin@dispatcher01:~$ ls /var/lib/lava/dispatcher/tmp/ grub.cfg-01-00-01-73-02-00-ce images pxelinux tmp6KdlqQ tmpAherPz tmpDAZlSq tmpEGZKF9 tmpgkab9E tmpN31i8X tmpOazQ_c tmpqKlvrQ tmpVfBx_E highbank midway rhel tmp8CSOoT tmpALbL4i tmpdTrY73 tmpevhoEx tmpJ9_9mo tmpN6Q8pi tmposzJ7W tmpRN0JZk tmpWfFSIe admin@dispatcher01:~$ sudo kill -9 25841 admin@dispatcher01:~$ echo $? 1 admin@dispatcher01:~$ ps auxf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 2 0.0 0.0 0 0 ? S Jul18 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S Jul18 0:29 \_ [ksoftirqd/0] root 5 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/0:0H] root 7 0.0 0.0 0 0 ? S Jul18 1:14 \_ [rcu_sched] root 8 0.0 0.0 0 0 ? S Jul18 0:00 \_ [rcu_bh] root 9 0.0 0.0 0 0 ? S Jul18 0:00 \_ [migration/0] root 10 0.0 0.0 0 0 ? S Jul18 0:01 \_ [watchdog/0] root 11 0.0 0.0 0 0 ? S Jul18 0:01 \_ [watchdog/1] root 12 0.0 0.0 0 0 ? S Jul18 0:00 \_ [migration/1] root 13 0.0 0.0 0 0 ? S Jul18 0:10 \_ [ksoftirqd/1] root 15 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/1:0H] root 16 0.0 0.0 0 0 ? S Jul18 0:00 \_ [watchdog/2] root 17 0.0 0.0 0 0 ? S Jul18 0:00 \_ [migration/2] root 18 0.0 0.0 0 0 ? S Jul18 0:08 \_ [ksoftirqd/2] root 20 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/2:0H] root 21 0.0 0.0 0 0 ? S Jul18 0:00 \_ [watchdog/3] root 22 0.0 0.0 0 0 ? S Jul18 0:00 \_ [migration/3] root 23 0.0 0.0 0 0 ? S Jul18 0:09 \_ [ksoftirqd/3] root 25 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/3:0H] root 26 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [khelper] root 27 0.0 0.0 0 0 ? S Jul18 0:00 \_ [kdevtmpfs] root 28 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [netns] root 29 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [writeback] root 30 0.0 0.0 0 0 ? SN Jul18 0:00 \_ [ksmd] root 31 0.0 0.0 0 0 ? SN Jul18 0:00 \_ [khugepaged] root 32 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kintegrityd] root 33 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [bioset] root 34 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [crypto] root 35 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kblockd] root 37 0.0 0.0 0 0 ? S Jul18 0:00 \_ [khungtaskd] root 38 0.0 0.0 0 0 ? S Jul18 0:01 \_ [kswapd0] root 39 0.0 0.0 0 0 ? S Jul18 0:00 \_ [fsnotify_mark] root 44 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kthrotld] root 45 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [ipv6_addrconf] root 47 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [deferwq] root 124 0.0 0.0 0 0 ? S Jul18 0:30 \_ [khubd] root 133 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [ata_sff] root 179 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_0] root 180 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_0] root 181 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_1] root 182 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_1] root 187 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_2] root 188 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_2] root 189 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_3] root 190 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_3] root 208 0.0 0.0 0 0 ? S< Jul18 0:08 \_ [kworker/0:1H] root 217 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [md] root 223 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [bioset] root 224 0.0 0.0 0 0 ? S Jul18 0:22 \_ [md0_raid1] root 247 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [bioset] root 256 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_4] root 257 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_4] root 258 0.0 0.0 0 0 ? S Jul18 0:00 \_ [usb-storage] root 272 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_6] root 273 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_6] root 274 0.0 0.0 0 0 ? S Jul18 0:00 \_ [usb-storage] root 277 0.0 0.0 0 0 ? S Jul18 0:00 \_ [scsi_eh_7] root 278 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [scsi_tmf_7] root 279 0.0 0.0 0 0 ? S Jul18 0:00 \_ [usb-storage] root 282 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/1:1H] root 283 0.0 0.0 0 0 ? D Jul18 0:16 \_ [jbd2/md0-8] root 284 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [ext4-rsv-conver] root 285 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/2:1H] root 429 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kworker/3:1H] root 558 0.0 0.0 0 0 ? SN Jul18 0:12 \_ [kipmi0] root 559 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kpsmoused] root 578 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [kvm-irqfd-clean] root 1899 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [rpciod] root 1901 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [nfsiod] root 2501 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [nfsd4] root 2502 0.0 0.0 0 0 ? S< Jul18 0:00 \_ [nfsd4_callbacks] root 2503 0.0 0.0 0 0 ? S Jul18 0:00 \_ [lockd] root 2506 0.0 0.0 0 0 ? S Jul18 0:05 \_ [nfsd] root 2507 0.0 0.0 0 0 ? S Jul18 0:02 \_ [nfsd] root 2508 0.0 0.0 0 0 ? S Jul18 0:03 \_ [nfsd] root 2509 0.0 0.0 0 0 ? S Jul18 0:02 \_ [nfsd] root 2510 0.0 0.0 0 0 ? S Jul18 0:07 \_ [nfsd] root 2511 0.0 0.0 0 0 ? S Jul18 0:03 \_ [nfsd] root 2512 0.0 0.0 0 0 ? S Jul18 0:05 \_ [nfsd] root 2513 0.0 0.0 0 0 ? S Jul18 0:04 \_ [nfsd] root 3289 0.0 0.0 0 0 ? S Jul18 0:00 \_ [kauditd] root 30791 0.0 0.0 0 0 ? S Jul22 0:00 \_ [scsi_eh_259] root 30792 0.0 0.0 0 0 ? S< Jul22 0:00 \_ [scsi_tmf_259] root 30793 0.0 0.0 0 0 ? S Jul22 0:00 \_ [usb-storage] root 2327 0.0 0.0 0 0 ? S 01:55 0:00 \_ [kworker/u128:0] root 14405 0.0 0.0 0 0 ? S 02:57 0:00 \_ [kworker/u128:1] root 28395 0.0 0.0 0 0 ? S 08:55 0:00 \_ [kworker/0:0] root 29538 0.0 0.0 0 0 ? S 09:01 0:00 \_ [kworker/3:3] root 29798 0.0 0.0 0 0 ? S 09:02 0:00 \_ [kworker/1:2] root 30317 0.0 0.0 0 0 ? S 09:05 0:00 \_ [kworker/2:1] root 33910 0.0 0.0 0 0 ? S 09:22 0:00 \_ [kworker/u128:2] root 33911 0.0 0.0 0 0 ? S 09:22 0:00 \_ [kworker/u128:3] root 34169 0.0 0.0 0 0 ? S 09:23 0:00 \_ [kworker/2:2] root 34302 0.0 0.0 0 0 ? S 09:23 0:00 \_ [scsi_eh_372] root 34303 0.0 0.0 0 0 ? S< 09:23 0:00 \_ [scsi_tmf_372] root 34304 0.0 0.0 0 0 ? S 09:23 0:00 \_ [usb-storage] root 34668 0.0 0.0 0 0 ? S 09:24 0:00 \_ [kworker/1:1] root 35817 0.0 0.0 0 0 ? S 09:25 0:00 \_ [kworker/0:1] root 36045 0.0 0.0 0 0 ? S 09:25 0:00 \_ [kworker/3:1] root 38550 0.0 0.0 0 0 ? S 09:28 0:00 \_ [kworker/2:0] root 38986 0.4 0.0 0 0 ? D 09:30 0:00 \_ [kworker/u128:4] root 39062 0.5 0.0 0 0 ? S< 09:30 0:00 \_ [loop0] root 39065 0.0 0.0 0 0 ? S 09:30 0:00 \_ [jbd2/loop0-8] root 39066 0.0 0.0 0 0 ? S< 09:30 0:00 \_ [ext4-rsv-conver] root 39105 0.0 0.0 0 0 ? S 09:30 0:00 \_ [kworker/0:2] root 39390 0.0 0.0 0 0 ? S 09:31 0:00 \_ [kworker/3:0] root 39426 0.0 0.0 0 0 ? S 09:31 0:00 \_ [kworker/1:0] root 1 0.0 0.0 13224 816 ? Ss Jul18 0:03 init [2] root 442 0.0 0.0 42240 1420 ? Ss Jul18 0:02 udevd --daemon root 1866 0.0 0.0 23272 1152 ? Ss Jul18 0:00 /sbin/rpcbind -w statd 1894 0.0 0.0 23476 1328 ? Ss Jul18 0:00 /sbin/rpc.statd root 1908 0.0 0.0 25428 424 ? Ss Jul18 0:00 /usr/sbin/rpc.idmapd root 2128 0.0 0.3 308044 37180 ? Ssl Jul18 0:25 /sbin/zfs-fuse --pidfile /var/run/zfs-fuse.pid root 2395 0.0 0.0 6448 368 ? S Jul18 0:00 /usr/sbin/ser2net -c /etc/ser2net.conf -P /run/ser2net.pid root 2477 0.0 0.0 258976 8456 ? Ssl Jul18 0:10 /usr/sbin/rsyslogd root 2520 0.0 0.0 28356 1948 ? Ss Jul18 0:00 /usr/sbin/rpc.mountd --manage-gids root 2535 0.0 0.0 4240 644 ? Ss Jul18 0:00 /usr/sbin/acpid root 2557 0.0 0.0 39600 7744 ? S Jul18 0:00 /usr/bin/python /usr/bin/lava-coordinator --loglevel=DEBUG root 2598 0.0 0.0 88380 3992 ? Ss Jul18 0:07 /usr/sbin/apache2 -k start lavaser+ 14425 0.0 0.0 213496 5688 ? Sl Jul22 0:01 \_ /usr/sbin/apache2 -k start www-data 14426 0.0 0.0 379716 4096 ? Sl Jul22 0:08 \_ /usr/sbin/apache2 -k start www-data 14427 0.0 0.0 379780 4208 ? Sl Jul22 0:11 \_ /usr/sbin/apache2 -k start root 2705 0.0 0.0 13264 828 ? Ss Jul18 0:00 /sbin/mdadm --monitor --pid-file /run/mdadm/monitor.pid --daemonise --scan --syslog root 2795 0.0 0.0 14984 316 ? Ss Jul18 0:00 /usr/sbin/in.tftpd --listen --user tftp --address 0.0.0.0:69 --secure /var/lib/lava/dispatcher/tmp/ root 2821 0.0 0.0 25564 1420 ? S Jul18 0:00 /usr/sbin/vsftpd ntp 2842 0.0 0.0 33308 2216 ? Ss Jul18 0:12 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 107:111 root 2865 0.0 0.0 25240 1072 ? Ss Jul18 0:00 /usr/sbin/cron daemon 2889 0.0 0.0 19000 164 ? Ss Jul18 0:00 /usr/sbin/atd root 2907 0.0 0.0 54976 1288 ? Ss Jul18 0:06 /usr/sbin/sshd root 33251 0.0 0.0 97256 4224 ? Ss 09:20 0:00 \_ sshd: admin [priv] matthew+ 33256 0.0 0.0 97256 1968 ? S 09:20 0:00 | \_ sshd: admin@pts/13 matthew+ 33257 0.0 0.0 21288 3848 pts/13 Ss 09:20 0:00 | \_ -bash matthew+ 39460 0.0 0.0 19244 1396 pts/13 R+ 09:31 0:00 | \_ ps auxf root 33665 0.0 0.0 97260 4220 ? Ss 09:22 0:00 \_ sshd: dpigott [priv] dpigott 33667 0.0 0.0 97260 2084 ? S 09:22 0:00 \_ sshd: dpigott@pts/1 dpigott 33668 0.0 0.0 21316 3876 pts/1 Ss+ 09:22 0:00 \_ -bash message+ 2926 0.0 0.0 41620 1256 ? Ss Jul18 0:24 /usr/bin/dbus-daemon --system root 2945 0.0 0.3 496988 46144 ? Sl Jul18 1:08 /usr/bin/python /usr/bin/salt-minion -d root 2948 0.0 0.1 378376 12512 ? Sl Jul18 0:01 /usr/sbin/libvirtd -d postgres 2990 0.0 0.1 220668 15760 ? S Jul18 0:02 /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf postgres 3028 0.0 0.0 220668 1844 ? Ss Jul18 0:00 \_ postgres: checkpointer process postgres 3029 0.0 0.0 220668 2640 ? Ss Jul18 0:01 \_ postgres: writer process postgres 3030 0.0 0.0 220668 1632 ? Ss Jul18 0:01 \_ postgres: wal writer process postgres 3031 0.0 0.0 221476 3004 ? Ss Jul18 0:02 \_ postgres: autovacuum launcher process postgres 3032 0.0 0.0 76344 1740 ? Ss Jul18 0:06 \_ postgres: stats collector process lavaser+ 3346 0.0 0.0 46436 3364 ? S Jul18 2:00 ssh -x -a -oClearAllForwardings=yes -oPasswordAuthentication=no -oStrictHostKeyChecking=no -2 lavaser...@validation.linaro.org -s sftp Debian-+ 3349 0.0 0.0 52804 1092 ? Ss Jul18 0:00 /usr/sbin/exim4 -bd -q30m lavaser+ 3371 0.0 0.0 387828 912 ? Ssl Jul18 1:49 sshfs -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o allow_root -o reconnect lavaser...@validation.linaro.org:/var/lib/lava-server//default/media /var/lib/lava- root 3425 0.0 0.0 16452 944 tty1 Ss+ Jul18 0:00 /sbin/getty 38400 tty1 root 3426 0.0 0.0 16452 940 tty2 Ss+ Jul18 0:00 /sbin/getty 38400 tty2 root 3427 0.0 0.0 16452 940 tty3 Ss+ Jul18 0:00 /sbin/getty 38400 tty3 root 3428 0.0 0.0 16452 936 tty4 Ss+ Jul18 0:00 /sbin/getty 38400 tty4 root 3429 0.0 0.0 16452 932 tty5 Ss+ Jul18 0:00 /sbin/getty 38400 tty5 root 3430 0.0 0.0 16452 944 tty6 Ss+ Jul18 0:00 /sbin/getty 38400 tty6 root 3507 0.0 0.0 37492 2160 ? S Jul18 0:18 /lib/systemd/systemd-logind codehelp 5119 0.0 0.0 25816 1644 ? Ss Jul18 0:00 SCREEN codehelp 5120 0.0 0.0 21384 4052 pts/3 Ss+ Jul18 0:00 \_ /bin/bash root 5428 0.0 0.0 498200 2488 ? SNl Jul18 4:45 adb -P 5037 fork-server server root 30360 0.0 0.5 366504 62540 ? Ssl Jul22 0:28 /usr/bin/python /usr/bin/lava-server manage schedulermonitor lava-dispatch wg-02 /tmp/tmpty846N -l info -f /var/log/lava-server/lava-scheduler.log root 30366 0.0 0.4 243260 50136 ? Sl Jul22 0:13 \_ /usr/bin/python /usr/bin/lava dispatch /tmp/tmpBdJJBO --output-dir /var/lib/lava-server/default/media/job-output/job-142527 root 30371 0.0 0.0 4220 352 ? S Jul22 0:01 \_ tee /var/lib/lava-server/default/media/job-output/job-142527/output.txt root 30372 0.0 0.0 25424 1484 pts/7 Ss+ Jul22 0:00 \_ /usr/bin/telnet serial01 7021 root 55936 0.0 0.0 8488 636 ? S Jul22 0:00 \_ timeout 72000s lava-android-test run cts -s 10.254.30.129:5555 -o /tmp/tmp5GdUAl/cts.2014-07-22T14:44:40Z.bundle -O --cts-pkg android-cts-armv7-master.zip --plan CTS root 55937 0.0 0.2 340368 28472 ? Sl Jul22 0:12 \_ /usr/bin/python /usr/bin/lava-android-test run cts -s 10.254.30.129:5555 -o /tmp/tmp5GdUAl/cts.2014-07-22T14:44:40Z.bundle -O --cts-pkg android-cts-armv7-master. root 56118 0.0 0.0 4312 640 ? S Jul22 0:00 \_ /bin/sh -c python /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/cts/cts_wrapper.py 10.254.30.129:5555 --cts-pkg android-cts-armv7-maste root 56120 0.4 0.1 274068 14968 ? S Jul22 4:17 \_ python /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/cts/cts_wrapper.py 10.254.30.129:5555 --cts-pkg android-cts-armv7-master.zip - root 56249 0.0 0.0 9412 1348 pts/6 Ss+ Jul22 0:00 \_ /bin/bash ./android-cts/tools/cts-tradefed run cts --serial 10.254.30.129:5555 --plan CTS --disable-reboot root 56268 0.7 0.6 5039960 79472 pts/6 Sl+ Jul22 7:30 \_ java -cp :./android-cts/tools/../../android-cts/tools/ddmlib-prebuilt.jar:./android-cts/tools/../../android-cts/tools/tradefed-prebuilt.jar:./and root 56128 0.0 0.0 9396 648 ? S Jul22 0:00 bash /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/cts/cts_redirect.sh kmsg.log adb -s 10.254.30.129:5555 shell cat /proc/kmsg root 56129 0.0 0.0 14700 1280 ? S Jul22 0:00 \_ adb -s 10.254.30.129:5555 shell cat /proc/kmsg root 56135 0.0 0.0 9400 800 ? S Jul22 0:00 bash /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/cts/cts_redirect.sh logcat.log adb -s 10.254.30.129:5555 logcat -c; adb -s 10.254.30.129:5555 logcat root 56140 0.0 0.0 14832 1280 ? S Jul22 0:00 \_ adb -s 10.254.30.129:5555 logcat -v time root 14523 0.0 0.0 36720 7136 ? S Jul22 0:00 /usr/bin/python /usr/bin/lava-daemon manage --instance-template=/etc/lava-server/{{filename}}.conf --instance=production scheduler --logfile /var/log/lava-server/lava-schedu root 14524 5.4 7.1 1182744 881664 ? Sl Jul22 53:27 \_ /usr/bin/python /usr/bin/lava-server manage --instance-template=/etc/lava-server/{{filename}}.conf --instance=production scheduler --logfile /var/log/lava-server/lava-sc root 47223 0.0 0.5 366580 62484 ? Ssl 00:10 0:11 \_ /usr/bin/python /usr/bin/lava-server manage schedulermonitor lava-dispatch arndale03 /tmp/tmp85NZjU -l info -f /var/log/lava-server/lava-scheduler.log root 47235 0.0 0.4 169524 49648 ? S 00:10 0:03 | \_ /usr/bin/python /usr/bin/lava dispatch /tmp/tmp0e1irr --output-dir /var/lib/lava-server/default/media/job-output/job-142697 root 47240 0.0 0.0 4220 352 ? S 00:10 0:00 | \_ tee /var/lib/lava-server/default/media/job-output/job-142697/output.txt root 47244 0.0 0.0 25424 1484 pts/0 Ss+ 00:10 0:00 | \_ /usr/bin/telnet serial1 7016 root 39165 0.0 0.0 4312 636 ? S 09:30 0:00 | \_ sh -c sync root 39166 0.0 0.0 4216 352 ? D 09:30 0:00 | \_ sync root 46062 0.0 0.5 292852 62576 ? Ssl 05:40 0:06 \_ /usr/bin/python /usr/bin/lava-server manage schedulermonitor lava-dispatch panda01 /tmp/tmpVxp2IY -l info -f /var/log/lava-server/lava-scheduler.log root 46078 0.0 0.4 243396 49704 ? Sl 05:40 0:11 | \_ /usr/bin/python /usr/bin/lava dispatch /tmp/tmpFDnq1w --output-dir /var/lib/lava-server/default/media/job-output/job-142750 root 46091 0.0 0.0 4220 348 ? S 05:40 0:00 | \_ tee /var/lib/lava-server/default/media/job-output/job-142750/output.txt root 46092 0.0 0.0 25424 1480 pts/2 Ss+ 05:40 0:01 | \_ /usr/bin/telnet serial1 7004 root 11710 0.0 0.0 8488 632 ? S 07:30 0:00 | \_ timeout 18000s lava-android-test run methanol -s 10.254.30.165:5555 -o /tmp/tmp12w6cY/methanol.2014-07-23T06:30:45Z.bundle -O -t canvas root 11711 0.0 0.1 331156 19004 ? Sl 07:30 0:00 | \_ /usr/bin/python /usr/bin/lava-android-test run methanol -s 10.254.30.165:5555 -o /tmp/tmp12w6cY/methanol.2014-07-23T06:30:45Z.bundle -O -t canvas root 11895 0.0 0.0 4312 644 ? S 07:30 0:00 | \_ /bin/sh -c bash /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/methanol/methanol.sh 10.254.30.165:5555 -t canvas root 11897 0.0 0.0 9652 1640 ? S 07:30 0:00 | \_ bash /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/methanol/methanol.sh 10.254.30.165:5555 -t canvas root 11926 0.0 0.0 43076 9316 ? S 07:30 0:00 | \_ python /usr/lib/python2.7/dist-packages/lava_android_test/test_definitions/methanol/start_server.py 10.2.1.1 /tmp/lava-android-test/tmphOOj3h root 39258 0.0 0.0 4216 352 ? S 09:30 0:00 | \_ sleep 300 root 2709 0.0 0.5 292848 62456 ? Ssl 06:58 0:03 \_ /usr/bin/python /usr/bin/lava-server manage schedulermonitor lava-dispatch panda03 /tmp/tmp1CDH6F -l info -f /var/log/lava-server/lava-scheduler.log root 2718 0.0 0.4 169532 49684 ? S 06:58 0:02 | \_ /usr/bin/python /usr/bin/lava dispatch /tmp/tmppMHFeK --output-dir /var/lib/lava-server/default/media/job-output/job-142759 root 2723 0.0 0.0 4220 348 ? S 06:58 0:00 | \_ tee /var/lib/lava-server/default/media/job-output/job-142759/output.txt root 2727 0.0 0.0 25424 1488 pts/9 Ss+ 06:58 0:00 | \_ /usr/bin/telnet serial1 7006 root 20393 0.0 0.5 292844 62428 ? Ssl 08:14 0:02 \_ /usr/bin/python /usr/bin/lava-server manage schedulermonitor lava-dispatch arndale01 /tmp/tmp28WIBB -l info -f /var/log/lava-server/lava-scheduler.log root 20402 0.1 0.4 169528 49644 ? S 08:14 0:05 | \_ /usr/bin/python /usr/bin/lava dispatch /tmp/tmpbcrNHZ --output-dir /var/lib/lava-server/default/media/job-output/job-142779 root 20410 0.0 0.0 4220 348 ? S 08:14 0:00 | \_ tee /var/lib/lava-server/default/media/job-output/job-142779/output.txt root 20411 0.0 0.0 25424 1484 pts/8 Ss+ 08:14 0:00 | \_ /usr/bin/telnet serial1 7012 root 20727 0.0 0.0 54648 1872 pts/12 Ss+ 08:15 0:00 | \_ /usr/bin/sudo flock /var/lock/lava-lmc.lck linaro-media-create --hwpack-force-yes --dev arndale --image-file /var/lib/lava/dispatcher/tmp/tmpN6Q8pi/lava.img root 20728 0.0 0.0 4200 348 pts/12 S+ 08:15 0:00 | \_ flock /var/lock/lava-lmc.lck linaro-media-create --hwpack-force-yes --dev arndale --image-file /var/lib/lava/dispatcher/tmp/tmpN6Q8pi/lava.img --binary / root 38951 24.5 0.5 129584 71172 pts/12 S+ 09:30 0:24 | \_ /usr/bin/python /usr/bin/linaro-media-create --hwpack-force-yes --dev arndale --image-file /var/lib/lava/dispatcher/tmp/tmpN6Q8pi/lava.img --binary / root 39230 0.0 0.0 4324 612 pts/12 D+ 09:30 0:00 | \_ sfdisk --force -D -uS -H 128 -S 32 -C 1536 /var/lib/lava/dispatcher/tmp/tmpN6Q8pi/lava.img root 39403 7.2 0.5 292848 62432 ? Ssl 09:31 0:00 \_ /usr/bin/python /usr/bin/lava-server manage schedulermonitor lava-dispatch vexpress-tc2-benchmark-01 /tmp/tmpUVPaT6 -l info -f /var/log/lava-server/lava-scheduler.lo root 39418 12.5 0.4 167452 49468 ? S 09:31 0:01 \_ /usr/bin/python /usr/bin/lava dispatch /tmp/tmpKy3CSi --output-dir /var/lib/lava-server/default/media/job-output/job-142815 root 39431 0.0 0.0 4220 352 ? S 09:31 0:00 \_ tee /var/lib/lava-server/default/media/job-output/job-142815/output.txt root 39435 0.0 0.0 25424 1488 pts/4 Ss+ 09:31 0:00 \_ /usr/bin/telnet serial01 7002