On Tue, 2012-05-15 at 17:37 +0530, Yogesh Kulkarni wrote: > hi Chris, Eduardo > > > yes, i am using the older version of the autotest. > (it has been setup with my local machine and newer one installed on > test-server in our lab :) > > > well, i executed ltp few days ago on the server machine via > 'client/tests/ltp/control' interface, it crashed. > it has some residues left in the 'client/tmp/*' directory and of > course results appeared in 'client/results/default/ltp/*', so no > matter whatever test i was executing (via 'server/autoserv'), setup > that was 'scp'ied on the target board was with this crashed ltp > execution with 'crash.vino-server.1864' report. > > > So when I removed 'client/tmp/*' and 'client/results/default/*', > problem appears to be solved. > > > BTW, another problem I observed (i feel it is on the similar lines, > and this one i am facing right from the start i installed the autotest > server) is that, sometimes, after job started on the server, it gets > stuck while 'scp'ing the tests on to the target. And appears to be > stuck forever. > > > On target board, > in the '/usr/local/autotest/tests' it shows only 'download' and > '__init__.py/pyc' as below. and it looks like tests are in the > 'scp'ing process and stuck. > > > root@tII:/usr/local/autotest# pwd > /usr/local/autotest > root@tII:/usr/local/autotest# ls > bin config control.autoserv.state > global_config.ini __init__.pyc packages results setup_modules.py > setup_modules_unittest.py stderr tmp virt common_lib > control.autoserv deps __init__.py LICENSE > profilers samples setup_modules.pyc site_tests tests > tools > root@tII:/usr/local/autotest# ls tests/ > download __init__.py __init__.pyc > root@tII:/usr/local/autotest# > > > At this point, > > > autoserv.DEBUG on the server side appears as, > > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > 05/15 04:25:04 INFO | autoserv:0201| Results placed in > /usr/local/autotest/results/114-autotest/10.0.0.136 > 05/15 04:25:04 DEBUG| base_job:0347| Persistent state global_properties.tag > now set to '114-autotest/10.0.0.136' > 05/15 04:25:04 DEBUG| base_job:0347| Persistent state > global_properties.last_boot_tag now set to None > 05/15 04:25:04 INFO |server_job:0534| Processing control file > 05/15 04:25:04 DEBUG|base_utils:0076| Running > '/usr/local/autotest/conmux/conmux-attach 10.0.0.136 echo 2> /dev/null' > 05/15 04:25:05 DEBUG| ssh_host:0114| Running (ssh) 'ls /var/log/kern.log' > 05/15 04:25:05 DEBUG|base_utils:0108| [stderr] Warning: Permanently added > '10.0.0.136' (RSA) to the list of known hosts. > 05/15 04:25:05 DEBUG|base_utils:0108| [stdout] /var/log/kern.log > 05/15 04:25:05 DEBUG| ssh_host:0114| Running (ssh) 'mkdir -p /var/tmp' > 05/15 04:25:06 DEBUG| ssh_host:0114| Running (ssh) 'mktemp -d > /var/tmp/autoserv-XXXXXX' > 05/15 04:25:07 DEBUG|base_utils:0108| [stdout] /var/tmp/autoserv-CkijtI > 05/15 04:25:07 INFO |logfile_mo:0052| Launching followfiles on target: > 10.0.0.136, /var/tmp/autoserv-CkijtI, ['/var/log/kern.log'] > 05/15 04:25:07 DEBUG| ssh_host:0114| Running (ssh) 'ls /usr/bin/python[0-9]*' > 05/15 04:25:08 DEBUG|base_utils:0108| [stdout] /usr/bin/python2.6 > 05/15 04:25:08 DEBUG| ssh_host:0114| Running (ssh) 'mkdir -p /tmp' > 05/15 04:25:08 DEBUG| ssh_host:0114| Running (ssh) 'mktemp -d > /tmp/autoserv-XXXXXX' > 05/15 04:25:09 DEBUG|base_utils:0108| [stdout] /tmp/autoserv-7VuS9L > 05/15 04:25:09 DEBUG| ssh_host:0114| Running (ssh) 'rsync --version' > 05/15 04:25:10 WARNI|abstract_s:0074| rsync not available on remote host > 10.0.0.136 -- disabled > 05/15 04:25:10 DEBUG|base_utils:0076| Running 'scp -rq -o > StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpcsovUa -P 22 > /usr/local/autotest/server/hosts/monitors > '[email protected]:"/tmp/autoserv-7VuS9L"'' > 05/15 04:25:12 DEBUG| ssh_host:0114| Running (ssh) 'test -f > /var/log/messages' > 05/15 04:25:13 DEBUG| remote:0169| Found remote path /var/log/messages > 05/15 04:25:13 DEBUG| ssh_host:0114| Running (ssh) 'rm -f > /var/tmp/messages.autotest_start' > 05/15 04:25:13 DEBUG| ssh_host:0114| Running (ssh) 'cp /var/log/messages > /var/tmp/messages.autotest_start' > 05/15 04:25:14 DEBUG| ssh_host:0114| Running (ssh) '/bin/uname -r' > 05/15 04:25:15 DEBUG|base_utils:0108| [stdout] 2.6.36.2 > 05/15 04:25:15 INFO |server_job:0117| INFO ---- ---- kernel=2.6.36.2 > localtime=May 15 04:25:15 timestamp=1337081115 > 05/15 04:25:15 DEBUG|base_utils:0076| Running 'tools/make_clean' > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning monotonic_time test > dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning spidev_test test dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning iosched_bugs test dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning synctest test dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning ipv6connect test dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning hackbench test dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning hello_world_cfg test > dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning rmaptest test dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning cyclic_test test dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning rtc test dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning aio_dio_bugs test dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning > tracing_microbenchmark test dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning signal_test test dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning disktest test dir > 05/15 04:25:16 DEBUG|base_utils:0108| [stdout] Cleaning tsc test dir > 05/15 04:25:16 DEBUG| ssh_host:0114| Running (ssh) 'true' > 05/15 04:25:17 DEBUG|abstract_s:0456| Host 10.0.0.136 is now up > 05/15 04:25:17 INFO |autotest_r:0198| Installing autotest on 10.0.0.136 > 05/15 04:25:17 DEBUG| ssh_host:0114| Running (ssh) 'test -x > /usr/local/autotest/bin/autotest' > 05/15 04:25:18 DEBUG|autotest_r:0078| /usr/local/autotest/bin/autotest does > not exist on 10.0.0.136 > 05/15 04:25:18 DEBUG| ssh_host:0114| Running (ssh) 'test -x > /home/autotest/bin/autotest' > 05/15 04:25:18 DEBUG|autotest_r:0078| /home/autotest/bin/autotest does not > exist on 10.0.0.136 > 05/15 04:25:18 DEBUG| ssh_host:0114| Running (ssh) 'mkdir -p > /usr/local/autotest' > 05/15 04:25:19 DEBUG| ssh_host:0114| Running (ssh) 'test -w > /usr/local/autotest' > 05/15 04:25:20 INFO |autotest_r:0203| Using installation dir > /usr/local/autotest > 05/15 04:25:20 DEBUG| ssh_host:0114| Running (ssh) 'mkdir -p > /usr/local/autotest' > 05/15 04:25:21 DEBUG| ssh_host:0114| Running (ssh) 'rm -rf > /usr/local/autotest/results/*' > 05/15 04:25:21 INFO |autotest_r:0220| Could not install autotest using the > packaging system: No repos to install an autotest client from. Trying other > methods > 05/15 04:25:21 DEBUG| ssh_host:0114| Running (ssh) 'test -x > /usr/local/autotest' > 05/15 04:25:22 DEBUG| ssh_host:0114| Running (ssh) 'test -d > /usr/local/autotest' > 05/15 04:25:23 DEBUG| ssh_host:0114| Running (ssh) 'rm -rf > /usr/local/autotest && mkdir /usr/local/autotest' > 05/15 04:25:23 DEBUG|base_utils:0076| Running 'scp -rq -o > StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpcsovUa -P 22 > /usr/local/autotest/client/__init__.pyc /usr/local/autotest/client/packages > /usr/local/autotest/client/tmp /usr/local/autotest/client/__init__.py > /usr/local/autotest/client/bin /usr/local/autotest/client/LICENSE > /usr/local/autotest/client/setup_modules.py /usr/local/autotest/client/stderr > /usr/local/autotest/client/config /usr/local/autotest/client/results > /usr/local/autotest/client/setup_modules_unittest.py > /usr/local/autotest/client/common_lib /usr/local/autotest/client/virt > /usr/local/autotest/client/setup_modules.pyc > /usr/local/autotest/client/samples /usr/local/autotest/client/tools > '[email protected]:"/usr/local/autotest"'' > 05/15 04:26:26 DEBUG| ssh_host:0114| Running (ssh) 'mkdir -p > '/usr/local/autotest/site_tests';touch > '/usr/local/autotest/site_tests'/__init__.py;mkdir -p > '/usr/local/autotest/tests';touch > '/usr/local/autotest/tests'/__init__.py;mkdir -p > '/usr/local/autotest/deps';touch '/usr/local/autotest/deps'/__init__.py;mkdir > -p '/usr/local/autotest/profilers';touch > '/usr/local/autotest/profilers'/__init__.py' > 05/15 04:26:28 INFO |autotest_r:0232| Installation of autotest completed > 05/15 04:26:28 DEBUG| ssh_host:0114| Running (ssh) 'true' > 05/15 04:26:29 DEBUG|abstract_s:0456| Host 10.0.0.136 is now up > 05/15 04:26:29 DEBUG|autotest_r:0066| Using existing host autodir: > /usr/local/autotest > 05/15 04:26:29 DEBUG| ssh_host:0114| Running (ssh) 'ls > /usr/local/autotest/bin/autotest > /dev/null 2>&1' > 05/15 04:26:30 DEBUG| ssh_host:0114| Running (ssh) 'umount > /usr/local/autotest/tmp' > 05/15 04:26:31 DEBUG|base_utils:0108| [stderr] umount: > /usr/local/autotest/tmp: not mounted > 05/15 04:26:31 DEBUG| ssh_host:0114| Running (ssh) 'umount > /usr/local/autotest/tests/download' > 05/15 04:26:32 DEBUG|base_utils:0108| [stderr] umount: > /usr/local/autotest/tests/download: not found > 05/15 04:26:32 DEBUG| ssh_host:0114| Running (ssh) 'ls > /usr/local/autotest/bin/autotest > /dev/null 2>&1' > 05/15 04:26:32 DEBUG| ssh_host:0114| Running (ssh) 'umount > /usr/local/autotest/tmp' > 05/15 04:26:33 DEBUG|base_utils:0108| [stderr] umount: > /usr/local/autotest/tmp: not mounted > 05/15 04:26:33 DEBUG| ssh_host:0114| Running (ssh) 'umount > /usr/local/autotest/tests/download' > 05/15 04:26:34 DEBUG|base_utils:0108| [stderr] umount: > /usr/local/autotest/tests/download: not found > 05/15 04:26:34 DEBUG| ssh_host:0114| Running (ssh) 'rm -f > /usr/local/autotest/control.autoserv;rm -f > /usr/local/autotest/control.autoserv.state;rm -f > /usr/local/autotest/control;rm -f /usr/local/autotest/control.state' > 05/15 04:26:35 DEBUG| base_job:0347| Persistent state client.sysinfo now set > to {'test': set([sysinfo.command('dmesg -c', 'dmesg', False), > sysinfo.command('df -mP', 'df', False)]), 'boot': set([sysinfo.command('gcc > --version', 'gcc_--version', False), sysinfo.command('ld --version', > 'ld_--version', False), sysinfo.logfile('/proc/partitions', 'partitions', > False), sysinfo.logfile('/proc/interrupts', 'interrupts', False), > sysinfo.command('hostname', 'hostname', False), sysinfo.command('mount', > 'mount', False), sysinfo.logfile('/proc/cmdline', 'cmdline', True), > sysinfo.logfile('/proc/version', 'version', False), sysinfo.command('lspci > -vvn', 'lspci_-vvn', False), sysinfo.logfile('/proc/slabinfo', 'slabinfo', > False), sysinfo.logfile('/proc/modules', 'modules', False), > sysinfo.logfile('/proc/mounts', 'proc_mounts', False), > sysinfo.logfile('/proc/cpuinfo', 'cpuinfo', False), sysinfo.command('uptime', > 'uptime', False), sysinfo.logfile('/proc/pci', 'pci', False), > sysinfo.logfile('/proc/meminfo', 'meminfo', False), sysinfo.command('uname > -a', 'uname', True)])} > 05/15 04:26:35 DEBUG|base_utils:0076| Running 'scp -rq -o > StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpcsovUa -P 22 > /usr/local/autotest/server/tmp/tmpO23_xQ > '[email protected]:"/usr/local/autotest/control.autoserv.init.state"'' > 05/15 04:26:36 DEBUG|base_utils:0076| Running 'scp -rq -o > StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpcsovUa -P 22 > /tmp/autoserv-gUroAd/tmpcrPhlS > '[email protected]:"/usr/local/autotest/control.autoserv"'' > 05/15 04:26:37 DEBUG| ssh_host:0114| Running (ssh) 'if [ -f > '/proc/sys/kernel/random/boot_id' ]; then cat > '/proc/sys/kernel/random/boot_id'; else echo 'no boot_id available'; fi' > 05/15 04:26:38 DEBUG|base_utils:0108| [stdout] > e0cb59e3-5583-431b-9756-1311f360e053 > 05/15 04:26:38 INFO |autotest_r:0661| Executing > /usr/local/autotest/bin/autotest /usr/local/autotest/control phase 0 > 05/15 04:26:38 DEBUG| ssh_host:0114| Running (ssh) 'mkdir -p /tmp' > 05/15 04:26:38 DEBUG| ssh_host:0114| Running (ssh) 'mktemp -d > /tmp/autoserv-XXXXXX' > 05/15 04:26:39 DEBUG|base_utils:0108| [stdout] /tmp/autoserv-8lu2tD > 05/15 04:26:39 DEBUG|base_utils:0076| Running 'scp -rq -o > StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpcsovUa -P 22 > /usr/local/autotest/server/tmp/tmpPTF0xQ > '[email protected]:"/usr/local/autotest/global_config.ini"'' > 05/15 04:26:40 DEBUG| ssh_host:0114| Running (ssh) 'nohup > /usr/local/autotest/bin/autotestd /tmp/autoserv-8lu2tD -H autoserv --verbose > --hostname=10.0.0.136 --user=autotest /usr/local/autotest/control.autoserv > >/dev/null 2>/dev/null &' > 05/15 04:26:40 DEBUG| ssh_host:0114| Running (ssh) > '/usr/local/autotest/bin/autotestd_monitor /tmp/autoserv-8lu2tD 0 0' > > > --------- stucked forever ---------- > > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > And weird thing is that it does occur sometimes not always. If it > occurs, every time we need to abort that particular job and start all > over again. > This behavior I observed on the latest autotest version also installed > somewhere around mid april, 2k12 > > > I wanted to post this particular question on the forum from long time, > as we faced it many times. > > > Is it related to '/tmp/*', '/var/tmp/*' directory contents ? or any > other ?
Hmm, it seems like some sort of race condition or bug in autotestd_monitor that is being triggered by something in the environment of your board. We'd need to debug autotestd_monitor and see what might be happening... I'd need to think of a way to instrument the code to help you out with this... _______________________________________________ Autotest mailing list [email protected] http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
