On 06/10/2011 09:31 AM, Lucas Meneghel Rodrigues wrote: > On Fri, 2011-05-20 at 14:55 +0800, Yiqiao Pu wrote: >> Transparent hugepage test includes: >> 1)smoking test and stress test as basic test >> Smoking test is test the transparent hugepage is used by kvm and guest. >> Stress test test use a parallel dd to test the stability of transparent >> hugepage >> 2)swapping test >> Bootup a vm and verify that it can be swapped out and swapped in correctly >> 3)defrag test >> Allocate hugepage for libhugetlbfs while defrag is on and off. >> Than compare the result >> And there is a script for env preparing named thp.py. It will set the >> khugepaged to a active mode and it also including a simple test for >> khugepaged. > Hi Yiqiao, I read through the test and have comments to make. Bottom > line, we need to bring the test to use more current autotest API, avoid > use of the commands API and reformulate the test stage setup. See below. > >> Signed-off-by: Yiqiao Pu<[email protected]> >> --- >> client/tests/kvm/scripts/thp.py | 130 >> +++++++++++++++++++++ >> client/tests/kvm/tests/trans_hugepage.py | 120 >> +++++++++++++++++++ >> client/tests/kvm/tests/trans_hugepage_defrag.py | 72 ++++++++++++ >> client/tests/kvm/tests/trans_hugepage_swapping.py | 104 ++++++++++++++++ >> client/tests/kvm/tests_base.cfg.sample | 18 +++ >> 5 files changed, 444 insertions(+), 0 deletions(-) >> create mode 100755 client/tests/kvm/scripts/thp.py >> create mode 100644 client/tests/kvm/tests/trans_hugepage.py >> create mode 100644 client/tests/kvm/tests/trans_hugepage_defrag.py >> create mode 100644 client/tests/kvm/tests/trans_hugepage_swapping.py >> >> diff --git a/client/tests/kvm/scripts/thp.py >> b/client/tests/kvm/scripts/thp.py >> new file mode 100755 >> index 0000000..a1cf288 >> --- /dev/null >> +++ b/client/tests/kvm/scripts/thp.py > ^ We have moved away from having scripts on the scripts dir to perform > pre-test setup. Now it's recommended that such setup code stays either > at: > > * The test itself, in case the code is only going to be used once, such > as unattended install: > > http://autotest.kernel.org/browser/trunk/client/tests/kvm/tests/unattended_install.py > > * The test setup library, for code that is going to be used in multiple > tests (that is the case for THP tests): > > http://autotest.kernel.org/browser/trunk/client/virt/virt_test_setup.py > > See HugePageConfig. Also, there are some other details, discussed > below. > OK. This is an old patch in our internal tree. I only test it to make it can run in new framework, I will update it to date. >> @@ -0,0 +1,130 @@ >> +#!/usr/bin/python >> +# -*- coding: utf-8 -*- >> +import os, sys, time, commands, string, re, stat, shelve >> + >> +""" >> +script to save and restore environment in transparent hugepage test >> +and khuagepaged test >> +""" >> + >> +class THPError(Exception): >> + """ >> + Exception from transparent hugepage preparing scripts >> + """ >> + pass > ^ Maybe we can add more exception types for different things that can go > wrong during the course of the test. > This is only for error message through out in the old scripts. But after move it to framework, we can use the existing error classes :) >> +class THP: >> + def __init__(self): >> + """ >> + Get the configuration path of transparent hugepage, ksm, >> + libhugetlbfs and etc. And the option config parameters from user >> + """ >> + if os.path.isdir("/sys/kernel/mm/redhat_transparent_hugepage"): >> + self.thp_path = '/sys/kernel/mm/redhat_transparent_hugepage' >> + elif os.path.isdir("/sys/kernel/mm/transparent_hugepage"): >> + self.thp_path = '/sys/kernel/mm/transparent_hugepage' >> + else: >> + raise THPError("System don't support transparent hugepage") > ^ Better to use variables, to handle the paths and reduce the amount of > typing: > > RH_THP_PATH = "/sys/kernel/mm/redhat_transparent_hugepage" > UPSTREAM_THP_PATH = "/sys/kernel/mm/transparent_hugepage" > > if os.path.isdir(RH_THP_PATH): > self.thp_path = RH_THP_PATH > > And so on and so forth > OK, I will modify it. >> + >> + self.default_config_file = '/tmp/thp_default_config' >> + # Update the test config dict from environment >> + test_cfg={"%s/defrag" % self.thp_path:"yes", >> + "%s/enabled" % self.thp_path:"always", >> + "%s/khugepaged/defrag" % self.thp_path:"yes", >> + "%s/khugepaged/scan_sleep_millisecs" % self.thp_path:"100", >> + "%s/khugepaged/pages_to_scan" % self.thp_path:"4096", >> + "%s/khugepaged/alloc_sleep_millisecs" % self.thp_path:"100", >> + "/sys/kernel/mm/ksm/run":"1", >> + "/proc/sys/vm/nr_hugepages":"0" >> + } >> + if os.path.isfile("%s/khugepaged/enabled" % self.thp_path): >> + test_cfg["%s/khugepaged/enabled" % self.thp_path] = "always" >> + if os.path.isfile("%s/khugepaged/max_ptes_none" % self.thp_path): >> + test_cfg["%s/khugepaged/max_ptes_none" % self.thp_path] = "511" >> + test_cfg["%s/defrag" % self.thp_path] = "always" >> + >> + tmp_list = [] >> + test_config = str(os.environ['KVM_TEST_thp_test_config']) > ^ Now rather than collecting env variables, we are instantiating the > setup object with the test parameters (see for example the enospc test). > Yes, I see. I will change it. >> + if len(test_config)> 0: >> + tmp_list = re.split(';', test_config) >> + while len(tmp_list)> 0: >> + tmp_cfg = tmp_list.pop() >> + test_cfg[re.split(":", tmp_cfg)[0]] = \ >> + re.split(":", tmp_cfg)[1] >> + self.test_config = test_cfg >> + >> + def save_env(self): >> + """ >> + Save and set the environment including related parts in kernel. >> + Such as ksm and libhugetlbfs. >> + """ >> + fd_default = shelve.open(str(self.default_config_file)) > ^ Now that we have moved from stand alone scripts, we can use classes > and internal object state to store such information, rather than saving > it on a file, so this entire function can be cut off. > I agree we can store it in a internal variable instead of write a file, but we still need this function. This is because in thp test cases, we will change the host system environment. We can not left this to influence the following tests running in the same system. So this function is using for saving the original system environment to make it possible to restore it. >> + for f in self.test_config.keys(): >> + parameter = file(f,'r').read() >> + if string.find(f, "enabled")> 0 or string.find(f, "defrag")> >> 0: >> + parameter = re.split("\[|\]", parameter)[1] + '\n' >> + fd_default[f] = parameter >> + fd_default.close() >> + >> + def set_env(self): >> + """ >> + After khugepaged test inuse_config is already set to an active mode >> of >> + transparent hugepage. Get some special config of sub test and set >> it. >> + """ >> + if len(self.test_config)> 0: > ^ You can test for an empty string just like: > > if self.test_config: > > Without using the length, as an empty string evals to False. > OK. >> + for path in self.test_config.keys(): >> + file(path, 'w').write(self.test_config[path]) >> + >> + def set_params(self, path_dict={}, filename="", value=""): >> + """ >> + Set the value of a config files in one path which is >> + store in a path dict >> + @param path_dict: Dict of files' pathes {path : value} >> + @param filename: Name of file to setup >> + @param value: Value set to the configuration files >> + """ >> + for path in path_dict.keys(): >> + if string.find(path, filename)> 0: >> + try: >> + file(path, "w").write(value) >> + except IOError, e: >> + raise THPError("Can not set %s to %s: %s" % \ >> + (value, filename, e)) >> + >> + def khugepaged_test(self): >> + """ >> + Start, stop and frequency change test for khugepaged >> + """ >> + action_list = [("never", 256),("always", 0),("never", 256)] >> + # ret is the (status, value) >> + ret = [self.set_params(self.test_config, "enabled", a) or\ >> + commands.getstatusoutput('pgrep khugepaged') for (a, r) in >> action_list] > ^ I don't understand, set_params does not return a value, how can we use > values it yields to populate the results on this list? I assume this > list will return always the result of commands.getstatusoutput('pgrep > khugepaged'), as self.set_params(self.test_config, "enabled", a) will > always evaluate to None... > The ret is the result of commands.getstatusoutput(). We only want to check the khugepaged process status here. And I already improve the code, now we only need call opne(files, "w").write() to set the value, it will raise an error if it failed. >> + for i in range(len(action_list)): >> + if ret[i][0] != action_list[i][1]: >> + raise THPError("khugepaged can not set to status %s" %\ >> + action_list[i][0]) >> + >> + >> + def restore_default_config(self): >> + """: >> + Restore the default configuration to host after test >> + """ >> + fd = shelve.open(self.default_config_file) >> + for path in fd.keys(): >> + file(path, 'w').write(fd[path]) >> + fd.close() >> + >> +if __name__ == "__main__": >> + if len(sys.argv)< 2: >> + raise THPError("Please use -s for set and -r for restore") > ^ Here we can replace calling -s or -r with functions setup() and > cleanup() of the config object. > OK. >> + run_time = sys.argv[1][1:] >> + trans_hugepage = THP() >> + if run_time == "s": >> + trans_hugepage.save_env() >> + trans_hugepage.khugepaged_test() >> + trans_hugepage.set_env() >> + elif run_time == "r": >> + trans_hugepage.restore_default_config() >> + else: >> + raise THPError("Please use -s for set and -r for restore") >> diff --git a/client/tests/kvm/tests/trans_hugepage.py >> b/client/tests/kvm/tests/trans_hugepage.py >> new file mode 100644 >> index 0000000..5cd3ef9 >> --- /dev/null >> +++ b/client/tests/kvm/tests/trans_hugepage.py >> @@ -0,0 +1,120 @@ >> +import logging, time, commands, os, string, re >> +from autotest_lib.client.common_lib import error >> +from autotest_lib.client.virt import virt_test_utils, aexpect >> + >> +def run_trans_hugepage(test, params, env): >> + """ >> + KVM khugepage user side test: >> + 1) Smoking test >> + 2) Stress test >> + >> + @param test: kvm test object. >> + @param params: Dictionary with test parameters. >> + @param env: Dictionary with the test environment. >> + """ >> + def get_mem_status(params, type): >> + if type == "host": >> + s, info = commands.getstatusoutput("cat /proc/meminfo") > ^ We should use utils.system_output here > OK. >> + else: >> + s, info = session.get_command_status_output("cat /proc/meminfo") > ^ Here we could use info = session.cmd("cat...") and handle the possible > ShellErrors that can be thrown. > OK. >> + if s != 0: >> + raise error.TestError("Can not get memory info in guest") >> + for h in re.split("\n+", info): >> + if h.startswith("%s" % params): >> + output = re.split('\s+', h)[1] >> + return output >> + # Check khugepage is used by guest >> + dd_timeout = float(params.get("dd_timeout", 900)) >> + fail = 0 >> + nr_ah = [] >> + mem = params['mem'] >> + >> + debugfs_flag = 1 >> + if not os.path.ismount('/mnt/debugfs'): >> + if not os.path.isdir('/mnt/debugfs'): >> + os.makedirs('/mnt/debugfs') >> + s, o = commands.getstatusoutput("mount -t debugfs none >> /mnt/debugfs") > ^ Similarly, better to use autotest API rather than commands. This > comment applies to other uses of the commands API. > Yes, I will update the interface to the new framework. >> + if s != 0: >> + debugfs_flag = 0 >> + logging.info("Warning: debugfs is unusable") >> + >> + logging.info("Smoking test start") >> + login_timeout = float(params.get("login_timeout", "3600")) >> + vm = virt_test_utils.get_living_vm(env, params.get("main_vm")) >> + session = virt_test_utils.wait_for_login(vm, timeout=login_timeout) >> + >> + nr_ah.append(get_mem_status('AnonHugePages', 'host')) >> + if nr_ah[0]<= 0: >> + raise error.TestFail("VM is not using transparent hugepage") >> + >> + # Protect system from oom killer >> + if int(get_mem_status('MemFree', 'guest')) / 1024< mem : >> + mem = int(get_mem_status('MemFree', 'guest')) / 1024 >> + cmd = "ls -d /space" >> + status = session.get_command_status(cmd) >> + if status != 0: >> + cmd = "mkdir /space&> /dev/null" >> + status = session.get_command_status(cmd) >> + if status != 0: >> + raise error.TestError("Can not mkdir in guest") > ^ Better to just use session.cmd, since that API is supposed to throw an > error anyway. > OK. >> + >> + cmd = "mount -t tmpfs -o size=%sM none /space" % str(mem) >> + status = session.get_command_status(cmd) >> + if status != 0: >> + raise error.TestError("Can not mount tmpfs in guest") >> + >> + count = mem / 4 >> + cmd = "dd if=/dev/zero of=/space/1 bs=4000000 count=%s" % count >> + try: >> + status = session.get_command_status(cmd, timeout=dd_timeout) >> + except aexpect.ShellStatusError, e: >> + logging.debug(e) >> + raise error.TestFail("Could not get exit status of command %s" % >> cmd) >> + except Exception, e: >> + logging.debug(e) >> + raise error.TestFail("Fail command: %s. Please check debug log!" % >> cmd) >> + >> + if status != 0: >> + raise error.TestError("dd failed in guest") >> + >> + nr_ah.append(get_mem_status('AnonHugePages', 'host')) >> + >> + if nr_ah[1]<= nr_ah[0]: >> + logging.info("WARNING: VM don't use transparent hugepage when dd") >> + >> + if debugfs_flag == 1: >> + if int(open('/mnt/debugfs/kvm/largepages', 'r').read())<= 0: >> + raise error.TestFail("KVM doesn't use transparenthugepage") >> + >> + logging.info("Smoking test finished") >> + >> + # Use parallel dd as stress for memory >> + count = count / 3 >> + logging.info("Stress test start") >> + cmd = "for i in `seq %s`;do dd if=/dev/zero of=/space/$i" % count \ >> + + " bs=4000000 count=1& done " >> + try: >> + status, output = session.get_command_status_output(cmd, >> + >> timeout=dd_timeout) >> + except aexpect.ShellStatusError, e: >> + logging.debug(e) >> + raise error.TestFail("Could not get exit status of command %s" % >> cmd) >> + except Exception, e: >> + logging.debug(e) >> + raise error.TestFail("Fail command: %s. Please check debug log!" % >> cmd) >> + >> + if len(re.findall("No space", output))> count * 0.05: >> + raise error.TestFail("Too many dd failed in guest") >> + >> + s, o = session.get_command_status_output('pidof dd') >> + if s == 0 : >> + for i in re.split('\n+', o): >> + session.get_command_status_output('kill -9 %s' % i) >> + >> + cmd = "umount /space" >> + status, output = session.get_command_status_output(cmd) >> + if status != 0: >> + raise error.TestFail("Can not umount tmpfs after stress test%s"% >> output) >> + logging.info("Stress test finished") >> + >> + session.close() >> diff --git a/client/tests/kvm/tests/trans_hugepage_defrag.py >> b/client/tests/kvm/tests/trans_hugepage_defrag.py >> new file mode 100644 >> index 0000000..14ce546 >> --- /dev/null >> +++ b/client/tests/kvm/tests/trans_hugepage_defrag.py >> @@ -0,0 +1,72 @@ >> + >> +import logging, time, commands, os, string, re >> +from autotest_lib.client.common_lib import error >> +from autotest_lib.client.virt import virt_test_utils >> + >> +def run_trans_hugepage_defrag(test, params, env): >> + """ >> + KVM khugepage user side test: >> + 1) Verify that the host supports khugepahe. If it does proceed with the >> test. >> + 2) Verify that the khugepage can be used in host >> + 3) Verify that the khugepage can be used in guest >> + 4) Migration while using khuge page >> + >> + @param test: kvm test object. >> + @param params: Dictionary with test parameters. >> + @param env: Dictionary with the test environment. >> + """ >> + def get_mem_status(params): >> + for line in file('/proc/meminfo', 'r').readlines(): >> + if line.startswith("%s" % params): >> + output = re.split('\s+', line)[1] >> + return output >> + >> + def set_libhugetlbfs(number): >> + fd = file("/proc/sys/vm/nr_hugepages", "w+") > ^ Better to not use fd (commonly stands for file descriptor) with a file > object (which is what fd actually is), to avoid people confused while > reading the code. > This is a file in proc system. Do you suggestion to give this variable a special name as nr_hugepages? >> + fd.write(number) >> + fd.seek(0) >> + ret = fd.read() >> + return int(ret) >> + >> + # Test the defrag >> + logging.info("Defrag test start") >> + login_timeout = float(params.get("login_timeout", 360)) >> + vm = virt_test_utils.get_living_vm(env, params.get("main_vm")) >> + session = virt_test_utils.wait_for_login(vm, timeout=login_timeout) >> + >> + try: >> + if not os.path.isdir('/space'): >> + os.makedirs('/space') >> + if os.system("mount -t tmpfs none /space"): >> + raise error.TestError("Can not mount tmpfs") >> + >> + # Try to make some fragment in memory, the total size of fragments >> is 1G >> + cmd = "for i in `seq 262144`; do dd if=/dev/urandom of=/space/$i >> bs=4K count=1& done" >> + s, o = commands.getstatusoutput(cmd) >> + if s != 0: >> + raise error.TestError("Can not dd in host") >> + finally: >> + s, o = commands.getstatusoutput("umount /space") >> + if s != 0: >> + raise error.TestError("Can not free tmpfs") >> + >> + total = int(get_mem_status('MemTotal')) >> + hugepagesize = int(get_mem_status('Hugepagesize')) >> + nr_full = str(total / hugepagesize) >> + # Allocate hugepage for libhugetlbfs before and after enable defrag. >> + # And see the different >> + nr = [] >> + nr.append(set_libhugetlbfs(nr_full)) >> + try: >> + defrag_path = >> '/sys/kernel/mm/redhat_transparent_hugepage/khugepaged/defrag' >> + file(str(defrag_path),'w').write('yes') >> + except IOError, e: >> + raise error.TestFail("Can not start defrag:%s" % e) >> + >> + time.sleep(1) >> + nr.append(set_libhugetlbfs(nr_full)) >> + >> + if nr[0]>= nr[1]: >> + raise error.TestFail("Do not defrag in the system %s" % nr) > ^ It was not possible to defrag memory in host... > We tried to dd regular pages in tmpfs(make fragment). And then release them. And try to use hugepage(hugetlbfs) in system. After there are fragments in system, the daemon process of thp will take some time to scan and relocated hugepages. This can not be finished in a very short time as we don't set it run period very short. So the page number of hugetlbfs should increase here. >> + session.close() >> + logging.info("Defrag test succeed") >> diff --git a/client/tests/kvm/tests/trans_hugepage_swapping.py >> b/client/tests/kvm/tests/trans_hugepage_swapping.py >> new file mode 100644 >> index 0000000..a58ee6b >> --- /dev/null >> +++ b/client/tests/kvm/tests/trans_hugepage_swapping.py >> @@ -0,0 +1,104 @@ >> +import logging, time, commands, os, string, re >> +from autotest_lib.client.common_lib import error >> +from autotest_lib.client.virt import virt_test_utils >> + >> +def run_trans_hugepage_swapping(test, params, env): >> + """ >> + KVM khugepage user side test: >> + 1) Verify that the hugepages can be swapped in/out. >> + >> + @param test: kvm test object. >> + @param params: Dictionary with test parameters. >> + @param env: Dictionary with the test environment. >> + """ >> + def get_args(args_list): >> + """ >> + Get the memory arguments from system > ^ Populates a dict with memory information extracted from the system > OK. >> + """ >> + args_list_tmp = args_list.copy() >> + for line in file('/proc/meminfo', 'r').readlines(): >> + for key in args_list_tmp.keys(): >> + if line.startswith("%s" % args_list_tmp[key]): >> + args_list_tmp[key] = int(re.split('\s+', line)[1]) >> + return args_list_tmp >> + >> + # Swapping test >> + logging.info("Swapping test start") >> + # Parameters of memory information >> + # @total: Memory size >> + # @free: Free memory size >> + # @swap_size: Swap size >> + # @swap_free: Free swap size >> + # @hugepage_size: Page size of one hugepage >> + # @page_size: The biggest page size that app can ask for >> + args_dict_check = {"free" : "MemFree", "swap_size" : "SwapTotal", >> + "swap_free" : "SwapFree", "total" : "MemTotal", >> + "hugepage_size" : "Hugepagesize",} >> + args_dict = get_args(args_dict_check) >> + swap_free = [] >> + total = int(args_dict['total']) / 1024 >> + free = int(args_dict['free']) / 1024 >> + swap_size = int(args_dict['swap_size']) / 1024 >> + swap_free.append(int(args_dict['swap_free'])/1024) >> + hugepage_size = int(args_dict['hugepage_size']) / 1024 >> + dd_timeout = float(params.get("dd_timeout", 900)) >> + login_timeout = float(params.get("login_timeout", 360)) >> + check_cmd_timeout = float(params.get("check_cmd_timeout", 900)) >> + # If swap is enough fill all memory with dd >> + if swap_free> (total - free): >> + count = total / hugepage_size >> + tmpfs_size = total >> + else: >> + count = free / hugepage_size >> + tmpfs_size = free >> + >> + if swap_size<= 0: >> + raise logging.info("Host don't have swap") >> + try: >> + if not os.path.isdir('/space'): >> + os.makedirs('/space') >> + if os.system("mount -t tmpfs -o size=%sM none /space" % >> tmpfs_size): >> + raise error.TestError("Can not mount tmpfs") > ^ Use utils.run() > OK. Will change the interface. >> + # Set the memory size of vm >> + # To ignore the oom killer set it to the free swap size >> + vm = virt_test_utils.get_living_vm(env, params.get("main_vm")) >> + if int(params['mem'])> swap_free[0]: >> + vm.destroy() >> + vm_name = 'vmsw' >> + vm0 = params.get("main_vm") >> + vm0_key = virt_utils.env_get_vm(env, vm0) >> + params['vms'] = params['vms'] + " " + vm_name >> + params['mem'] = str(swap_free[0]) >> + vm_key = vm0_key.clone(vm0, params) >> + virt_utils.env_register_vm(env, vm_name, vm_key) >> + virt_env_preprocessing.preprocess_vm(test, params, env, vm_name) >> + vm_key.create() >> + session = virt_utils.wait_for(vm_key.remote_login, >> timeout=login_timeout) >> + else: >> + session = virt_test_utils.wait_for_login(vm, >> timeout=login_timeout) >> + >> + cmd = "dd if=/dev/zero of=/space/zero bs=%s000000 count=%s" % \ >> + (hugepage_size, count) >> + s, o = commands.getstatusoutput(cmd) >> + if s != 0: >> + raise error.TestError("dd failed in host: %s" % o) >> + >> + args_dict = get_args(args_dict_check) >> + swap_free.append(int(args_dict['swap_free'])/1024) >> + >> + if swap_free[1] - swap_free[0]>= 0: >> + raise error.TestFail("Nothing is swapped") >> + >> + # Check the vm still alive >> + s = session.get_command_status("find / -name \"*\"", >> + timeout=check_cmd_timeout) > ^ why a command that takes so long to execute to check if the VM is > still alive? I suppose any simple command could verify whether the > session is alive, not to mention we can also use a vm.verify_alive() > here to check if the VM process is well and monitors responsive. > Sorry. This is a bad comments. I just try to use find to make the all possible part of guest is swapped in to memory again and should be OK, not just check the guest is alive as a process. >> + if s != 0 : >> + raise error.TestFail("There is something wrong after swap") >> + finally: >> + s, o = commands.getstatusoutput("umount /space") >> + if s != 0: >> + logging.warning("Can not umount tmpfs after swap") >> + logging.info("Swapping test succeed") >> + >> + session.close() >> diff --git a/client/tests/kvm/tests_base.cfg.sample >> b/client/tests/kvm/tests_base.cfg.sample >> index 78c84c6..3c8e552 100644 >> --- a/client/tests/kvm/tests_base.cfg.sample >> +++ b/client/tests/kvm/tests_base.cfg.sample >> @@ -965,6 +965,24 @@ variants: >> kill_vm_gracefully = no >> # Do not define test variants below shutdown >> >> + - trans_hugepage: >> + pre_command = python scripts/thp.py -s >> + post_command = python scripts/thp.py -r >> + thp_default_config = "/tmp/thp_default_config" >> + thp_test_config = "" >> + kill_vm = yes >> + login_timeout = 360 >> + variants: >> + - base: >> + type = trans_hugepage >> + dd_timeout = 900 >> + - defrag: >> + type = trans_hugepage_defrag >> + - swapping: >> + type = trans_hugepage_swapping >> + dd_timeout = 900 >> + check_cmd_timeout = 900 >> + >> >> # NICs >> variants: >
_______________________________________________ Autotest mailing list [email protected] http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
