Re: [OE-core] [PATCH] test-result-log: testcase management tool to store test result
Hi Richard,
To address the maintainable concern, I had restructured the codes in order to
enable oeqa framework to write out a json file directly for testresult.
Attached were the patches to enble oeqa framework to write testresult into json
files, where these files will later be used by the future QA test case
management tools (eg. store testresult, test reporting, execute manual test
case and write testresult to json).
This patch include enable oe-selftest to write json testresult. I had tested
these patches on our local server where it will write out json files for
testresult as expected.
Please let me know if you have any more feedback. Thank you very much!
Best regards,
Yeoh Ee Peng
-Original Message-
From: richard.pur...@linuxfoundation.org
[mailto:richard.pur...@linuxfoundation.org]
Sent: Tuesday, September 11, 2018 11:09 PM
To: Yeoh, Ee Peng ;
openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH] test-result-log: testcase management tool to
store test result
Hi Ee Peng,
I've been having a look at this code and whilst some of it is good, I also have
some concerns. With it coming in so late in the cycle, its made it hard to have
time to review it and allow time to get it right.
With something as important as this to the way the future QA work is done, we
do need to ensure we use the right approach and that its maintainable.
The patches are ok as a first attempt at this. My biggest concern is that its
currently parsing log files which we control and generate within out own
codebase and that parsing is likely to break. In particular, these lines worry
me from the qalogparser:
regex = ".*RESULTS - (?P.*) - Testcase .*:
(?PPASSED|FAILED|SKIPPED|ERROR|UNKNOWN).*$"
regex = "core-image.*().*Ran.*tests in .*s"
regex = "DEBUG: launchcmd=runqemu*
qemu_list = ['qemuarm', 'qemuarm64', 'qemumips', 'qemumips64', 'qemuppc',
'qemux86', 'qemux86-64']
since here we're hardcoding the list of qemu's we support, we're also only
allowing core-image-* and we're relying upon the results format not changing.
That makes it hard for anyone to extend/reuse this or to use it with real
hardware?
For example the recent oe-selftest parallelisation code did change the output
of the tests slightly. I'm not sure if this broke the parsing or not but it is
an example of the kind of fragility this code has.
What would probably work better for us is if the oeqa framework wrote out a
json file directly containing the information we need in it, then this code
would just need to collect up the json files.
I'm also a little concerned at the way unittest discovery is being done,
grepping for *.py files as far as I understand it. We should probably use the
list options to the various current test pieces? Also, this is something we
probably only ever need to do once to seed the QA results store?
Finally, much of the code is using "internal" methods prefixed with "_". I can
understand why but it seems the code doesn't have a good well structured public
API as a result.
As such there may me a little too much work needed on this to get it in for 2.6
:(
Cheers,
Richard
--- Begin Message ---
To enable selftest to write testresult into json files, where
these json files will be used by future test case management
tools for test reporting.
Signed-off-by: Yeoh Ee Peng
---
meta/lib/oeqa/selftest/context.py | 12 ++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oeqa/selftest/context.py
b/meta/lib/oeqa/selftest/context.py
index c78947e..9ed22ff 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -73,6 +73,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
parser.add_argument('--machine', required=False, choices=['random',
'all'],
help='Run tests on different machines
(random/all).')
+
+parser.add_argument('-ej', '--export-json', action='store_true',
+help='Output test result in json format to files.')
parser.set_defaults(func=self.run)
@@ -99,8 +102,8 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
return cases_paths
def _process_args(self, logger, args):
-args.output_log = '%s-results-%s.log' % (self.name,
-time.strftime("%Y%m%d%H%M%S"))
+args.test_start_time = time.strftime("%Y%m%d%H%M%S")
+args.output_log = '%s-results-%s.log' % (self.name,
args.test_start_time)
args.test_data_file = None
args.CASES_PATHS = None
@@ -222,6 +225,11 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
rc = self.tc.runTests(**self.tc_kwargs['run'])
rc.logDetails()
rc.logSummary(self.name)
+if args.export_json:
+json_result_di
Re: [OE-core] [PATCH] test-result-log: testcase management tool to store test result
Hi Richard, Great thanks for your feedbacks! I will work on the improvement to address your concerns. Best regards, Ee Peng -Original Message- From: richard.pur...@linuxfoundation.org [mailto:richard.pur...@linuxfoundation.org] Sent: Tuesday, September 11, 2018 11:09 PM To: Yeoh, Ee Peng ; openembedded-core@lists.openembedded.org Subject: Re: [OE-core] [PATCH] test-result-log: testcase management tool to store test result Hi Ee Peng, I've been having a look at this code and whilst some of it is good, I also have some concerns. With it coming in so late in the cycle, its made it hard to have time to review it and allow time to get it right. With something as important as this to the way the future QA work is done, we do need to ensure we use the right approach and that its maintainable. The patches are ok as a first attempt at this. My biggest concern is that its currently parsing log files which we control and generate within out own codebase and that parsing is likely to break. In particular, these lines worry me from the qalogparser: regex = ".*RESULTS - (?P.*) - Testcase .*: (?PPASSED|FAILED|SKIPPED|ERROR|UNKNOWN).*$" regex = "core-image.*().*Ran.*tests in .*s" regex = "DEBUG: launchcmd=runqemu* qemu_list = ['qemuarm', 'qemuarm64', 'qemumips', 'qemumips64', 'qemuppc', 'qemux86', 'qemux86-64'] since here we're hardcoding the list of qemu's we support, we're also only allowing core-image-* and we're relying upon the results format not changing. That makes it hard for anyone to extend/reuse this or to use it with real hardware? For example the recent oe-selftest parallelisation code did change the output of the tests slightly. I'm not sure if this broke the parsing or not but it is an example of the kind of fragility this code has. What would probably work better for us is if the oeqa framework wrote out a json file directly containing the information we need in it, then this code would just need to collect up the json files. I'm also a little concerned at the way unittest discovery is being done, grepping for *.py files as far as I understand it. We should probably use the list options to the various current test pieces? Also, this is something we probably only ever need to do once to seed the QA results store? Finally, much of the code is using "internal" methods prefixed with "_". I can understand why but it seems the code doesn't have a good well structured public API as a result. As such there may me a little too much work needed on this to get it in for 2.6 :( Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] test-result-log: testcase management tool to store test result
Hi Ee Peng, I've been having a look at this code and whilst some of it is good, I also have some concerns. With it coming in so late in the cycle, its made it hard to have time to review it and allow time to get it right. With something as important as this to the way the future QA work is done, we do need to ensure we use the right approach and that its maintainable. The patches are ok as a first attempt at this. My biggest concern is that its currently parsing log files which we control and generate within out own codebase and that parsing is likely to break. In particular, these lines worry me from the qalogparser: regex = ".*RESULTS - (?P.*) - Testcase .*: (?PPASSED|FAILED|SKIPPED|ERROR|UNKNOWN).*$" regex = "core-image.*().*Ran.*tests in .*s" regex = "DEBUG: launchcmd=runqemu* qemu_list = ['qemuarm', 'qemuarm64', 'qemumips', 'qemumips64', 'qemuppc', 'qemux86', 'qemux86-64'] since here we're hardcoding the list of qemu's we support, we're also only allowing core-image-* and we're relying upon the results format not changing. That makes it hard for anyone to extend/reuse this or to use it with real hardware? For example the recent oe-selftest parallelisation code did change the output of the tests slightly. I'm not sure if this broke the parsing or not but it is an example of the kind of fragility this code has. What would probably work better for us is if the oeqa framework wrote out a json file directly containing the information we need in it, then this code would just need to collect up the json files. I'm also a little concerned at the way unittest discovery is being done, grepping for *.py files as far as I understand it. We should probably use the list options to the various current test pieces? Also, this is something we probably only ever need to do once to seed the QA results store? Finally, much of the code is using "internal" methods prefixed with "_". I can understand why but it seems the code doesn't have a good well structured public API as a result. As such there may me a little too much work needed on this to get it in for 2.6 :( Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH] test-result-log: testcase management tool to store test result
These scripts were developed as an alternative testcase management
tool to Testopia. Using these scripts, user can store test result
from OEQA automated testcase execution.
These scripts will store test result & log in GIT repository.
To use these scripts, first source oe environment, then run the
entry point script to look for help.
$ test-result-log
To store test result for OEQA automated testcase, execute the below
$ test-result-log store-auto /
Signed-off-by: Yeoh Ee Peng
---
scripts/lib/testresultlog/__init__.py | 1 +
scripts/lib/testresultlog/gitstore.py | 250 ++
scripts/lib/testresultlog/oeqalogparser.py| 97 ++
scripts/lib/testresultlog/oeqatestdiscover.py | 51 ++
scripts/lib/testresultlog/storeauto.py| 125 +
scripts/test-result-log | 108 +++
6 files changed, 632 insertions(+)
create mode 100644 scripts/lib/testresultlog/__init__.py
create mode 100644 scripts/lib/testresultlog/gitstore.py
create mode 100644 scripts/lib/testresultlog/oeqalogparser.py
create mode 100644 scripts/lib/testresultlog/oeqatestdiscover.py
create mode 100644 scripts/lib/testresultlog/storeauto.py
create mode 100755 scripts/test-result-log
diff --git a/scripts/lib/testresultlog/__init__.py
b/scripts/lib/testresultlog/__init__.py
new file mode 100644
index 000..d3f5a12
--- /dev/null
+++ b/scripts/lib/testresultlog/__init__.py
@@ -0,0 +1 @@
+
diff --git a/scripts/lib/testresultlog/gitstore.py
b/scripts/lib/testresultlog/gitstore.py
new file mode 100644
index 000..866d7d6
--- /dev/null
+++ b/scripts/lib/testresultlog/gitstore.py
@@ -0,0 +1,250 @@
+# test case management tool - store test result to git repository
+#
+# Copyright (c) 2018, Intel Corporation.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms and conditions of the GNU General Public License,
+# version 2, as published by the Free Software Foundation.
+#
+# This program is distributed in the hope it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+# more details.
+#
+import tempfile
+import os
+import pathlib
+import json
+import subprocess
+import shutil
+import scriptpath
+scriptpath.add_bitbake_lib_path()
+scriptpath.add_oe_lib_path()
+from oeqa.utils.git import GitRepo, GitError
+
+class GitStore(object):
+
+def __init__(self):
+self.script_path = os.path.dirname(os.path.realpath(__file__))
+self.base_path = self.script_path + '/../../..'
+
+def _get_project_environment_directory_path(self, project_dir,
test_environment_list):
+project_env_dir = project_dir
+for env in test_environment_list:
+project_env_dir = os.path.join(project_env_dir, env)
+return project_env_dir
+
+def _get_testmodule_list(self, testmodule_testsuite_dict):
+return sorted(list(testmodule_testsuite_dict.keys()))
+
+def _get_testcase_list(self, testsuite_list, testsuite_testcase_dict):
+testcase_list = []
+for testsuite in sorted(testsuite_list):
+if testsuite in testsuite_testcase_dict:
+for testcase in testsuite_testcase_dict[testsuite]:
+testcase_list.append(testcase)
+return testcase_list
+
+def _get_testcase_status(self, testcase, testcase_status_dict):
+if testcase in testcase_status_dict:
+return testcase_status_dict[testcase]
+return ""
+
+def _create_testcase_dict(self, testcase_list, testcase_status_dict):
+testcase_dict = {}
+for testcase in sorted(testcase_list):
+testcase_status = self._get_testcase_status(testcase,
testcase_status_dict)
+testcase_dict[testcase] = {"testresult": testcase_status,"bugs":
""}
+return testcase_dict
+
+def _create_testsuite_testcase_teststatus_json_object(self,
testsuite_list, testsuite_testcase_dict, testcase_status_dict):
+json_object = {'testsuite':{}}
+testsuite_dict = json_object['testsuite']
+for testsuite in sorted(testsuite_list):
+testsuite_dict[testsuite] = {'testcase': {}}
+testsuite_dict[testsuite]['testcase'] =
self._create_testcase_dict(testsuite_testcase_dict[testsuite],
testcase_status_dict)
+return json_object
+
+def _create_testsuite_json_formatted_string(self, testsuite_list,
testsuite_testcase_dict, testcase_status_dict):
+testsuite_testcase_list =
self._create_testsuite_testcase_teststatus_json_object(testsuite_list,
testsuite_testcase_dict, testcase_status_dict)
+return json.dumps(testsuite_testcase_list, sort_keys=True, indent=4)
+
+def _write_testsuite_testcase_json_formatted_string_to_file(self,
file_path, file_content):
+with open(file_path, 'w') as the_file:
+
[OE-core] [PATCH] test-result-log: testcase management tool to store test result
These scripts were developed as an alternative testcase management
tool to Testopia. Using these scripts, user can store test result
from OEQA automated testcase execution.
These scripts will store test result & log in GIT repository.
To use these scripts, first source oe environment, then run the
entry point script to look for help.
$ test-result-log
To store test result for OEQA automated testcase, execute the below
$ test-result-log store-auto /
Signed-off-by: Yeoh Ee Peng
---
scripts/lib/testresultlog/__init__.py | 1 +
scripts/lib/testresultlog/gitstore.py | 250 ++
scripts/lib/testresultlog/oeqalogparser.py| 97 ++
scripts/lib/testresultlog/oeqatestdiscover.py | 51 ++
scripts/lib/testresultlog/storeauto.py| 125 +
scripts/test-result-log | 97 ++
6 files changed, 621 insertions(+)
create mode 100644 scripts/lib/testresultlog/__init__.py
create mode 100644 scripts/lib/testresultlog/gitstore.py
create mode 100644 scripts/lib/testresultlog/oeqalogparser.py
create mode 100644 scripts/lib/testresultlog/oeqatestdiscover.py
create mode 100644 scripts/lib/testresultlog/storeauto.py
create mode 100755 scripts/test-result-log
diff --git a/scripts/lib/testresultlog/__init__.py
b/scripts/lib/testresultlog/__init__.py
new file mode 100644
index 000..d3f5a12
--- /dev/null
+++ b/scripts/lib/testresultlog/__init__.py
@@ -0,0 +1 @@
+
diff --git a/scripts/lib/testresultlog/gitstore.py
b/scripts/lib/testresultlog/gitstore.py
new file mode 100644
index 000..866d7d6
--- /dev/null
+++ b/scripts/lib/testresultlog/gitstore.py
@@ -0,0 +1,250 @@
+# test case management tool - store test result to git repository
+#
+# Copyright (c) 2018, Intel Corporation.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms and conditions of the GNU General Public License,
+# version 2, as published by the Free Software Foundation.
+#
+# This program is distributed in the hope it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+# more details.
+#
+import tempfile
+import os
+import pathlib
+import json
+import subprocess
+import shutil
+import scriptpath
+scriptpath.add_bitbake_lib_path()
+scriptpath.add_oe_lib_path()
+from oeqa.utils.git import GitRepo, GitError
+
+class GitStore(object):
+
+def __init__(self):
+self.script_path = os.path.dirname(os.path.realpath(__file__))
+self.base_path = self.script_path + '/../../..'
+
+def _get_project_environment_directory_path(self, project_dir,
test_environment_list):
+project_env_dir = project_dir
+for env in test_environment_list:
+project_env_dir = os.path.join(project_env_dir, env)
+return project_env_dir
+
+def _get_testmodule_list(self, testmodule_testsuite_dict):
+return sorted(list(testmodule_testsuite_dict.keys()))
+
+def _get_testcase_list(self, testsuite_list, testsuite_testcase_dict):
+testcase_list = []
+for testsuite in sorted(testsuite_list):
+if testsuite in testsuite_testcase_dict:
+for testcase in testsuite_testcase_dict[testsuite]:
+testcase_list.append(testcase)
+return testcase_list
+
+def _get_testcase_status(self, testcase, testcase_status_dict):
+if testcase in testcase_status_dict:
+return testcase_status_dict[testcase]
+return ""
+
+def _create_testcase_dict(self, testcase_list, testcase_status_dict):
+testcase_dict = {}
+for testcase in sorted(testcase_list):
+testcase_status = self._get_testcase_status(testcase,
testcase_status_dict)
+testcase_dict[testcase] = {"testresult": testcase_status,"bugs":
""}
+return testcase_dict
+
+def _create_testsuite_testcase_teststatus_json_object(self,
testsuite_list, testsuite_testcase_dict, testcase_status_dict):
+json_object = {'testsuite':{}}
+testsuite_dict = json_object['testsuite']
+for testsuite in sorted(testsuite_list):
+testsuite_dict[testsuite] = {'testcase': {}}
+testsuite_dict[testsuite]['testcase'] =
self._create_testcase_dict(testsuite_testcase_dict[testsuite],
testcase_status_dict)
+return json_object
+
+def _create_testsuite_json_formatted_string(self, testsuite_list,
testsuite_testcase_dict, testcase_status_dict):
+testsuite_testcase_list =
self._create_testsuite_testcase_teststatus_json_object(testsuite_list,
testsuite_testcase_dict, testcase_status_dict)
+return json.dumps(testsuite_testcase_list, sort_keys=True, indent=4)
+
+def _write_testsuite_testcase_json_formatted_string_to_file(self,
file_path, file_content):
+with open(file_path, 'w') as the_file:
+
Re: [OE-core] [PATCH] test-result-log: testcase management tool to store test result
Same codes were available in github with better viewing and reviewing. https://github.com/epyeoh/openembedded-core/tree/test-result-log/scripts/lib/testresultlog -Original Message- From: Yeoh, Ee Peng Sent: Friday, August 10, 2018 4:42 PM To: openembedded-core@lists.openembedded.org Cc: Eggleton, Paul ; Burton, Ross ; Mittal, Anuj ; Sangal, Apoorv ; richard.pur...@linuxfoundation.org Subject: RE: [PATCH] test-result-log: testcase management tool to store test result Hi Richard, Ross, Paul, Anuj, As per our conversation in the past, there was a need to provide alternative testcase management tool to Testopia in order to achieve our goals of: 1) To unblocked Bugzilla upgrade to a newer version in order to receive the improvement in WebServices, performance, and GUI. 2) To improve overall QA testing efficiency where it reduce maintenance cost when Testopia need to manually synchronize the test case content between Testopia and the available automated test cases inside OEQA, also it minimize test execution rerun where someone need to retrieve test failure log information from previous QA cycle (Testopia does not store test log due to its limitation on Testopia API for automation). To provide alternative testcase management tool to Testopia, here was the first batch of patch target to enable storing test result for OEQA automated testcase execution and view test summary report. There are another patch planned to enable manual test execution and store/view test result. With inputs from Paul and Anuj, I had refined this patches. Please review this and give me your inputs. Thank you very much for your attention and feedback! Best regards, Yeoh Ee Peng -Original Message- From: Yeoh, Ee Peng Sent: Friday, August 10, 2018 4:23 PM To: openembedded-core@lists.openembedded.org Cc: Yeoh, Ee Peng Subject: [PATCH] test-result-log: testcase management tool to store test result These scripts were developed as an alternative testcase management tool to Testopia. Using these scripts, user can store test result from OEQA automated testcase execution. These scripts will store test result & log in GIT repository. To use these scripts, first source oe environment, then run the entry point script to look for help information. $ test-result-log To store test result for OEQA automated testcase, execute the below $ test-result-log store-auto / Signed-off-by: Yeoh Ee Peng --- scripts/lib/testresultlog/gitstore.py | 250 ++ scripts/lib/testresultlog/oeqalogparser.py| 97 ++ scripts/lib/testresultlog/oeqatestdiscover.py | 51 ++ scripts/lib/testresultlog/storeauto.py| 125 + scripts/test-result-log | 106 +++ 5 files changed, 629 insertions(+) create mode 100644 scripts/lib/testresultlog/gitstore.py create mode 100644 scripts/lib/testresultlog/oeqalogparser.py create mode 100644 scripts/lib/testresultlog/oeqatestdiscover.py create mode 100644 scripts/lib/testresultlog/storeauto.py create mode 100755 scripts/test-result-log diff --git a/scripts/lib/testresultlog/gitstore.py b/scripts/lib/testresultlog/gitstore.py new file mode 100644 index 000..866d7d6 --- /dev/null +++ b/scripts/lib/testresultlog/gitstore.py @@ -0,0 +1,250 @@ +# test case management tool - store test result to git repository # # +Copyright (c) 2018, Intel Corporation. +# +# This program is free software; you can redistribute it and/or modify +it # under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope it will be useful, but +WITHOUT # ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU +General Public License for # more details. +# +import tempfile +import os +import pathlib +import json +import subprocess +import shutil +import scriptpath +scriptpath.add_bitbake_lib_path() +scriptpath.add_oe_lib_path() +from oeqa.utils.git import GitRepo, GitError + +class GitStore(object): + +def __init__(self): +self.script_path = os.path.dirname(os.path.realpath(__file__)) +self.base_path = self.script_path + '/../../..' + +def _get_project_environment_directory_path(self, project_dir, test_environment_list): +project_env_dir = project_dir +for env in test_environment_list: +project_env_dir = os.path.join(project_env_dir, env) +return project_env_dir + +def _get_testmodule_list(self, testmodule_testsuite_dict): +return sorted(list(testmodule_testsuite_dict.keys())) + +def _get_testcase_list(self, testsuite_list, testsuite_testcase_dict): +testcase_list = [] +for testsuite in sorted(testsuite_list): +if testsuite in testsuite_testcase_dict: +for testcase in testsuite_testcase_dict[testsuite]: +
Re: [OE-core] [PATCH] test-result-log: testcase management tool to store test result
Hi Richard, Ross, Paul, Anuj,
As per our conversation in the past, there was a need to provide alternative
testcase management tool to Testopia in order to achieve our goals of:
1) To unblocked Bugzilla upgrade to a newer version in order to receive the
improvement in WebServices, performance, and GUI.
2) To improve overall QA testing efficiency where it reduce maintenance cost
when Testopia need to manually synchronize the test case content between
Testopia and the available automated test cases inside OEQA, also it minimize
test execution rerun where someone need to retrieve test failure log
information from previous QA cycle (Testopia does not store test log due to its
limitation on Testopia API for automation).
To provide alternative testcase management tool to Testopia, here was the first
batch of patch target to enable storing test result for OEQA automated testcase
execution and view test summary report. There are another patch planned to
enable manual test execution and store/view test result.
With inputs from Paul and Anuj, I had refined this patches.
Please review this and give me your inputs.
Thank you very much for your attention and feedback!
Best regards,
Yeoh Ee Peng
-Original Message-
From: Yeoh, Ee Peng
Sent: Friday, August 10, 2018 4:23 PM
To: openembedded-core@lists.openembedded.org
Cc: Yeoh, Ee Peng
Subject: [PATCH] test-result-log: testcase management tool to store test result
These scripts were developed as an alternative testcase management tool to
Testopia. Using these scripts, user can store test result from OEQA automated
testcase execution.
These scripts will store test result & log in GIT repository. To use these
scripts, first source oe environment, then run the entry point script to look
for help information.
$ test-result-log
To store test result for OEQA automated testcase, execute the below
$ test-result-log store-auto /
Signed-off-by: Yeoh Ee Peng
---
scripts/lib/testresultlog/gitstore.py | 250 ++
scripts/lib/testresultlog/oeqalogparser.py| 97 ++
scripts/lib/testresultlog/oeqatestdiscover.py | 51 ++
scripts/lib/testresultlog/storeauto.py| 125 +
scripts/test-result-log | 106 +++
5 files changed, 629 insertions(+)
create mode 100644 scripts/lib/testresultlog/gitstore.py
create mode 100644 scripts/lib/testresultlog/oeqalogparser.py
create mode 100644 scripts/lib/testresultlog/oeqatestdiscover.py
create mode 100644 scripts/lib/testresultlog/storeauto.py
create mode 100755 scripts/test-result-log
diff --git a/scripts/lib/testresultlog/gitstore.py
b/scripts/lib/testresultlog/gitstore.py
new file mode 100644
index 000..866d7d6
--- /dev/null
+++ b/scripts/lib/testresultlog/gitstore.py
@@ -0,0 +1,250 @@
+# test case management tool - store test result to git repository # #
+Copyright (c) 2018, Intel Corporation.
+#
+# This program is free software; you can redistribute it and/or modify
+it # under the terms and conditions of the GNU General Public License,
+# version 2, as published by the Free Software Foundation.
+#
+# This program is distributed in the hope it will be useful, but
+WITHOUT # ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+General Public License for # more details.
+#
+import tempfile
+import os
+import pathlib
+import json
+import subprocess
+import shutil
+import scriptpath
+scriptpath.add_bitbake_lib_path()
+scriptpath.add_oe_lib_path()
+from oeqa.utils.git import GitRepo, GitError
+
+class GitStore(object):
+
+def __init__(self):
+self.script_path = os.path.dirname(os.path.realpath(__file__))
+self.base_path = self.script_path + '/../../..'
+
+def _get_project_environment_directory_path(self, project_dir,
test_environment_list):
+project_env_dir = project_dir
+for env in test_environment_list:
+project_env_dir = os.path.join(project_env_dir, env)
+return project_env_dir
+
+def _get_testmodule_list(self, testmodule_testsuite_dict):
+return sorted(list(testmodule_testsuite_dict.keys()))
+
+def _get_testcase_list(self, testsuite_list, testsuite_testcase_dict):
+testcase_list = []
+for testsuite in sorted(testsuite_list):
+if testsuite in testsuite_testcase_dict:
+for testcase in testsuite_testcase_dict[testsuite]:
+testcase_list.append(testcase)
+return testcase_list
+
+def _get_testcase_status(self, testcase, testcase_status_dict):
+if testcase in testcase_status_dict:
+return testcase_status_dict[testcase]
+return ""
+
+def _create_testcase_dict(self, testcase_list, testcase_status_dict):
+testcase_dict = {}
+for testcase in sorted(testcase_list):
+testcase_status = self._get_testcase_status(testcase,
[OE-core] [PATCH] test-result-log: testcase management tool to store test result
These scripts were developed as an alternative testcase management
tool to Testopia. Using these scripts, user can store test result
from OEQA automated testcase execution.
These scripts will store test result & log in GIT repository. To use
these scripts, first source oe environment, then run the entry point
script to look for help information.
$ test-result-log
To store test result for OEQA automated testcase, execute the below
$ test-result-log store-auto /
Signed-off-by: Yeoh Ee Peng
---
scripts/lib/testresultlog/gitstore.py | 250 ++
scripts/lib/testresultlog/oeqalogparser.py| 97 ++
scripts/lib/testresultlog/oeqatestdiscover.py | 51 ++
scripts/lib/testresultlog/storeauto.py| 125 +
scripts/test-result-log | 106 +++
5 files changed, 629 insertions(+)
create mode 100644 scripts/lib/testresultlog/gitstore.py
create mode 100644 scripts/lib/testresultlog/oeqalogparser.py
create mode 100644 scripts/lib/testresultlog/oeqatestdiscover.py
create mode 100644 scripts/lib/testresultlog/storeauto.py
create mode 100755 scripts/test-result-log
diff --git a/scripts/lib/testresultlog/gitstore.py
b/scripts/lib/testresultlog/gitstore.py
new file mode 100644
index 000..866d7d6
--- /dev/null
+++ b/scripts/lib/testresultlog/gitstore.py
@@ -0,0 +1,250 @@
+# test case management tool - store test result to git repository
+#
+# Copyright (c) 2018, Intel Corporation.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms and conditions of the GNU General Public License,
+# version 2, as published by the Free Software Foundation.
+#
+# This program is distributed in the hope it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+# more details.
+#
+import tempfile
+import os
+import pathlib
+import json
+import subprocess
+import shutil
+import scriptpath
+scriptpath.add_bitbake_lib_path()
+scriptpath.add_oe_lib_path()
+from oeqa.utils.git import GitRepo, GitError
+
+class GitStore(object):
+
+def __init__(self):
+self.script_path = os.path.dirname(os.path.realpath(__file__))
+self.base_path = self.script_path + '/../../..'
+
+def _get_project_environment_directory_path(self, project_dir,
test_environment_list):
+project_env_dir = project_dir
+for env in test_environment_list:
+project_env_dir = os.path.join(project_env_dir, env)
+return project_env_dir
+
+def _get_testmodule_list(self, testmodule_testsuite_dict):
+return sorted(list(testmodule_testsuite_dict.keys()))
+
+def _get_testcase_list(self, testsuite_list, testsuite_testcase_dict):
+testcase_list = []
+for testsuite in sorted(testsuite_list):
+if testsuite in testsuite_testcase_dict:
+for testcase in testsuite_testcase_dict[testsuite]:
+testcase_list.append(testcase)
+return testcase_list
+
+def _get_testcase_status(self, testcase, testcase_status_dict):
+if testcase in testcase_status_dict:
+return testcase_status_dict[testcase]
+return ""
+
+def _create_testcase_dict(self, testcase_list, testcase_status_dict):
+testcase_dict = {}
+for testcase in sorted(testcase_list):
+testcase_status = self._get_testcase_status(testcase,
testcase_status_dict)
+testcase_dict[testcase] = {"testresult": testcase_status,"bugs":
""}
+return testcase_dict
+
+def _create_testsuite_testcase_teststatus_json_object(self,
testsuite_list, testsuite_testcase_dict, testcase_status_dict):
+json_object = {'testsuite':{}}
+testsuite_dict = json_object['testsuite']
+for testsuite in sorted(testsuite_list):
+testsuite_dict[testsuite] = {'testcase': {}}
+testsuite_dict[testsuite]['testcase'] =
self._create_testcase_dict(testsuite_testcase_dict[testsuite],
testcase_status_dict)
+return json_object
+
+def _create_testsuite_json_formatted_string(self, testsuite_list,
testsuite_testcase_dict, testcase_status_dict):
+testsuite_testcase_list =
self._create_testsuite_testcase_teststatus_json_object(testsuite_list,
testsuite_testcase_dict, testcase_status_dict)
+return json.dumps(testsuite_testcase_list, sort_keys=True, indent=4)
+
+def _write_testsuite_testcase_json_formatted_string_to_file(self,
file_path, file_content):
+with open(file_path, 'w') as the_file:
+the_file.write(file_content)
+
+def _write_log_file(self, file_path, logs):
+with open(file_path, 'w') as the_file:
+for line in logs:
+the_file.write(line + '\n')
+
+def _write_test_log_files_for_list_of_testcase(self, file_dir,
testcase_list, testcase_logs_dict):
