Change in osmo-gsm-tester[master]: Implement per-test timeout guard
pespin has submitted this change. (
https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 )
Change subject: Implement per-test timeout guard
..
Implement per-test timeout guard
Timeout value can be specified by test in suite.conf:
config:
suite:
:
:
timeout: 2 # 2 seconds timeout
Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706
---
M doc/manuals/chapters/config.adoc
M selftest/suite_test/suite_test.ok
M selftest/suite_test/suite_test.py
M selftest/suite_test/suitedirA/test_suite/suite.conf
A selftest/suite_test/suitedirA/test_suite/test_timeout.py
M src/osmo_gsm_tester/core/suite.py
M src/osmo_gsm_tester/core/test.py
M src/osmo_gsm_tester/testenv.py
8 files changed, 78 insertions(+), 20 deletions(-)
Approvals:
pespin: Looks good to me, approved; Verified
diff --git a/doc/manuals/chapters/config.adoc b/doc/manuals/chapters/config.adoc
index 118f056..f4e08b3 100644
--- a/doc/manuals/chapters/config.adoc
+++ b/doc/manuals/chapters/config.adoc
@@ -190,7 +190,18 @@
a_suite_test_foo:
one_test_parameter_for_test_foo: 'str'
another_test_parameter_for_test_foo: ['bool_str']
+
+config:
+ suite:
+:
+ some_suite_parameter: 3
+ a_suite_test_foo:
+one_test_parameter_for_test_foo: 'hello'
+timeout: 30 <1>
+<1> The per-test _timeout_ attribute is implicitly defined for all tests with
+type _duration_, and will trigger a timeout if test doesn't finish in time
+specified.
[[scenarios_dir]]
'scenarios_dir'
diff --git a/selftest/suite_test/suite_test.ok
b/selftest/suite_test/suite_test.ok
index c7c76d7..58593fd 100644
--- a/selftest/suite_test/suite_test.ok
+++ b/selftest/suite_test/suite_test.ok
@@ -15,6 +15,11 @@
cnf [PATH]/selftest/suite_test/suitedirA/empty_dir/suite.conf: ERR:
FileNotFoundError: [Errno 2] No such file or directory:
'[PATH]/selftest/suite_test/suitedirA/empty_dir/suite.conf'
[empty_dir↪[PATH]/selftest/suite_test/suitedirA/empty_dir/suite.conf]
- valid suite dir
cnf test_suite: DBG: reading suite.conf
+config:
+ suite:
+test_suite:
+ test_timeout:
+timeout: '1'
resources:
bts:
- label: sysmoCell 5000
@@ -28,7 +33,7 @@
- run hello world test
tst test_suite: DBG: {combining='config'}
-tst {combining_scenarios='config'}: DBG: {definition_conf={}}
[test_suite↪{combining_scenarios='config'}]
+tst {combining_scenarios='config'}: DBG:
{definition_conf={suite={test_suite={test_timeout={timeout='1'}
[test_suite↪{combining_scenarios='config'}]
-
trial test_suite
@@ -101,7 +106,7 @@
-
trial test_suite PASS
-
-PASS: test_suite (pass: 1, skip: 6)
+PASS: test_suite (pass: 1, skip: 7)
pass: hello_world.py (N.N sec)
skip: mo_mt_sms.py
skip: mo_sms.py
@@ -109,6 +114,7 @@
skip: test_fail.py
skip: test_fail_raise.py
skip: test_suite_params.py
+skip: test_timeout.py
- a test with an error
@@ -125,7 +131,7 @@
-
trial test_suite FAIL
-
-FAIL: test_suite (fail: 1, skip: 6)
+FAIL: test_suite (fail: 1, skip: 7)
skip: hello_world.py (N.N sec)
skip: mo_mt_sms.py
skip: mo_sms.py
@@ -133,6 +139,7 @@
skip: test_fail.py
skip: test_fail_raise.py
skip: test_suite_params.py
+skip: test_timeout.py
- a test with a failure
@@ -149,7 +156,7 @@
-
trial test_suite FAIL
-
-FAIL: test_suite (fail: 1, skip: 6)
+FAIL: test_suite (fail: 1, skip: 7)
skip: hello_world.py (N.N sec)
skip: mo_mt_sms.py
skip: mo_sms.py
@@ -157,6 +164,7 @@
FAIL: test_fail.py (N.N sec) EpicFail: This failure is expected
skip: test_fail_raise.py
skip: test_suite_params.py
+skip: test_timeout.py
- a test with a raised failure
@@ -172,7 +180,7 @@
-
trial test_suite FAIL
-
-FAIL: test_suite (fail: 1, skip: 6)
+FAIL: test_suite (fail: 1, skip: 7)
skip: hello_world.py (N.N sec)
skip: mo_mt_sms.py
skip: mo_sms.py
@@ -180,9 +188,10 @@
skip: test_fail.py (N.N sec)
FAIL: test_fail_raise.py (N.N sec) ExpectedFail: This failure is expected
skip: test_suite_params.py
+skip: test_timeout.py
- test with half empty scenario
tst test_suite: DBG: {combining='config'} [suite.py:[LINENR]]
-tst {combining_scenarios='config'}: DBG: {definition_conf={}}
[test_suite↪{combining_scenarios='config'}] [suite.py:[LINENR]]
+tst {combining_
Change in osmo-gsm-tester[master]: Implement per-test timeout guard
Hello neels, laforge, Jenkins Builder, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 to look at the new patch set (#5). Change subject: Implement per-test timeout guard .. Implement per-test timeout guard Timeout value can be specified by test in suite.conf: config: suite: : : timeout: 2 # 2 seconds timeout Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706 --- M doc/manuals/chapters/config.adoc M selftest/suite_test/suite_test.ok M selftest/suite_test/suite_test.py M selftest/suite_test/suitedirA/test_suite/suite.conf A selftest/suite_test/suitedirA/test_suite/test_timeout.py M src/osmo_gsm_tester/core/suite.py M src/osmo_gsm_tester/core/test.py M src/osmo_gsm_tester/testenv.py 8 files changed, 78 insertions(+), 20 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/99/18799/5 -- To view, visit https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-gsm-tester Gerrit-Branch: master Gerrit-Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706 Gerrit-Change-Number: 18799 Gerrit-PatchSet: 5 Gerrit-Owner: pespin Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge Gerrit-Reviewer: neels Gerrit-MessageType: newpatchset
Change in osmo-gsm-tester[master]: Implement per-test timeout guard
pespin has posted comments on this change. ( https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 ) Change subject: Implement per-test timeout guard .. Patch Set 5: Verified+1 Code-Review+2 -- To view, visit https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-gsm-tester Gerrit-Branch: master Gerrit-Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706 Gerrit-Change-Number: 18799 Gerrit-PatchSet: 5 Gerrit-Owner: pespin Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge Gerrit-Reviewer: neels Gerrit-Reviewer: pespin Gerrit-Comment-Date: Mon, 15 Jun 2020 08:54:55 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in osmo-gsm-tester[master]: Implement per-test timeout guard
neels has posted comments on this change. (
https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 )
Change subject: Implement per-test timeout guard
..
Patch Set 4: Code-Review+1
(2 comments)
https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799/4/src/osmo_gsm_tester/core/suite.py
File src/osmo_gsm_tester/core/suite.py:
https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799/4/src/osmo_gsm_tester/core/suite.py@61
PS4, Line 61: #self.log('dictionary is: %r' % repr(d))
(you want to keep this comment?)
https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799/4/src/osmo_gsm_tester/testenv.py
File src/osmo_gsm_tester/testenv.py:
https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799/4/src/osmo_gsm_tester/testenv.py@130
PS4, Line 130:
(two blank lines)
--
To view, visit https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799
To unsubscribe, or for help writing mail filters, visit
https://gerrit.osmocom.org/settings
Gerrit-Project: osmo-gsm-tester
Gerrit-Branch: master
Gerrit-Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706
Gerrit-Change-Number: 18799
Gerrit-PatchSet: 4
Gerrit-Owner: pespin
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: laforge
Gerrit-Reviewer: neels
Gerrit-Comment-Date: Sun, 14 Jun 2020 11:31:33 +
Gerrit-HasComments: Yes
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment
Change in osmo-gsm-tester[master]: Implement per-test timeout guard
laforge has posted comments on this change. ( https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 ) Change subject: Implement per-test timeout guard .. Patch Set 4: Code-Review+1 -- To view, visit https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-gsm-tester Gerrit-Branch: master Gerrit-Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706 Gerrit-Change-Number: 18799 Gerrit-PatchSet: 4 Gerrit-Owner: pespin Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge Gerrit-Comment-Date: Sat, 13 Jun 2020 11:49:33 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in osmo-gsm-tester[master]: Implement per-test timeout guard
Hello Jenkins Builder, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 to look at the new patch set (#4). Change subject: Implement per-test timeout guard .. Implement per-test timeout guard Timeout value can be specified by test in suite.conf: config: suite: : : timeout: 2 # 2 seconds timeout Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706 --- M doc/manuals/chapters/config.adoc M selftest/suite_test/suite_test.ok M selftest/suite_test/suite_test.py M selftest/suite_test/suitedirA/test_suite/suite.conf A selftest/suite_test/suitedirA/test_suite/test_timeout.py M src/osmo_gsm_tester/core/suite.py M src/osmo_gsm_tester/core/test.py M src/osmo_gsm_tester/testenv.py 8 files changed, 80 insertions(+), 20 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/99/18799/4 -- To view, visit https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-gsm-tester Gerrit-Branch: master Gerrit-Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706 Gerrit-Change-Number: 18799 Gerrit-PatchSet: 4 Gerrit-Owner: pespin Gerrit-Reviewer: Jenkins Builder Gerrit-MessageType: newpatchset
Change in osmo-gsm-tester[master]: Implement per-test timeout guard
Hello Jenkins Builder, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 to look at the new patch set (#3). Change subject: Implement per-test timeout guard .. Implement per-test timeout guard Timeout value can be specified by test in suite.conf: config: suite: : : timeout: 2 # 2 seconds timeout Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706 --- M doc/manuals/chapters/config.adoc M selftest/suite_test/suite_test.ok M selftest/suite_test/suite_test.py M selftest/suite_test/suitedirA/test_suite/suite.conf A selftest/suite_test/suitedirA/test_suite/test_timeout.py M src/osmo_gsm_tester/core/suite.py M src/osmo_gsm_tester/core/test.py M src/osmo_gsm_tester/testenv.py 8 files changed, 80 insertions(+), 20 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/99/18799/3 -- To view, visit https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-gsm-tester Gerrit-Branch: master Gerrit-Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706 Gerrit-Change-Number: 18799 Gerrit-PatchSet: 3 Gerrit-Owner: pespin Gerrit-Reviewer: Jenkins Builder Gerrit-MessageType: newpatchset
Change in osmo-gsm-tester[master]: Implement per-test timeout guard
pespin has uploaded a new patch set (#2). ( https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 ) Change subject: Implement per-test timeout guard .. Implement per-test timeout guard Timeout value can be specified by test in suite.conf: config: suite: : : timeout: 2 # 2 seconds timeout Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706 --- M selftest/suite_test/suite_test.ok M selftest/suite_test/suite_test.py M selftest/suite_test/suitedirA/test_suite/suite.conf A selftest/suite_test/suitedirA/test_suite/test_timeout.py M src/osmo_gsm_tester/core/suite.py M src/osmo_gsm_tester/core/test.py M src/osmo_gsm_tester/testenv.py 7 files changed, 66 insertions(+), 20 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/99/18799/2 -- To view, visit https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-gsm-tester Gerrit-Branch: master Gerrit-Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706 Gerrit-Change-Number: 18799 Gerrit-PatchSet: 2 Gerrit-Owner: pespin Gerrit-MessageType: newpatchset
Change in osmo-gsm-tester[master]: Implement per-test timeout guard
pespin has uploaded this change for review. (
https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18799 )
Change subject: Implement per-test timeout guard
..
Implement per-test timeout guard
Timeout value can be specified by test in suite.conf:
config:
suite:
:
:
timeout: 2 # 2 seconds timeout
Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706
---
M selftest/suite_test/suite_test.py
M selftest/suite_test/suitedirA/test_suite/suite.conf
A selftest/suite_test/suitedirA/test_suite/test_timeout.py
M src/osmo_gsm_tester/core/suite.py
M src/osmo_gsm_tester/core/test.py
M src/osmo_gsm_tester/testenv.py
6 files changed, 30 insertions(+), 5 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester
refs/changes/99/18799/1
diff --git a/selftest/suite_test/suite_test.py
b/selftest/suite_test/suite_test.py
index 260b9c4..9708037 100755
--- a/selftest/suite_test/suite_test.py
+++ b/selftest/suite_test/suite_test.py
@@ -102,7 +102,7 @@
s = suite.SuiteRun(trial, 'test_suite', s_def, [sc])
s.reserve_resources()
print(repr(s.reserved_resources))
-results = s.run_tests('test_suite_params.py')
+results = s.run_tests(['test_suite_params.py', 'test_timeout.py'])
print(report.suite_to_text(s))
print('- test with template overlay')
diff --git a/selftest/suite_test/suitedirA/test_suite/suite.conf
b/selftest/suite_test/suitedirA/test_suite/suite.conf
index ff4899a..0426ea7 100644
--- a/selftest/suite_test/suitedirA/test_suite/suite.conf
+++ b/selftest/suite_test/suitedirA/test_suite/suite.conf
@@ -15,3 +15,9 @@
one_bool_parameter: 'bool_str'
second_list_parameter: ['uint']
+
+config:
+ suite:
+test_suite:
+ test_timeout:
+timeout: 1 # timeout in 1 second
diff --git a/selftest/suite_test/suitedirA/test_suite/test_timeout.py
b/selftest/suite_test/suitedirA/test_suite/test_timeout.py
new file mode 100644
index 000..eeddb70
--- /dev/null
+++ b/selftest/suite_test/suitedirA/test_suite/test_timeout.py
@@ -0,0 +1,6 @@
+from osmo_gsm_tester.testenv import *
+
+timeout = int(tenv.config_test_specific()['timeout'])
+print('starting test and waiting to receive Timeout after %d seconds' %
timeout)
+sleep(10)
+print('test failed, we expected timeout after %d seconds' % timeout)
diff --git a/src/osmo_gsm_tester/core/suite.py
b/src/osmo_gsm_tester/core/suite.py
index 9b9062d..aa09cd6 100644
--- a/src/osmo_gsm_tester/core/suite.py
+++ b/src/osmo_gsm_tester/core/suite.py
@@ -44,6 +44,8 @@
self.suite_dir = suite_dir
self.conf = None
self._schema = None
+self.test_basenames = []
+self.load_test_basenames()
self.read_conf()
def read_conf(self):
@@ -54,13 +56,17 @@
SuiteDefinition.CONF_FILENAME))
# Drop schema part since it's dynamically defining content, makes no
sense to validate it.
self._schema = self.conf.pop('schema', {})
+# Add per-test 'timeout' attribute:
+d = {t.rstrip('.py'):{'timeout': schema.DURATION} for t in
self.test_basenames}
+#self.log('dictionary is: %r' % repr(d))
+schema.combine(self._schema, d)
+# Convert config file format to proper schema format and register it:
sdef = schema.config_to_schema_def(self._schema, "%s." %
self._suite_name)
schema.register_config_schema('suite', sdef)
+# Finally validate the file:
schema.validate(self.conf, schema.get_all_schema())
-self.load_test_basenames()
def load_test_basenames(self):
-self.test_basenames = []
for basename in sorted(os.listdir(self.suite_dir)):
if not basename.endswith('.py'):
continue
diff --git a/src/osmo_gsm_tester/core/test.py b/src/osmo_gsm_tester/core/test.py
index 45dfd41..2fa4fe9 100644
--- a/src/osmo_gsm_tester/core/test.py
+++ b/src/osmo_gsm_tester/core/test.py
@@ -35,12 +35,12 @@
PASS = 'pass'
FAIL = 'FAIL'
-def __init__(self, suite_run, test_basename, test_specific_config):
+def __init__(self, suite_run, test_basename, config_test_specific):
self.basename = test_basename
super().__init__(log.C_TST, self.basename)
self._run_dir = None
self.suite_run = suite_run
-self._config_test_specific = test_specific_config
+self._config_test_specific = config_test_specific
self.path = os.path.join(self.suite_run.definition.suite_dir,
self.basename)
self.status = Test.UNKNOWN
self.start_timestamp = 0
@@ -49,6 +49,7 @@
self.fail_message = None
self.log_targets = []
self._report_stdout = None
+self.timeout = int(config_test_specific['timeout']) if 'timeout' in
config_test_specific else None
def module_name(self):
'Return test name without trailing .py'
@@ -110,6 +111,10 @@
'time elapsed since test was started'
