Patch Set 1: (2 comments)
https://gerrit.osmocom.org/#/c/2669/1/src/osmo-gsm-tester.py File src/osmo-gsm-tester.py: Line 198: trials_run.append((current_trial.name(), report)) > I don't see a good use case to be able to run it for several trials tbh. I' let's remove multi-trial invocations then. you can do this now (best in separate patch) or later. https://gerrit.osmocom.org/#/c/2669/1/suites/debug/error.py File suites/debug/error.py: Line 5: assert False > I think the intention is the opposite: Failures are typified issues or test I accept your argument, but. All in all, I don't think we even need to have this way of two distinct failure classes. Classifying unsuccessful tests in two categories won't help us with anything. We still need to look at the precise failure every time. So actually I would even go as far as reporting all of them as Failure and igoring the Error category completely. If we keep Error: it is hard to distinguish which is which. If I am trying to get the list of subscribers from the CTRL interface, and the NITB's CTRL impl has been broken by a regression, then I will get some arbitrary exception but still it is semantically a test failure, and not an error in the testing framework. We know (assume) that our python code and dependencies work; otherwise we can sanitize our python code to not "pollute" by raising arbitrary exceptions. >From that standpoint, I would like test errors to be things that prevented us >from starting the test (e.g. could not obtain a resource), and have everything >else end up as a test failure. Test API wise, I would prefer if we can continue to write things like 'assert nitb.connected()' instead of complicating those with raising exceptions and composing messages every time (that should be optional IMHO). If we can't agree on which is which, then let's drop the Error class completely. -- To view, visit https://gerrit.osmocom.org/2669 To unsubscribe, visit https://gerrit.osmocom.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iedf6d912b3cce3333a187a4ac6d5c6b70fe9d5c5 Gerrit-PatchSet: 1 Gerrit-Project: osmo-gsm-tester Gerrit-Branch: master Gerrit-Owner: Pau Espin Pedrol <pes...@sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: Neels Hofmeyr <nhofm...@sysmocom.de> Gerrit-Reviewer: Pau Espin Pedrol <pes...@sysmocom.de> Gerrit-Reviewer: neels <nhofm...@sysmocom.de> Gerrit-HasComments: Yes
