Dne 5.4.2016 v 21:28 Cleber Rosa napsal(a):
On 04/05/2016 03:23 PM, Jeff Nelson wrote:
On Fri, Apr 01, 2016 at 11:00:39AM -0300, Cleber Rosa wrote:
MOTIVATION
==========
The idea of adding hooks to be run by Avocado before and after tests
is general enough, and may be used by the community in unpredictable
ways. And that is good.
For this team, the initial motivation was to be able to bring back an
Autotest feature that some of our users are missing: the ability to
set the system-wide "kernel core pattern" configuration for tests.
Having a pre-test hook would allow "/proc/sys/kernel/core_pattern" to
be read, saved and modified to point to the test results directory.
Having a post-test hook would allow "/proc/sys/kernel/core_pattern" to
be reverted back to its original state.
Other currently core features such as sysinfo collection, could be
re-implemented as pre/post test hooks.
GENERAL DESIGN POINTS
=====================
These are the most important design decisions to be acknowledged or
questioned. Please reply with either ACK or your questions/suggestions.
I have some questions (hope you don't mind).
What are the outputs of pre- and post-test hooks?
The outputs are defined by the actual plugin "hooked in the hook". The
current interface doesn't define it. They can be completely silent, they
can generate output to the UI, they can write to files at the test's
result directory, etc.
Are there limits to the actions that are permitted in pre- and
post-test hooks? Of course, the primary use-case of the pre-test hook
is to set up the environment for the test--so environment changes are
permitted--and the use-case for a matching post-hook is to restore the
environment. About the only operation I can imagine NOT being
permitted is to abort (kill itself, or kill its controlling parent
process).
The pre/post test hooks are fed the test status, so they can gather
information, but not (directly) influence its outcome.
Can a pre-test hook return a status that causes the test execution to
be skipped? I can imagine this being done for another use-case:
validate the test environment (e.g., check to see if required hardware
is present).
No, this should be done at the test's "setUp" stage. Pointers:
https://github.com/avocado-framework/avocado/blob/master/avocado/core/test.py#L368
But I understand the value in influencing all tests with a single,
plugable, block of code. Still, this looks like something that
could/should be addressed by what we're calling "Job API". With some
Python code and such an API, you could select the exact tests you want
to run.
Can a post-test hook alter the result (status) of the test?
In the current implementation draft it could. But just because we're
sharing the same (last, final) test state among the post plugins and
test result handlers. I'm not sure that a plugin *should* change it,
though.
I can imagine a BZ plugin, which marks the results as `SKIP` if assigned
BZ exists, so maybe it can be useful.
Has there been any thought given to having pre- and post-job hooks?
For example, setting the kernel core pattern is something I would want
to do globally, for all tests in a job. It would be faster and more
convenient to do this just once. But I admit this is a +1 optimization
and need not be considered now. (It also complicates things when tests
run on multiple machines.)
I'm not sure you want to set the kernel core pattern per job. I'd much
rather see the coredumps inside each test result. (still there is the
problem with concurrent jobs...)
Pre/Post job hooks are actually simpler, and implemented in the
following PR:
https://github.com/avocado-framework/avocado/pull/1106
Can there be multiple hooks for a given test? If so, how does one
define the order in which they are executed? Since there are pre- and
post-test hooks, there are really two orders to consider.
You mean multiple plugins registered under a given hook, or simply put,
multiple pre (and/or) post plugins active, right? The answer is yes.
We have not implemented any kind of ordering management, simply because
we have not hit a use case that requires it.
I found myself wanting to make an assumption so I better ask: must
hooks come in pairs (for every pre-test there must be a post-test, and
vice-versa)?
The alternative proposal by me handled this differently. It'd allow you
to create a `job` plugin, which would be notified of certain situations
like `pre` and `post` test. So in this version yes, all plugins would
contain pre and post section, even dummy one.
The benefit would be that if we add `pre_job` and `post_job`, you'd
control the whole flow inside one plugin (the plugin would be
initialized once, then it'd be called when the hooks are reached)
The benefit of Cleber's proposal is that each plugin is separated and
can be registered as pre/post or in the future pre_job, post_job (if we
decide like adding it). (separate instance of the plugin would be
initialized for each entry-point - pre, post - and called when the hook
is reached).
I have one more questions for you, Cleber. How do you plan to manage
those plugins? Because right now (if I'm not wrong) the only way is to
install or uninstall the plugin into certain namespace.
Lukáš
Not at all. You can have only pre hooks enabled, only post hooks enabled
or both.
The given *job* mail notification plugin example, for instance, is
something that makes more sense activated only in a post hook:
https://github.com/clebergnu/avocado/blob/job_pre_post/examples/plugins/job-pre-post/mail/avocado_job_mail.py
That's all for now.
-Jeff
Thanks a lot for the feedback!
- Cleber.
_______________________________________________
Avocado-devel mailing list
Avocado-devel@redhat.com
https://www.redhat.com/mailman/listinfo/avocado-devel
_______________________________________________
Avocado-devel mailing list
Avocado-devel@redhat.com
https://www.redhat.com/mailman/listinfo/avocado-devel
