Re: [Avocado-devel] [RFC] Environment Variables - V2
Dne 15.6.2016 v 11:20 Lukáš Doktor napsal(a): Dne 10.6.2016 v 11:20 Amador Pahim napsal(a): Hi After some discussion about the V1, this is the V2 of the RFC on Environment Variables. The original request is here: https://trello.com/c/Ddcly0oG/312-mechanism-to-provide-environment-variables-to-tests-run-on-a-virtual-machine-remote Motivation == When Avocado is executed locally, it has access to the local environment variables, including that ones set by the user. For remote executions, like using remote or vm plugins, the local environment variables are not reflected on the remote environment. The motivation of this RFC is to provide the standard Unix interface of environment variables for when tests are run remotely. Use case - User defines an environment variable locally: $ export FOOVAR=myfoovar - User runs avocado using a remote plugin, pointing the local environment variable(s) he wants to keep the in the remote environment: $ avocado --env-keep FOOVAR run passtest.py --remote-hostname host01 - The environment variable defined locally will be present on host01 environment. Alternative flow: - Instead of pointing in the command line the local environment variable(s) he wants to keep the in the remote environment, uses can set it in avocado.conf so it will affect every Avocado execution. Proposal - To create an Avocado command line option to set environment variables that will be available in remote environment: $ avocado --env-keep FOOVAR run passtest.py --remote-hostname host01 - Create an option in config file with a list of environment variable names to copy from Avocado local environment to the remote environment (similar to env_keep in the /etc/sudoers file): [remoter.behavior] env_keep = ['FOO', 'FOO1', 'FOO2'] I don't think the config file would be necessary, but apart from that I agree with this. Maybe in the future we might need `--env-unset` as some ugly software checks for value presence even when the value is null. Actually not, I noticed you're talking about generic `avocado` param. I think this should be `avocado run` only option, therefor the usage would be: $avocado run passtest.py --env-keep FOOVAR --remote-hostname host01 Another reason for using this is that `nargs='+'` would consume the `run` argument. So if we are to support `avocado --env-keep` than users would have to use `avocado --env-keep foo --env-keep bar run test` or we'd have to split the values ourselves (eg. using `,`). Kind regards, Lukáš Lukáš Best Regards, ___ Avocado-devel mailing list Avocado-devel@redhat.com https://www.redhat.com/mailman/listinfo/avocado-devel signature.asc Description: OpenPGP digital signature ___ Avocado-devel mailing list Avocado-devel@redhat.com https://www.redhat.com/mailman/listinfo/avocado-devel
Re: [Avocado-devel] [RFC] Environment Variables
On 06/01/2016 09:39 PM, Ademar Reis wrote: On Wed, Jun 01, 2016 at 04:02:54PM -0300, Cleber Rosa wrote: On 06/01/2016 03:07 PM, Ademar Reis wrote: On Tue, May 31, 2016 at 07:30:43AM -0300, Cleber Rosa wrote: I'm replying on top of Cleber because he already said a few things I was going to say. On 05/25/2016 05:31 AM, Amador Pahim wrote: Hi folks, We have requests to handle the environment variables that we can set to the tests. This is the RFC in that regard, with a summary of the ideas already exposed in the original request and some additional planning. The original request is here: https://trello.com/c/Ddcly0oG/312-mechanism-to-provide-environment-variables-to-tests-run-on-a-virtual-machine-remote Motivation == Avocado tests are executed in a fork process or even in a remote machine. Regardless the fact that Avocado is hard coded to set some environment variables, they are for internal consumption and user is not allowed to control/configure its behavior. You mean this: http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#environment-variables-for-simple-tests Right? Basically, the fact that Avocado sets some of the job/test state as environment variables, that can be used by SIMPLE tests. The motivation is the request to provide users an interface to set and/or keep environment variables for test consumption. I'm not sure if they're necessarily for test consumption. I think the motivation for the original request was to provide the standard Unix interface of environment variables for when tests are run remotely. If the motivation is basically about setting the env vars when running tests remotely, than this brings the discussion about the *local* behavior to: 1. Should Avocado default to the standard UNIX behavior of cloning the environment? A: IMHO, yes. That's the current behavior (see my example at the end of the previous email). Except when one runs tests remotely, which is precisely the use case this feature would "fix". Yes, agreed we should only extend the current behavior in local tests to remote tests. That seems to be the best approach for this RFC. 2. Could Avocado have have a feature to start tests in a clean(er) environment? A: Possibly yes, but seems low priority. The use case here could be seen as a plus in predictability, helping to achieve expected test results in spite of the runner environment. A real world example could be a CI environment that sets a VERBOSE environment variable. This env var will be passed over to Avocado, to the test process and finally to a custom binary (say a benchmark tool) that will produce different output depending on that environment variable. Doing that type of cleaning in the test code is possible, but the framework could help with that. 2.1. If Avocado provides a "clean(er) test environment" feature, how to determine which environment variables are passed along? A: The "env-keep" approach seems like the obvious way to do it. If the mechanism is enabled, which I believe should be disabled by default (see #1), its default list could contain the more or less standard UNIX environment variables (TERM, SHELL, LANG, etc). Agree. But like you said such a feature would be low priority and optional. The important thing is that the implementation of what we're discussing in this RFC would not interfere with it. "clean(er) test environment" would affect both local and remote implementations and should be considered regardless this RFC. Still, I don't see relevance in have a cleaner env right now. Agreed it's low priority or even unwanted feature. These environment variables can change the behavior of both Avocado (the runner itself), the tests (after all nothing prevents the test writer from using them) and all sub-processes executed by the test. Right. Locally, this is standard: $ TMPDIR=/whatever/tmp VAR=foo ./avocado run test1.py But when running avocado remotely, there's no way to configure the environment in the destination. The environment variables set in the command line below will not be "forwarded" to the remote environment: $ TMPDIR=/whatever/tmp VAR=foo ./avocado run test1.py \ --remote... Right. Use cases = 1) Use the command line or the config file to set the environment variables in tests processes environment; access those variables from inside the test. 2) Copy from current environment some environment variable(s) to the tests processes environment; access those variables from inside the test. I think we don't even have to go that far. We can simply say the intention is to set the environment variables in the environment where Avocado is run. The mechanism is quite standard and well understood. And here comes an important point: I don't think this should be a mechanism to pass variables to tests. Although, again, environment variables can be used for that purpose, Avocado should have a proper
Re: [Avocado-devel] [RFC] Environment Variables
On 06/01/2016 04:39 PM, Ademar Reis wrote: On Wed, Jun 01, 2016 at 04:02:54PM -0300, Cleber Rosa wrote: On 06/01/2016 03:07 PM, Ademar Reis wrote: On Tue, May 31, 2016 at 07:30:43AM -0300, Cleber Rosa wrote: I'm replying on top of Cleber because he already said a few things I was going to say. On 05/25/2016 05:31 AM, Amador Pahim wrote: Hi folks, We have requests to handle the environment variables that we can set to the tests. This is the RFC in that regard, with a summary of the ideas already exposed in the original request and some additional planning. The original request is here: https://trello.com/c/Ddcly0oG/312-mechanism-to-provide-environment-variables-to-tests-run-on-a-virtual-machine-remote Motivation == Avocado tests are executed in a fork process or even in a remote machine. Regardless the fact that Avocado is hard coded to set some environment variables, they are for internal consumption and user is not allowed to control/configure its behavior. You mean this: http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#environment-variables-for-simple-tests Right? Basically, the fact that Avocado sets some of the job/test state as environment variables, that can be used by SIMPLE tests. The motivation is the request to provide users an interface to set and/or keep environment variables for test consumption. I'm not sure if they're necessarily for test consumption. I think the motivation for the original request was to provide the standard Unix interface of environment variables for when tests are run remotely. If the motivation is basically about setting the env vars when running tests remotely, than this brings the discussion about the *local* behavior to: 1. Should Avocado default to the standard UNIX behavior of cloning the environment? A: IMHO, yes. That's the current behavior (see my example at the end of the previous email). Except when one runs tests remotely, which is precisely the use case this feature would "fix". 2. Could Avocado have have a feature to start tests in a clean(er) environment? A: Possibly yes, but seems low priority. The use case here could be seen as a plus in predictability, helping to achieve expected test results in spite of the runner environment. A real world example could be a CI environment that sets a VERBOSE environment variable. This env var will be passed over to Avocado, to the test process and finally to a custom binary (say a benchmark tool) that will produce different output depending on that environment variable. Doing that type of cleaning in the test code is possible, but the framework could help with that. 2.1. If Avocado provides a "clean(er) test environment" feature, how to determine which environment variables are passed along? A: The "env-keep" approach seems like the obvious way to do it. If the mechanism is enabled, which I believe should be disabled by default (see #1), its default list could contain the more or less standard UNIX environment variables (TERM, SHELL, LANG, etc). Agree. But like you said such a feature would be low priority and optional. The important thing is that the implementation of what we're discussing in this RFC would not interfere with it. These environment variables can change the behavior of both Avocado (the runner itself), the tests (after all nothing prevents the test writer from using them) and all sub-processes executed by the test. Right. Locally, this is standard: $ TMPDIR=/whatever/tmp VAR=foo ./avocado run test1.py But when running avocado remotely, there's no way to configure the environment in the destination. The environment variables set in the command line below will not be "forwarded" to the remote environment: $ TMPDIR=/whatever/tmp VAR=foo ./avocado run test1.py \ --remote... Right. Use cases = 1) Use the command line or the config file to set the environment variables in tests processes environment; access those variables from inside the test. 2) Copy from current environment some environment variable(s) to the tests processes environment; access those variables from inside the test. I think we don't even have to go that far. We can simply say the intention is to set the environment variables in the environment where Avocado is run. The mechanism is quite standard and well understood. And here comes an important point: I don't think this should be a mechanism to pass variables to tests. Although, again, environment variables can be used for that purpose, Avocado should have a proper interface to provide a dictionary of configuration and variables to each test. The only valid reason for having such a mechanism to pass *different* environment variables to tests, talking about local environment, would be *if and only if* the same environment variable to be set when running Avocado would change the behavior of Avocado itself. Example: $ AVOCADO_LOG_EARLY=1 avocado run
Re: [Avocado-devel] [RFC] Environment Variables
On Wed, Jun 01, 2016 at 04:02:54PM -0300, Cleber Rosa wrote: > On 06/01/2016 03:07 PM, Ademar Reis wrote: > > On Tue, May 31, 2016 at 07:30:43AM -0300, Cleber Rosa wrote: > > > > > > > I'm replying on top of Cleber because he already said a few > > things I was going to say. > > > > > On 05/25/2016 05:31 AM, Amador Pahim wrote: > > > > Hi folks, > > > > > > > > We have requests to handle the environment variables that we can set to > > > > the tests. This is the RFC in that regard, with a summary of the ideas > > > > already exposed in the original request and some additional planning. > > > > > > > > The original request is here: > > > > https://trello.com/c/Ddcly0oG/312-mechanism-to-provide-environment-variables-to-tests-run-on-a-virtual-machine-remote > > > > > > > > > > > > Motivation > > > > == > > > > Avocado tests are executed in a fork process or even in a remote > > > > machine. Regardless the fact that Avocado is hard coded to set some > > > > environment variables, they are for internal consumption and user is not > > > > allowed to control/configure its behavior. > > > > > > You mean this: > > > > > > http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#environment-variables-for-simple-tests > > > > > > Right? Basically, the fact that Avocado sets some of the job/test state as > > > environment variables, that can be used by SIMPLE tests. > > > > > > > The motivation is the request to provide users an interface to set > > > > and/or keep environment variables for test consumption. > > > > I'm not sure if they're necessarily for test consumption. I think > > the motivation for the original request was to provide the > > standard Unix interface of environment variables for when tests > > are run remotely. > > > > If the motivation is basically about setting the env vars when running tests > remotely, than this brings the discussion about the *local* behavior to: > > 1. Should Avocado default to the standard UNIX behavior of cloning the > environment? > > A: IMHO, yes. That's the current behavior (see my example at the end of the previous email). Except when one runs tests remotely, which is precisely the use case this feature would "fix". > > 2. Could Avocado have have a feature to start tests in a clean(er) > environment? > > A: Possibly yes, but seems low priority. The use case here could be seen > as a plus in predictability, helping to achieve expected test results in > spite of the runner environment. A real world example could be a CI > environment that sets a VERBOSE environment variable. This env var will be > passed over to Avocado, to the test process and finally to a custom binary > (say a benchmark tool) that will produce different output depending on that > environment variable. Doing that type of cleaning in the test code is > possible, but the framework could help with that. > > 2.1. If Avocado provides a "clean(er) test environment" feature, how to > determine which environment variables are passed along? > > A: The "env-keep" approach seems like the obvious way to do it. If the > mechanism is enabled, which I believe should be disabled by default (see > #1), its default list could contain the more or less standard UNIX > environment variables (TERM, SHELL, LANG, etc). Agree. But like you said such a feature would be low priority and optional. The important thing is that the implementation of what we're discussing in this RFC would not interfere with it. > > > These environment variables can change the behavior of both > > Avocado (the runner itself), the tests (after all nothing > > prevents the test writer from using them) and all sub-processes > > executed by the test. > > > > Right. > > > Locally, this is standard: > > > > $ TMPDIR=/whatever/tmp VAR=foo ./avocado run test1.py > > > > But when running avocado remotely, there's no way to configure > > the environment in the destination. The environment variables set > > in the command line below will not be "forwarded" to the remote > > environment: > > > > $ TMPDIR=/whatever/tmp VAR=foo ./avocado run test1.py \ > > --remote... > > > > Right. > > > > > > > > > Use cases > > > > = > > > > 1) Use the command line or the config file to set the environment > > > > variables in tests processes environment; access those variables from > > > > inside the test. > > > > 2) Copy from current environment some environment variable(s) to the > > > > tests processes environment; access those variables from inside the > > > > test. > > > > I think we don't even have to go that far. We can simply say the > > intention is to set the environment variables in the environment > > where Avocado is run. The mechanism is quite standard and well > > understood. > > > > And here comes an important point: I don't think this should be a > > mechanism to pass variables to tests. Although, again, > > environment variables can be used for that
Re: [Avocado-devel] [RFC] Environment Variables
On 06/01/2016 03:07 PM, Ademar Reis wrote: On Tue, May 31, 2016 at 07:30:43AM -0300, Cleber Rosa wrote: I'm replying on top of Cleber because he already said a few things I was going to say. On 05/25/2016 05:31 AM, Amador Pahim wrote: Hi folks, We have requests to handle the environment variables that we can set to the tests. This is the RFC in that regard, with a summary of the ideas already exposed in the original request and some additional planning. The original request is here: https://trello.com/c/Ddcly0oG/312-mechanism-to-provide-environment-variables-to-tests-run-on-a-virtual-machine-remote Motivation == Avocado tests are executed in a fork process or even in a remote machine. Regardless the fact that Avocado is hard coded to set some environment variables, they are for internal consumption and user is not allowed to control/configure its behavior. You mean this: http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#environment-variables-for-simple-tests Right? Basically, the fact that Avocado sets some of the job/test state as environment variables, that can be used by SIMPLE tests. The motivation is the request to provide users an interface to set and/or keep environment variables for test consumption. I'm not sure if they're necessarily for test consumption. I think the motivation for the original request was to provide the standard Unix interface of environment variables for when tests are run remotely. If the motivation is basically about setting the env vars when running tests remotely, than this brings the discussion about the *local* behavior to: 1. Should Avocado default to the standard UNIX behavior of cloning the environment? A: IMHO, yes. 2. Could Avocado have have a feature to start tests in a clean(er) environment? A: Possibly yes, but seems low priority. The use case here could be seen as a plus in predictability, helping to achieve expected test results in spite of the runner environment. A real world example could be a CI environment that sets a VERBOSE environment variable. This env var will be passed over to Avocado, to the test process and finally to a custom binary (say a benchmark tool) that will produce different output depending on that environment variable. Doing that type of cleaning in the test code is possible, but the framework could help with that. 2.1. If Avocado provides a "clean(er) test environment" feature, how to determine which environment variables are passed along? A: The "env-keep" approach seems like the obvious way to do it. If the mechanism is enabled, which I believe should be disabled by default (see #1), its default list could contain the more or less standard UNIX environment variables (TERM, SHELL, LANG, etc). These environment variables can change the behavior of both Avocado (the runner itself), the tests (after all nothing prevents the test writer from using them) and all sub-processes executed by the test. Right. Locally, this is standard: $ TMPDIR=/whatever/tmp VAR=foo ./avocado run test1.py But when running avocado remotely, there's no way to configure the environment in the destination. The environment variables set in the command line below will not be "forwarded" to the remote environment: $ TMPDIR=/whatever/tmp VAR=foo ./avocado run test1.py \ --remote... Right. Use cases = 1) Use the command line or the config file to set the environment variables in tests processes environment; access those variables from inside the test. 2) Copy from current environment some environment variable(s) to the tests processes environment; access those variables from inside the test. I think we don't even have to go that far. We can simply say the intention is to set the environment variables in the environment where Avocado is run. The mechanism is quite standard and well understood. And here comes an important point: I don't think this should be a mechanism to pass variables to tests. Although, again, environment variables can be used for that purpose, Avocado should have a proper interface to provide a dictionary of configuration and variables to each test. The only valid reason for having such a mechanism to pass *different* environment variables to tests, talking about local environment, would be *if and only if* the same environment variable to be set when running Avocado would change the behavior of Avocado itself. Example: $ AVOCADO_LOG_EARLY=1 avocado run avocado-self-tests.py This way, both the first level avocado process (our "real" runner) and other instances run by the "avocado-self-test.py" code would react to that variable. *BUT* this seems a corner case, and I wouldn't think it justifies the implementation of such a feature at this point. Currently, this is erroneously provided by the multiplexer (including --mux-inject), but it should be cleaned up in the near future. Right.
Re: [Avocado-devel] [RFC] Environment Variables
On Tue, May 31, 2016 at 07:30:43AM -0300, Cleber Rosa wrote:
>
I'm replying on top of Cleber because he already said a few
things I was going to say.
> On 05/25/2016 05:31 AM, Amador Pahim wrote:
> > Hi folks,
> >
> > We have requests to handle the environment variables that we can set to
> > the tests. This is the RFC in that regard, with a summary of the ideas
> > already exposed in the original request and some additional planning.
> >
> > The original request is here:
> > https://trello.com/c/Ddcly0oG/312-mechanism-to-provide-environment-variables-to-tests-run-on-a-virtual-machine-remote
> >
> >
> > Motivation
> > ==
> > Avocado tests are executed in a fork process or even in a remote
> > machine. Regardless the fact that Avocado is hard coded to set some
> > environment variables, they are for internal consumption and user is not
> > allowed to control/configure its behavior.
>
> You mean this:
>
> http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#environment-variables-for-simple-tests
>
> Right? Basically, the fact that Avocado sets some of the job/test state as
> environment variables, that can be used by SIMPLE tests.
>
> > The motivation is the request to provide users an interface to set
> > and/or keep environment variables for test consumption.
I'm not sure if they're necessarily for test consumption. I think
the motivation for the original request was to provide the
standard Unix interface of environment variables for when tests
are run remotely.
These environment variables can change the behavior of both
Avocado (the runner itself), the tests (after all nothing
prevents the test writer from using them) and all sub-processes
executed by the test.
Locally, this is standard:
$ TMPDIR=/whatever/tmp VAR=foo ./avocado run test1.py
But when running avocado remotely, there's no way to configure
the environment in the destination. The environment variables set
in the command line below will not be "forwarded" to the remote
environment:
$ TMPDIR=/whatever/tmp VAR=foo ./avocado run test1.py \
--remote...
> >
> > Use cases
> > =
> > 1) Use the command line or the config file to set the environment
> > variables in tests processes environment; access those variables from
> > inside the test.
> > 2) Copy from current environment some environment variable(s) to the
> > tests processes environment; access those variables from inside the test.
I think we don't even have to go that far. We can simply say the
intention is to set the environment variables in the environment
where Avocado is run. The mechanism is quite standard and well
understood.
And here comes an important point: I don't think this should be a
mechanism to pass variables to tests. Although, again,
environment variables can be used for that purpose, Avocado
should have a proper interface to provide a dictionary of
configuration and variables to each test.
Currently, this is erroneously provided by the multiplexer
(including --mux-inject), but it should be cleaned up in the near
future.
> >
> > Proposal
> >
> > - To create a command line option, under the `run` command, to set
> > environment variables that will be available in tests environment process:
> >
> > $ avocado run --test-env='FOO=BAR,FOO1=BAR1' passtest.py
> >
>
> I can relate to this use case...
This would be a simple way of doing it, but something like
"--env-keep=FOO,FOO1" could be a better approach (more about it
below). So one would write it this way:
$ FOO=BAR FOO1=BAR1 avocado run --env-keep='FOO,FOO1' passtest.py
>
> > - To create an option in config file with a dictionary of environment
> > variables to set in test process environment. It can be used as a
> > replacement or complement to the command line option (with lower priority):
> >
> > [tests.env]
> > test_env_vars = {'FOO': 'BAR', 'FOO1': 'BAR1'}
> >
>
> ... while putting those in a config file does not seem like something one
> would do.
>
> In all cases, and more explicitly in the config file example, this is only
> really necessary if/when the environment variable to pass to the test
> actually harms Avocado (considering a local execution, that is, in a forked
> process).
>
> So, if Avocado and the test, share the use of environment variables by the
> same name, then this is a must. Also in the case of execution in other
> "runners", such as remote/vm, this can be quite valuable.
I don't like this interface because I think this opens the door
for abuse. Like I said in a previous paragraph, this interface
should not be a mechanism for passing variables to tests.
>
> > - Create an option in config file with a list of environment variable
> > names to copy from avocado main process environment to the test process
> > environment (similar to env_keep in the /etc/sudoers file):
> >
> > [tests.env]
> > env_keep = ['FOO', 'FOO1', 'FOO2']
> >
I like this approach because it reinforces the
