On Tue, 7 Jan 2014 11:23:02 -0500 (EST) Kamil Paral <kpa...@redhat.com> wrote:
> I've finally read it as well :) Comments below. Thanks for the comments, I really appreciate it. > > At the moment, that task yaml file for rpmlint [1] looks like: > > > > dependencies: > > - rpmlint > > - libtaskbot > > > > input: > > args: envr,arch > > These arguments are provided by the event trigger, right? So, the > event trigger offers a set of arguments and we pick some of them > inside "input" section; they can be then used in other sections. Is > that right? ('arch' is not actually used anywhere in this demo file). At the moment, it's mostly verification that the correct args were passed in but that sums the idea up pretty well. Yeah, rpmlint isn't arch specific - I had it in there as a test and probably should have taken it out. > > > > preparation: > > koji: download $envr > > This is great, a preparation phase is exactly what I wanted. > > > > > execution: > > python: run_rpmlint.py $workdir > > 1) You don't actually use $workdir in run_rpmlint.py. I guess that's > a bug? Or will we guarantee that CWD is set to $workdir? Yeah, the variables haven't been implemented yet. I want to start using something like jinja2 for variable substitution (which would change variables to {{ workdir }}, but I see that as a minor change) but haven't implemented the code for that yet. In the case of $workdir, that variable would be expanded to the directory where the task is being run by the runner @ runtime. > 2) Can we have multiple execution lines that are run sequentially (or > in parallel?) and then consider the aggregate result as a test > result? One example is in our outdated rats_install test: for ks in > ['fedora','updates','updates-testing']: job.run_test('rats_install', > tag=ks, ks=ks, **autoqa_args) We can of course define this as three > separate tests, but sometimes running a few different variants > together in a single test run might make sense. At the moment, no. I hadn't really thought about that, to be honest. The problem that I see with multiple steps is how to express interactions with yaml. How would we send output from one step to another? Would we need to exchange data between steps? If we're talking about multiple, independent (ie, don't require communication between them) steps that each produce results which are aggregated, I don't see much of an issue. If we're talking about combining results or communication between execution steps, that could end up being a bit more complicated. > > > > post: > > shell: clean $workdir > > How is this supposed to work? 'clean' is not an existing command, is > that a keyword? It's a potentially outdated concept that I've been using for the previous taskotron demo deployments - it deletes any downloaded files and stuff created while running. If we do end up needing a 'clean', it should be a command in itself instead of a shell command. > Will we support running arbitrary shell commands or custom scripts > (e.g. './myscript') in this or other phases (like preparation)? Eventually, yes. The code is not implemented in the demo yet, though. > > > > report: > > resultdb: something > > I guess it's possible to run and report multiple results from a > single test run? I.e. upgradepath checking 20 builds and reporting 20 > separate test results. That's the use case that I had in mind initially, yes. The execution phase would output 1 or more results which would each be fed into the report section. > Can I also report a single result from several execution lines? Do we have a use case for this yet? I'd be tempted to put any multiple execution steps into a single python (or other supported language) script. The flow I had in mind is admittedly rather not-just-stdout centric. Execution would produce logging messages on stdout and return TAP/subunit separately. Each TAP/subunit result would be reported, assuming that a report section exists. I suppose that we could add a "resultsmunging" section where all results are fed in and the results are used for reporting but I'm not sure that's a route we really want to take. Off the top of my head example: ---------------------------------------------------------------------- ... execution: python: run_something.py method=first_part workdir=$workdir python: run_something.py method=second_part workdir=$workdir resultmunging: python: munge_results.py output=mungedresults report: resultsdb: $mungedresults ---------------------------------------------------------------------- At what point do we just limit the functionality of the runner and tell folks that they need to do the fancy stuff in the task code called in the execution phase? > I know this is just a demo and we will probably need to discover the > answers for many of the questions above collectively. I was trying to > think about different use cases a maybe discover some missing ones. That's pretty much what I was hoping to do with the demo - spark discussion. At some point, we should probably take this to devel@ to see what people would do with taskotron but for now, I'm tempted to limit the functionality of the yaml based descriptions and wait until we have a working system before soliciting feature requests. Another question is whether this yaml based task description is worth doing or if we should just use a python-based task format similar to what autotest does (inherit from a python class and override methods as needed). Tim
signature.asc
Description: PGP signature
_______________________________________________ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel