Re: fedora_openqa (scheduler/reporter) and createhdds split from openqa_fedora_tools, moved to Pagure
On Tue, 2017-02-14 at 15:55 -0800, Adam Williamson wrote: > I'm trying to clean up all the appropriate bits in Pagure, READMEs, > wiki, ansible tasks etc. this afternoon. If it doesn't all look to be > in line tomorrow AM, please do let me know (or just go ahead and fix > anything straightforward you find that I've overlooked). Yes, I know > that right now the fedora_openqa README references a 'Fedora openQA > wiki page' that doesn't exist, I'm planning to write it soon, and move > the 'how to install openQA' content from openqa_fedora_tools into it. Here it is: https://fedoraproject.org/wiki/openQA -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
fedora_openqa (scheduler/reporter) and createhdds split from openqa_fedora_tools, moved to Pagure
Hi folks! So continuing with the agreed plan (for openQA bits) to move git repos to Pagure and split up openqa_fedora_tools, I've split out and moved the scheduler/reporter library and CLI - now called 'fedora_openqa', since 'fedora_openqa_schedule' was a dumb name for a thing that actually does more than just scheduling - and createhdds: https://pagure.io/fedora-qa/fedora_openqa https://pagure.io/fedora-qa/createhdds I have renamed the Phabricator projects for the openQA tests and scheduler too: openqa_fedora -> os-autoinst-distri-fedora openqa_fedora_tools -> fedora_openqa The new names match the git repo names. Almost all the issues and diffs for openqa_fedora_tools were really for the scheduler; for createhdds I think we can just use Pagure issues / PRs. I will move the one outstanding createhdds diff to Pagure manually. For fedora_openqa we will still be using Phab for issues and diffs for now, just with the new project name. The .arcconfig in the new repo should be correct. I'm trying to clean up all the appropriate bits in Pagure, READMEs, wiki, ansible tasks etc. this afternoon. If it doesn't all look to be in line tomorrow AM, please do let me know (or just go ahead and fix anything straightforward you find that I've overlooked). Yes, I know that right now the fedora_openqa README references a 'Fedora openQA wiki page' that doesn't exist, I'm planning to write it soon, and move the 'how to install openQA' content from openqa_fedora_tools into it. Thanks! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Re: Taskotron CI in Taskotron
On Tue, 2017-02-14 at 14:46 -0500, Kamil Paral wrote: > > The only problem with this kind of testing is, that we still don't really > > have a good way to test trigger, as it is tied to external events. My idea > > here was that I could add something like wiki edit consumer, and trigger > > tasks off of that, making that one "triggering" edit from inside the > > testsuite. > > In my experience, some fedmsg notifications are quite delayed from > time to time, easily by several hours. But I don't know if the delay > is in actual fedmsg bus, or just in FMN (which I use to get notified > on IRC). I suspect rather FMN. But if that turned out to be a fedmsg > delay, that might make this approach impractical (or at least less > reliable). So something to consider and possibly investigate. I believe it's FMN. I've noticed the same delay with FMN notifications, but I've never noticed any delay with the actual fedmsg notifications used by all the stuff we've built around release validation (compose complete fedmsgs, openQA test complete fedmsgs etc.) > I think it would be better to fake the input data. Either by having > our own fedmsg producer and/or bus (I don't know precisely how fedmsg > works and how hard is to set up something like that), Are you aware of fedmsg-dg-replay? It's a fairly easy way to 'replay' fedmsgs for testing. All you need (IIRC) is the fedmsg-relay service running on the same system, and you can run 'fedmsg-dg-replay --msg-id ' and the message will be replayed on the local bus (if the message is .prod. or .stg. , this will be changed to .dev. , and the message will not be signed). I don't remember exactly how the taskotron trigger code works, but it shouldn't be too hard to configure a trigger to accept unsigned messages with .dev. topics, for the purpose of testing with -replay. In the fedmsg consumers I've written I have a pattern of providing three consumers, one which listens for .prod. messages, one for .stg. messages, and one which listens for for .dev. messages and is intended for testing with fedmsg-dg-replay. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Re: Taskotron CI in Taskotron
> The only problem with this kind of testing is, that we still don't really > have a good way to test trigger, as it is tied to external events. My idea > here was that I could add something like wiki edit consumer, and trigger > tasks off of that, making that one "triggering" edit from inside the > testsuite. In my experience, some fedmsg notifications are quite delayed from time to time, easily by several hours. But I don't know if the delay is in actual fedmsg bus, or just in FMN (which I use to get notified on IRC). I suspect rather FMN. But if that turned out to be a fedmsg delay, that might make this approach impractical (or at least less reliable). So something to consider and possibly investigate. I think it would be better to fake the input data. Either by having our own fedmsg producer and/or bus (I don't know precisely how fedmsg works and how hard is to set up something like that), or by making some code adjustments in trigger that will allow us to bypass the fedmsg reception and replace it with our own fedmsg, but cover as much surrounding code as possible. ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Re: Libtaskotron - allow non-cli data input
> On Wed, Feb 8, 2017 at 2:26 PM, Kamil Paral < kpa...@redhat.com > wrote:
> > > This is what I meant - keeping item as is, but being able to pass another
> > > structure to the formula, which can then be used from it. I'd still like
> > > to
> > > keep the item to a single string, so it can be queried easily in the
> > > resultsdb. The item should still represent what was tested. It's just
> > > that
> > > I
> > > want to be able to pass arbitrary data to the formulae, without the need
> > > for
> > > ugly hacks like we have seen with the git commits lately.
> >
>
> > So, the question is now how much we want the `item` to uniquely identify
> > the
> > item under test. Currently we mostly do (rpmlint, rpmgrill) and sometimes
> > don't (depcheck, because item is NVR, but the full ID is NEVRA, and we
> > store
> > arch in the results extradata section).
>
> I still kind of believe that the `item` should be chosen with great respect
> to what actually is the item under test, but it also really depends on what
> you want to do with it later on. Note that the `item` is actually a
> convention (yay, more water to adamw's "if we only had some awesome new
> project" mill), and is not enforced in any way. I believe that there should
> be firm rules (once again - conventions) on what the item is for each "well
> known" item type, so you can kind-of assume that if you query for
> `item=foo=koji_build` you are getting the results related to that
> build.
> As we were discussing privately with the item types (I'm not going to go into
> much detail here, but for the rest of you guys - I'm contemplating making
> the types more general, and using more of the 'metadata' to store additional
> spefics - like replacing `type=koji_build` with `type=build, source=koji`,
> or `type=build, source=brew` - on the high level, you know that a
> package/build was tested, and you don't really care where it came from, but
> you sometimes might care, and so there is the additional metadata stored. We
> could even have more types stored for one results, or I don't know... It's
> complicated), the idea behind item is that it should be a reasonable value,
> that carries the "what was tested" information, and you will use the other
> "extra-data" fields to provide more details (like we kind-of want to do with
> arch, but we don't really..). The reason for it to be 'reasonable value" and
> not "superset of all values that we have" is to make the general querying a
> bit more straightforward.
> > If we have structured input data, what happens to `item` for
> > check_modulemd?
> > Currently it is "namespace/module#commithash". Will it stay the same, and
> > they'll just avoid parsing it because we'll also provide ${data.namespace},
> > ${data.module} and ${data.hash}? Or will the `item` be perhaps just
> > "module"
> > (and the rest will be stored as extradata)? What happens when we have a
> > generic git_commit type, and the source can be an arbitrary service? Will
> > have some convention to use item as "giturl#commithash"?
>
> Once again - whatever makes sense as the item. For me that would be the
> Repo/SHA combo, with server, repo, branch, and commit in extradata.
> And it comes to "storing as much relevant metadata as possible" once again.
> The thing is, that as long as stuff is predictable, it almost does not
> matter what it is, and it once again points out how good of an idea is the
> conventions stuff. I believe that we are now storing much less metadata in
> resultsdb than we should, and it is caused mostly by the fact that
> - we did not really need to use the results much so far
> - it is pretty hard to pass data into libtaskotron, and querying all the
> services all the time, to get the metadata, is/was deemed a bad idea - why
> do it ourselves, if the consumer can get it himself. They know that it is
> koji_build, so they can query koji.
> There is a fine balance to be struck, IMO, so we don't end up storing "all
> the data" in resultsdb. But I believe that the stuff relevant for the result
> consumption should be there.
> > Because the ideal way would be to store the whole item+data structure as
> > item
> > in resultsdb. But that's hard to query for humans, so we want a simple
> > string as an identifier.
>
> This, for me, is once again about being predictable. As I said above, I still
> think that `item` should be a reasonable identifier, but not necessary a
> superset of all the info. That is what the extra data is for. Talking
> about...
> > But sometimes there can be a lot of data points which uniquely identify the
> > thing under test only when you specify it all (for example what Dan wrote,
> > sometimes the ID is the old NVR *plus* the new NVR). Will we want to
> > somehow
> > combine them into a single item value? We should give some directions how
> > people should construct their items.
>
> My gut feeling here would be storing the "new NVR" (the thing that actually
> caused the test to be executed)
