Re: Taskotron CI in Taskotron
On Wed, Feb 15, 2017 at 5:55 PM, Adam Williamson wrote: > On Wed, 2017-02-15 at 12:59 +0100, Josef Skladanka wrote: > > On Tue, Feb 14, 2017 at 8:51 PM, Adam Williamson < > adamw...@fedoraproject.org > > > wrote: > > > Are you aware of fedmsg-dg-replay? It's a fairly easy way to 'replay' > > > fedmsgs for testing. All you need (IIRC) is the fedmsg-relay service > > > running on the same system, and you can run > > > > > > > I am, but it has this bad quality of changing the topic, so we would have > > to change the consumers' topics, so we would need to change those too. Or > > make it configurable in some way... > > I'd rather do it the way I have it now - using the trigger's internal > > replay functionality instead of doing unnecessary complicated changes > just > > for the sake of using it to test stuff once in a while. > > I was thinking that it's probably not that difficult to set up a > testing fedmsg bus as a test fixture with some canned messages that can > be replayed on request, but I haven't looked at doing it so I really > don't know how much work it is. I wonder if fedmsg's test suite does > it. > > Ah so, I did not get that at the first read, and now it is obvious even from the previous email *facepalm*. Yeah, that would make sense, I guess. We'll see about that, at the moment, we have bigger fish to fry, but in the end I'd like to have this stuff covered too. Thanks for the good idea, though! joza ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Re: Taskotron CI in Taskotron
On Wed, 2017-02-15 at 12:59 +0100, Josef Skladanka wrote: > On Tue, Feb 14, 2017 at 8:51 PM, Adam Williamson > wrote: > > Are you aware of fedmsg-dg-replay? It's a fairly easy way to 'replay' > > fedmsgs for testing. All you need (IIRC) is the fedmsg-relay service > > running on the same system, and you can run > > > > I am, but it has this bad quality of changing the topic, so we would have > to change the consumers' topics, so we would need to change those too. Or > make it configurable in some way... > I'd rather do it the way I have it now - using the trigger's internal > replay functionality instead of doing unnecessary complicated changes just > for the sake of using it to test stuff once in a while. I was thinking that it's probably not that difficult to set up a testing fedmsg bus as a test fixture with some canned messages that can be replayed on request, but I haven't looked at doing it so I really don't know how much work it is. I wonder if fedmsg's test suite does it. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Re: Taskotron CI in Taskotron
> So, the repo now has working PoC > https://pagure.io/taskotron/task-taskotron-ci > Readme contains example on how to run the task. > Works on my setup, and I'd be glad if somebody else tried. I tried, and it didn't eat my laptop. Also it worked. Good job. /me throws cookies at jskladan ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Re: Taskotron CI in Taskotron
On Tue, Feb 14, 2017 at 8:51 PM, Adam Williamson wrote: > Are you aware of fedmsg-dg-replay? It's a fairly easy way to 'replay' > fedmsgs for testing. All you need (IIRC) is the fedmsg-relay service > running on the same system, and you can run > I am, but it has this bad quality of changing the topic, so we would have to change the consumers' topics, so we would need to change those too. Or make it configurable in some way... I'd rather do it the way I have it now - using the trigger's internal replay functionality instead of doing unnecessary complicated changes just for the sake of using it to test stuff once in a while. J. ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Re: Taskotron CI in Taskotron
> On Tue, 2017-02-14 at 14:46 -0500, Kamil Paral wrote: > > > The only problem with this kind of testing is, that we still don't really > > > have a good way to test trigger, as it is tied to external events. My > > > idea > > > here was that I could add something like wiki edit consumer, and trigger > > > tasks off of that, making that one "triggering" edit from inside the > > > testsuite. > > > > In my experience, some fedmsg notifications are quite delayed from > > time to time, easily by several hours. But I don't know if the delay > > is in actual fedmsg bus, or just in FMN (which I use to get notified > > on IRC). I suspect rather FMN. But if that turned out to be a fedmsg > > delay, that might make this approach impractical (or at least less > > reliable). So something to consider and possibly investigate. > > I believe it's FMN. I've noticed the same delay with FMN notifications, > but I've never noticed any delay with the actual fedmsg notifications > used by all the stuff we've built around release validation (compose > complete fedmsgs, openQA test complete fedmsgs etc.) > > > I think it would be better to fake the input data. Either by having > > our own fedmsg producer and/or bus (I don't know precisely how fedmsg > > works and how hard is to set up something like that), > > Are you aware of fedmsg-dg-replay? It's a fairly easy way to 'replay' > fedmsgs for testing. All you need (IIRC) is the fedmsg-relay service > running on the same system, and you can run > 'fedmsg-dg-replay --msg-id ' and the message will be > replayed on the local bus (if the message is .prod. or .stg. , this > will be changed to .dev. , and the message will not be signed). > > I don't remember exactly how the taskotron trigger code works, but it > shouldn't be too hard to configure a trigger to accept unsigned > messages with .dev. topics, for the purpose of testing with -replay. In > the fedmsg consumers I've written I have a pattern of providing three > consumers, one which listens for .prod. messages, one for .stg. > messages, and one which listens for for .dev. messages and is intended > for testing with fedmsg-dg-replay. Great info, thanks. ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Re: Taskotron CI in Taskotron
On Tue, 2017-02-14 at 14:46 -0500, Kamil Paral wrote: > > The only problem with this kind of testing is, that we still don't really > > have a good way to test trigger, as it is tied to external events. My idea > > here was that I could add something like wiki edit consumer, and trigger > > tasks off of that, making that one "triggering" edit from inside the > > testsuite. > > In my experience, some fedmsg notifications are quite delayed from > time to time, easily by several hours. But I don't know if the delay > is in actual fedmsg bus, or just in FMN (which I use to get notified > on IRC). I suspect rather FMN. But if that turned out to be a fedmsg > delay, that might make this approach impractical (or at least less > reliable). So something to consider and possibly investigate. I believe it's FMN. I've noticed the same delay with FMN notifications, but I've never noticed any delay with the actual fedmsg notifications used by all the stuff we've built around release validation (compose complete fedmsgs, openQA test complete fedmsgs etc.) > I think it would be better to fake the input data. Either by having > our own fedmsg producer and/or bus (I don't know precisely how fedmsg > works and how hard is to set up something like that), Are you aware of fedmsg-dg-replay? It's a fairly easy way to 'replay' fedmsgs for testing. All you need (IIRC) is the fedmsg-relay service running on the same system, and you can run 'fedmsg-dg-replay --msg-id ' and the message will be replayed on the local bus (if the message is .prod. or .stg. , this will be changed to .dev. , and the message will not be signed). I don't remember exactly how the taskotron trigger code works, but it shouldn't be too hard to configure a trigger to accept unsigned messages with .dev. topics, for the purpose of testing with -replay. In the fedmsg consumers I've written I have a pattern of providing three consumers, one which listens for .prod. messages, one for .stg. messages, and one which listens for for .dev. messages and is intended for testing with fedmsg-dg-replay. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Re: Taskotron CI in Taskotron
> The only problem with this kind of testing is, that we still don't really > have a good way to test trigger, as it is tied to external events. My idea > here was that I could add something like wiki edit consumer, and trigger > tasks off of that, making that one "triggering" edit from inside the > testsuite. In my experience, some fedmsg notifications are quite delayed from time to time, easily by several hours. But I don't know if the delay is in actual fedmsg bus, or just in FMN (which I use to get notified on IRC). I suspect rather FMN. But if that turned out to be a fedmsg delay, that might make this approach impractical (or at least less reliable). So something to consider and possibly investigate. I think it would be better to fake the input data. Either by having our own fedmsg producer and/or bus (I don't know precisely how fedmsg works and how hard is to set up something like that), or by making some code adjustments in trigger that will allow us to bypass the fedmsg reception and replace it with our own fedmsg, but cover as much surrounding code as possible. ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Re: Taskotron CI in Taskotron
So, the repo now has working PoC https://pagure.io/taskotron/task-taskotron-ci Readme contains example on how to run the task. Works on my setup, and I'd be glad if somebody else tried. J. On Fri, Feb 10, 2017 at 7:31 AM, Josef Skladanka wrote: > > > On Thu, Feb 9, 2017 at 5:58 PM, Matthew Miller > wrote: > >> On Thu, Feb 09, 2017 at 03:29:13AM +0100, Josef Skladanka wrote: >> > I finally got some work done on the CI task for Taskotron in Taskotron. >> The >> > idea here is that after each commit (of a relevant project - trigger, >> > execdb, resultsdb, libtaskotron) to pagure, we will run the whole stack >> in >> > docker containers, and execute a known "phony" task, to see whether it >> all >> > goes fine. >> >> This is excellent. I'd love, eventually, to get to a point where we can >> run the checks _pre_ commit and gate on them. Is there a path from this >> to that? > > > Absolutely, that is the goal. > > Generally speaking, we'd like to run tests on Pagure's PRs. > For taskotron specifically, we'll need to figure out some Phabricator > plugin that fires off a fedmsg (or calls some API, whatever) on new > Differential request, but generally it is the same idea. > > Joza > ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Re: Taskotron CI in Taskotron
On Thu, Feb 9, 2017 at 5:58 PM, Matthew Miller wrote: > On Thu, Feb 09, 2017 at 03:29:13AM +0100, Josef Skladanka wrote: > > I finally got some work done on the CI task for Taskotron in Taskotron. > The > > idea here is that after each commit (of a relevant project - trigger, > > execdb, resultsdb, libtaskotron) to pagure, we will run the whole stack > in > > docker containers, and execute a known "phony" task, to see whether it > all > > goes fine. > > This is excellent. I'd love, eventually, to get to a point where we can > run the checks _pre_ commit and gate on them. Is there a path from this > to that? Absolutely, that is the goal. Generally speaking, we'd like to run tests on Pagure's PRs. For taskotron specifically, we'll need to figure out some Phabricator plugin that fires off a fedmsg (or calls some API, whatever) on new Differential request, but generally it is the same idea. Joza ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Re: Taskotron CI in Taskotron
On Thu, Feb 09, 2017 at 03:29:13AM +0100, Josef Skladanka wrote: > I finally got some work done on the CI task for Taskotron in Taskotron. The > idea here is that after each commit (of a relevant project - trigger, > execdb, resultsdb, libtaskotron) to pagure, we will run the whole stack in > docker containers, and execute a known "phony" task, to see whether it all > goes fine. This is excellent. I'd love, eventually, to get to a point where we can run the checks _pre_ commit and gate on them. Is there a path from this to that? -- Matthew Miller Fedora Project Leader ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Taskotron CI in Taskotron
Gang, I finally got some work done on the CI task for Taskotron in Taskotron. The idea here is that after each commit (of a relevant project - trigger, execdb, resultsdb, libtaskotron) to pagure, we will run the whole stack in docker containers, and execute a known "phony" task, to see whether it all goes fine. The way I devised is that I'll build a 'testsuite' container based on the Trigger, and instead of running the fedmsg hub, I'll just use the CLI to "replay" what would happen on a known, predefined fedmsg. The testsuite will then watch execdb and resultsdb, whether everything went fine. It is not at all finished, but I started hacking on it here: https://pagure.io/taskotron/task-taskotron-ci I hope to finish it (to a point where it runs the phony task) till the end of the week. At that point, I'd be glad for any actual, sensible task ideas to ideally test as much of the capabilities of the libtaskotron/execdb/resultsdb as possible. The only problem with this kind of testing is, that we still don't really have a good way to test trigger, as it is tied to external events. My idea here was that I could add something like wiki edit consumer, and trigger tasks off of that, making that one "triggering" edit from inside the testsuite. But As it's almost 4am here, I'm not sure it is the best idea. Once again, I'll be glad for any input/ideas/evil laughter. Joza ___ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
