Re: Possible QA Devel Projects for GSoC 2014
On Tue, 11 Mar 2014 07:52:06 -0400 (EDT) Kamil Paral kpa...@redhat.com wrote: Well, are we sure now how exactly the client setup process will be hooked into taskotron or its underlying tools? I'm not exactly sure how this will work, either. It's going to depend on what we end up using for graphical testing, what openstack is capable of, what cloud resources we have access to and what the cloud SIG ends up needing for their testing. Are we committed to using buildbot, or might it change? I don't really see how this is relevant. Can you elaborate on how using buildbot or not would factor in here? Hmm. In AutoQA, we used Autotest for managing test clients. Any disposable client support would most probably needed some support in Autotest. I assumed it's the same for Buildbot. Instead of using a pre-defined machine, it will need to be able to say you there, create me a machine matching these requirements; zzz; thank you. There are several ways that we could go about implementing disposable clients. Buildbot already has some support for ephemeral slaves using OpenStack but I've not spent any time with it yet so I can't speak to how well it would work for our needs. The only potential problem I see with openstack is that I'm not sure we can do graphical testing with openstack instances (ie, direct access to the VNC interface or through libvirt). We'd have to investigate that to be sure, though. So, two different systems (i.e. two different databases) displayed in a single web frontend, right? I guess it makes sense. Yeah, that's what I had in mind, anyways. Since student registration has started, I'd like to get our proposed ideas in the wiki soon. The question of whether any of these projects would be worth distracting folks from other dev/testing work remains - any thoughts on that front? So far it seems you're the only candidate for mentoring, you it's probably up to your decision and past experience. Of course any of us will help the student when needed, but I assume most of the communication will be between the mentor and the student. Josef says that he recommend picking a project for which we don't need to spend weeks to introduce and explain to the student the whole project, our needs, etc. Something that is simple to explain, and can be implemented without being blocked on us. Yeah, that's one of my concerns - the time that at least one of us would need to invest. I concur with Josef's suggestion of something that doesn't require too much background but I also think that having multiple mentors and involved folks is also important so it's not just 1 person interacting with a student. It sounds like the results middleware project, the graphical installation project, the gnome-continuous project and _maybe_ the disposable client project are the best candidates. Any thoughts on the value for those? I don't know much about gnome-continuous, but the rest of the projects you mentioned really seems to be the best picks. Results middleware is probably closest to Josef, graphical testing is closest to me and disposable clients is closest to you. When it comes to importance, disposable clients have probably the highest priority, then results middleware, and then comes the rest. But that's just my guesses. I think that gnome-continuous would be a good way to start getting some graphical testing done without needing disposable clients but I'm not clear on how much setup, integration and hw overhead there would be for the support infrastructure (rpm-ostree, at least AFAIK). I think that either disposable clients or the results middleware are going to be the highest priorities of the projects I listed. They're both things that have been (at least indirectly) asked for several times over the last couple years and I think that getting them implemented would help restore some confidence that we're going in the right direction. At this point, there are 3 or so days left in the application process and since we all seem to be on the fence on this a bit, I'm thinking it would be wise to skip GSoC for this year. I might put the ideas up on the wiki page tomorrow in case someone sees them and gets inspired for next year, though. Tim signature.asc Description: PGP signature ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
Re: Possible QA Devel Projects for GSoC 2014
On Tue, 11 Mar 2014 05:02:28 -0400 (EDT) Kamil Paral kpa...@redhat.com wrote: Graphical Installation Testing Continue the work that Jan started with his thesis or look into integrating something like openqa. The emphasis here is on the graphical interface since ks-based installation testing could be covered by stuff already written for beaker After talking to Michal Hrusecky from OpenSUSE on DevConf, I'm pretty convinced we should collaborate with them on OpenQA. They have unleashed their whole Boosters team to work on it, and they're fixing many of the previous pain-points (except for Perl, unfortunately). They also try to have it pretty generic, without unneeded ties to OpenSUSE infrastructure (e.g. they've just implemented OpenID login), and they would really appreciate our collaboration. We keep running into this and I really need to spend some time with OpenQA again. When I looked at it a couple years ago, there were several things that I didn't like about how the framework actually works (entire screenshot comparision, forcing keyboard interactions etc.) but it's possible that they've fixed those issues. Look here: https://www.google.cz/#q=openqa+site:http:%2F%2Flizards.opensuse.org They use OpenCV instead of screenshot checksuming now. I'm not sure what you mean by keyboard interactions. IIRC, they were using opencv the last time I looked at openqa. The image checksumming stuff is worse than the bits I had concerns about, to be honest :) What I mean by keyboard interactions is that you can't use the mouse - it was a strict script of keyboard actions. The runner made keypresses as scripted and nothing more. One major drawback is that they still don't support task distribution (to test clients). Everything is executed on a single machine. But they say they are able to run lots of test cases every single day, and we intend to run just a fraction of it, so performance-wise it shouldn't be a problem. We'd still need to evaluate the system to see if it can do in reality what we need it to do, what the level of integration work will be and what kind of patches we'd need to write and submit. I'm really not itching to write our own system here but at the same time, I'm also not thrilled about the idea of jumping into a system we have little to no control over just because it looks like it'd save us time in the short term. As bad as NIH syndrome is, shoehorning an existing library/system into a place where it isn't going to work well and may cause us just as many problems is also not a good thing. Disposable Client Support This is another of the big features that we'll be implementing before too long. It's one of the reasons that we made the shift from AutoQA to taskotron and is blocking features which folks say they want to see (user-submitted tasks, mostly). This would involve some investigation into whether OpenStack would be practical, if there is another provisioning system we could use or if we'll be forced to roll our own (which I'd rather avoid). There should be some tie-in with the graphical installation support and possibly the gnome integration tests. As usual, we're still missing the required pieces the student should work with. But as a pilot and a way how to discover and evaluate possible options, this could be interesting. What are we missing that wouldn't be part of this project? Well, are we sure now how exactly the client setup process will be hooked into taskotron or its underlying tools? I'm not exactly sure how this will work, either. It's going to depend on what we end up using for graphical testing, what openstack is capable of, what cloud resources we have access to and what the cloud SIG ends up needing for their testing. Are we committed to using buildbot, or might it change? I don't really see how this is relevant. Can you elaborate on how using buildbot or not would factor in here? System for apparent results storage and modification There has to be a better title for this but it would be one of the last major steps in enabling bodhi/koji to block builds/updates on check failures. The idea would be to provide an interface which can decide whether a build/update is OK based on what checks were passed/failed. It would have a mechanism for manual overrides and algorithmic overrides (ie, we know that foo has problem X and are working on it, ignore failures for now) so that we don't upset packagers more than
Re: Possible QA Devel Projects for GSoC 2014
Well, are we sure now how exactly the client setup process will be hooked into taskotron or its underlying tools? I'm not exactly sure how this will work, either. It's going to depend on what we end up using for graphical testing, what openstack is capable of, what cloud resources we have access to and what the cloud SIG ends up needing for their testing. Are we committed to using buildbot, or might it change? I don't really see how this is relevant. Can you elaborate on how using buildbot or not would factor in here? Hmm. In AutoQA, we used Autotest for managing test clients. Any disposable client support would most probably needed some support in Autotest. I assumed it's the same for Buildbot. Instead of using a pre-defined machine, it will need to be able to say you there, create me a machine matching these requirements; zzz; thank you. So, two different systems (i.e. two different databases) displayed in a single web frontend, right? I guess it makes sense. Yeah, that's what I had in mind, anyways. Since student registration has started, I'd like to get our proposed ideas in the wiki soon. The question of whether any of these projects would be worth distracting folks from other dev/testing work remains - any thoughts on that front? So far it seems you're the only candidate for mentoring, you it's probably up to your decision and past experience. Of course any of us will help the student when needed, but I assume most of the communication will be between the mentor and the student. Josef says that he recommend picking a project for which we don't need to spend weeks to introduce and explain to the student the whole project, our needs, etc. Something that is simple to explain, and can be implemented without being blocked on us. It sounds like the results middleware project, the graphical installation project, the gnome-continuous project and _maybe_ the disposable client project are the best candidates. Any thoughts on the value for those? I don't know much about gnome-continuous, but the rest of the projects you mentioned really seems to be the best picks. Results middleware is probably closest to Josef, graphical testing is closest to me and disposable clients is closest to you. When it comes to importance, disposable clients have probably the highest priority, then results middleware, and then comes the rest. But that's just my guesses. ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
Re: Possible QA Devel Projects for GSoC 2014
On Mon, 10 Mar 2014 08:23:16 -0400 (EDT) Kamil Paral kpa...@redhat.com wrote: Fedora has been accepted as a mentoring org for GSoC 2014 and I'm planning to sign up as a mentor again this year. I'm trying to think of good projects that we could put on the list of suggestions of things that we'd like students to work on and figured it was a good topic for wider discussion. I'd like to avoid any blockerbugs projects this year so that we can focus on taskotron and keeping forward momentum. I've made a quick list of the possible projects that I can think of. Please comment on any of them that you think would benefit from having a dedicated intern over the summer or add to the list if you can think of other projects. Ideally, the projects would be self-contained enough for the student to demonstrate their progress over the summer but not so isolated that they wouldn't be interacting with the community. Projects should be far enough out that we wouldn't be critically blocked on them but close enough that the effects of their work are visible before the end of GSoC. Thanks for thinking about this. Personally I'm really bad at coming up with such project topics. One of the major concerns I have is whether it is efficient to mentor someone, or whether it would be better to have your (and/or someone else's) time fully devoted to the project itself. We would have to come up with such topics that don't require much mentoring from our side, and which are not blocked on our future actions (we don't want the student to wait until we implement feature X). Yeah, that's one concern and one of the reason that I'd be more interested in a project that isn't so isolated - that way we could have multiple folks helping out with the project instead of the 1:1 stuff that we've done for the last 2 years of GSoC. Tim Graphical Installation Testing Continue the work that Jan started with his thesis or look into integrating something like openqa. The emphasis here is on the graphical interface since ks-based installation testing could be covered by stuff already written for beaker After talking to Michal Hrusecky from OpenSUSE on DevConf, I'm pretty convinced we should collaborate with them on OpenQA. They have unleashed their whole Boosters team to work on it, and they're fixing many of the previous pain-points (except for Perl, unfortunately). They also try to have it pretty generic, without unneeded ties to OpenSUSE infrastructure (e.g. they've just implemented OpenID login), and they would really appreciate our collaboration. We keep running into this and I really need to spend some time with OpenQA again. When I looked at it a couple years ago, there were several things that I didn't like about how the framework actually works (entire screenshot comparision, forcing keyboard interactions etc.) but it's possible that they've fixed those issues. I'm just not sure about timing. If the task is to integrate it into our test infrastructure, we need to have some infrastructure to begin with :-) True, but we do need to start somewhere. I think that we'll have some new machines by that point but it's certainly something that we should be more sure of before committing to GSoC. I don't know if all of them, but a large number of the Boosters team is located in CZ timezone, so Europe-based student would be a better fit for this, in order to talk to them on their IRC channel. Well, we don't often have much control over where interested students live but in principle, that makes sense. Beaker Integration This is on our roadmap and is certainly something that would be useful. It would require a bit of infrastructure work and likely the cooperation of the beaker devs but seems like it could be a good project even if it isn't the most exciting thing ever. On the other hand, this could end up being rather critical and may not be something that we want to mostly hand off to a student. I'm not sure this is a good project for a student, because it's likely to involve a lot of communication with internal teams. And again, I don't think our test infra is ready yet. Gnome Integration Test Support An over-simplification of this would be to say take the stuff that's run as part of gnome continuous [1] and run it on fedora packages. The goal would be to have gnome's integration test suites running with any new gnome builds. [1] https://wiki.gnome.org/action/show/Projects/GnomeContinuous
Re: Possible QA Devel Projects for GSoC 2014
Fedora has been accepted as a mentoring org for GSoC 2014 and I'm planning to sign up as a mentor again this year. I'm trying to think of good projects that we could put on the list of suggestions of things that we'd like students to work on and figured it was a good topic for wider discussion. I'd like to avoid any blockerbugs projects this year so that we can focus on taskotron and keeping forward momentum. I've made a quick list of the possible projects that I can think of. Please comment on any of them that you think would benefit from having a dedicated intern over the summer or add to the list if you can think of other projects. Ideally, the projects would be self-contained enough for the student to demonstrate their progress over the summer but not so isolated that they wouldn't be interacting with the community. Projects should be far enough out that we wouldn't be critically blocked on them but close enough that the effects of their work are visible before the end of GSoC. Thanks for thinking about this. Personally I'm really bad at coming up with such project topics. One of the major concerns I have is whether it is efficient to mentor someone, or whether it would be better to have your (and/or someone else's) time fully devoted to the project itself. We would have to come up with such topics that don't require much mentoring from our side, and which are not blocked on our future actions (we don't want the student to wait until we implement feature X). Tim Graphical Installation Testing Continue the work that Jan started with his thesis or look into integrating something like openqa. The emphasis here is on the graphical interface since ks-based installation testing could be covered by stuff already written for beaker After talking to Michal Hrusecky from OpenSUSE on DevConf, I'm pretty convinced we should collaborate with them on OpenQA. They have unleashed their whole Boosters team to work on it, and they're fixing many of the previous pain-points (except for Perl, unfortunately). They also try to have it pretty generic, without unneeded ties to OpenSUSE infrastructure (e.g. they've just implemented OpenID login), and they would really appreciate our collaboration. I'm just not sure about timing. If the task is to integrate it into our test infrastructure, we need to have some infrastructure to begin with :-) I don't know if all of them, but a large number of the Boosters team is located in CZ timezone, so Europe-based student would be a better fit for this, in order to talk to them on their IRC channel. Beaker Integration This is on our roadmap and is certainly something that would be useful. It would require a bit of infrastructure work and likely the cooperation of the beaker devs but seems like it could be a good project even if it isn't the most exciting thing ever. On the other hand, this could end up being rather critical and may not be something that we want to mostly hand off to a student. I'm not sure this is a good project for a student, because it's likely to involve a lot of communication with internal teams. And again, I don't think our test infra is ready yet. Gnome Integration Test Support An over-simplification of this would be to say take the stuff that's run as part of gnome continuous [1] and run it on fedora packages. The goal would be to have gnome's integration test suites running with any new gnome builds. [1] https://wiki.gnome.org/action/show/Projects/GnomeContinuous Disposable Client Support This is another of the big features that we'll be implementing before too long. It's one of the reasons that we made the shift from AutoQA to taskotron and is blocking features which folks say they want to see (user-submitted tasks, mostly). This would involve some investigation into whether OpenStack would be practical, if there is another provisioning system we could use or if we'll be forced to roll our own (which I'd rather avoid). There should be some tie-in with the graphical installation support and possibly the gnome integration tests. As usual, we're still missing the required pieces the student should work with. But as a pilot and a way how to discover and evaluate possible options, this could be interesting. RPM-OSTree Support/Integration I haven't
Possible QA Devel Projects for GSoC 2014
Fedora has been accepted as a mentoring org for GSoC 2014 and I'm planning to sign up as a mentor again this year. I'm trying to think of good projects that we could put on the list of suggestions of things that we'd like students to work on and figured it was a good topic for wider discussion. I'd like to avoid any blockerbugs projects this year so that we can focus on taskotron and keeping forward momentum. I've made a quick list of the possible projects that I can think of. Please comment on any of them that you think would benefit from having a dedicated intern over the summer or add to the list if you can think of other projects. Ideally, the projects would be self-contained enough for the student to demonstrate their progress over the summer but not so isolated that they wouldn't be interacting with the community. Projects should be far enough out that we wouldn't be critically blocked on them but close enough that the effects of their work are visible before the end of GSoC. Tim Graphical Installation Testing Continue the work that Jan started with his thesis or look into integrating something like openqa. The emphasis here is on the graphical interface since ks-based installation testing could be covered by stuff already written for beaker Beaker Integration This is on our roadmap and is certainly something that would be useful. It would require a bit of infrastructure work and likely the cooperation of the beaker devs but seems like it could be a good project even if it isn't the most exciting thing ever. On the other hand, this could end up being rather critical and may not be something that we want to mostly hand off to a student. Gnome Integration Test Support An over-simplification of this would be to say take the stuff that's run as part of gnome continuous [1] and run it on fedora packages. The goal would be to have gnome's integration test suites running with any new gnome builds. [1] https://wiki.gnome.org/action/show/Projects/GnomeContinuous Disposable Client Support This is another of the big features that we'll be implementing before too long. It's one of the reasons that we made the shift from AutoQA to taskotron and is blocking features which folks say they want to see (user-submitted tasks, mostly). This would involve some investigation into whether OpenStack would be practical, if there is another provisioning system we could use or if we'll be forced to roll our own (which I'd rather avoid). There should be some tie-in with the graphical installation support and possibly the gnome integration tests. RPM-OSTree Support/Integration I haven't used rpm-ostree enough yet to figure out how good of a fit it'd be with taskotron but from the description of the project and the discussions I've had with cwalters, it sounds like it could be a good fit as part of our provisioning system for disposable clients. If we're serious about proposing this as a GSoC project, we should probably explore it a bit more to be certain that we'd want it now but I figured it was worth putting on the list. [2] https://github.com/cgwalters/rpm-ostree System for apparent results storage and modification There has to be a better title for this but it would be one of the last major steps in enabling bodhi/koji to block builds/updates on check failures. The idea would be to provide an interface which can decide whether a build/update is OK based on what checks were passed/failed. It would have a mechanism for manual overrides and algorithmic overrides (ie, we know that foo has problem X and are working on it, ignore failures for now) so that we don't upset packagers more than we need to. When Josef and I last talked about this, we weren't sure that putting this functionality into our results storage mechanism was wise. It's a different concern that has the potential to make a mess out of the results storage. signature.asc Description: PGP signature ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
