Re: Thoughts about Travis-CI integration
На 18.12.2013 20:09, Tim Flink написа: On a side note, it might be interesting to find out what percentage of packages are running things in %check. I don't know what we would do with that metric, but I think it would be interesting :) Hi Tim, I did a quick measurement and the number of packages which execute something in %check is around 30%. 35% is the number of packages which appear to have a test suite at all. More details here: http://atodorov.org/blog/2013/12/24/upstream-test-suite-status-of-fedora-20/ Happy holidays, Alex -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Thoughts about Travis-CI integration
Well, I think travis can just be a supplemental part of %check, like we can run it after package building, and paste its results as "for reference only". -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Thoughts about Travis-CI integration
На 18.12.2013 20:09, Tim Flink написа: On a side note, it might be interesting to find out what percentage of packages are running things in %check. I don't know what we would do with that metric, but I think it would be interesting :) I have something in the works which can be modified to extract this info. It should be fairly easy to make a good guesstimate of the presence/non presence of test suites as well. I think I'll start from here. Does Fedora have its own CI infrastructure coupled with Koji ? That depends on what you consider CI to be, I guess. I consider both autoqa and taskotron to be forms of CI but not the exact same thing as travis or jenkins. Both autoqa and taskotron are designed to run various things based on a package's state in either koji or bodhi. Thanks, I have to look into these. I gave Travis as example because it is well known and because somebody else decided to make use of it in python-bugzilla. I'm still unclear on what you're looking to do. Are you talking about looking for test suites in package upstreams and running those tests, regardless of whether they're run in %check during build? Are you talking about creating and maintaining an out-of-band set of unit tests for packages without an upstream unit test suite? Are you talking about creating a set of package-specific functional/integration test suites that are run when packages are built? Pretty much all of these in the long term but I wanted to get a general feeling so I can concentrate the effort on something specific. * For packages which have upstream test suites (e.g. parted) - contribute more tests where necessary; Ideally covering bugs reported against Fedora or RHEL for which there isn't a test case. * For packages without upstream test suite my preference is to create one and contribute it back upstream. If upstream doesn't accept it and the package is critical enough maybe maintain that in the Fedora branch. * Functional/integration test suites (e.g. using Beaker tasks) are less priority goal for the moment but also need to be considered. For some packages these are more suitable (e.g. anaconda), for others maybe not (e.g. pykickstart seems to have fairly good amount of unit tests). Whatever the test suite(s) come out to be we need an environment(or environments) where to execute them and a method of starting the suite. I can see how some test cases are part of %check and can halt the build process and others are started async after the build and are left for Fedora QA and devel to investigate. In the case of Travis, test runs are triggered by git commits and Travis is not connected to any build infrastructure from what I know. It provides the results to whoever is interested in them. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Thoughts about Travis-CI integration
On Wed, 18 Dec 2013 17:52:59 +0200 Alexander Todorov wrote: > Hi folks, > this message sparked an interest: > https://lists.fedorahosted.org/pipermail/python-bugzilla/2013-December/000200.html > > In addition to that I've inspected around 30 packages which seem to > be missing an upstream test suite (a few have one but it is not > automatically executed in %check section in the spec file). > > I'm pretty sure many more packages are like this, I just didn't have > the time to investigate all several thousands of them. On a side note, it might be interesting to find out what percentage of packages are running things in %check. I don't know what we would do with that metric, but I think it would be interesting :) > My idea is simple - starting after the holidays to call for help in > writing test suites (or more test cases) for packages. This can be > coupled with settings to execute them in Travis CI or another CI > system of choice. > > > > My questions are: > > * What is the general feeling of using Travis CI in Fedora? It is > well established in Ruby and Python circles but I know we like to > keep dependency on external services to minumum. I've not used it personally, so I can't speak much about Travis in particular. Depending on what you're looking to do, I would hesitate to add yet another CI system to Fedora - infra already has a jenkins instance set up and qa-devel is starting to use buildbot for both CI and as a part of taskotron (the project formerly known as taskbot). > Does Fedora have its own CI infrastructure coupled with Koji ? That depends on what you consider CI to be, I guess. I consider both autoqa and taskotron to be forms of CI but not the exact same thing as travis or jenkins. Both autoqa and taskotron are designed to run various things based on a package's state in either koji or bodhi. > Maybe deploy our own instance or contribute to Travis with a pool of > systems sponsored by Fedora? > > What to do with packages whose test suite is not suitable to be > executed during build (e.g. due to requirements or limitations on the > build servers) ? > > What's your take ? > > ( Adding Tim Flink to CC to answer from the infrastructure side. ) I'm still unclear on what you're looking to do. Are you talking about looking for test suites in package upstreams and running those tests, regardless of whether they're run in %check during build? Are you talking about creating and maintaining an out-of-band set of unit tests for packages without an upstream unit test suite? Are you talking about creating a set of package-specific functional/integration test suites that are run when packages are built? > * Are there any volunteers to join me in planning and coordinating > this project? We need to somehow prioritize which packages need > inspection and working on, count the available test cases, report > bugs if missing, produce patches, etc. It will be a long run one and > needs lots of work just because the great number of packages. > > > * Who else should I be talking to ? Kevin already mentioned this, but definitely infra. Tim signature.asc Description: PGP signature -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Thoughts about Travis-CI integration
On Wed, 18 Dec 2013 17:52:59 +0200 Alexander Todorov wrote: > Hi folks, > this message sparked an interest: > https://lists.fedorahosted.org/pipermail/python-bugzilla/2013-December/000200.html > > In addition to that I've inspected around 30 packages which seem to > be missing an upstream test suite (a few have one but it is not > automatically executed in %check section in the spec file). > > I'm pretty sure many more packages are like this, I just didn't have > the time to investigate all several thousands of them. > > My idea is simple - starting after the holidays to call for help in > writing test suites (or more test cases) for packages. This can be > coupled with settings to execute them in Travis CI or another CI > system of choice. Sounds interesting. We couldn't run travis in %check... so that would be something run async somewhere? I think adding %checks would be longer term more handy, but would also be more work (you would involve upstreams of projects and work with them to create checks,etc) > My questions are: > > * What is the general feeling of using Travis CI in Fedora? It is > well established in Ruby and Python circles but I know we like to > keep dependency on external services to minumum. I guess that somewhat depends on whats done with the information. Would a failure there block updating a Fedora package? Or would it simply be informational? Would we then contribute the travis info to upstream to use, or this is tied to the Fedora package version? > Does Fedora have its own CI infrastructure coupled with Koji ? No. > Maybe deploy our own instance or contribute to Travis with a pool of > systems sponsored by Fedora? In order for us to deploy it, it would need to be packaged up (looks like it's a lot of moving parts) and you would need to get enough folks who know ruby/jruby/etc to commit to maintain it long term. > What to do with packages whose test suite is not suitable to be > executed during build (e.g. due to requirements or limitations on the > build servers) ? Note also that builds allow no network, so you can't depend on remote resources. There's various ways to test things even so, depending on what you are testing. > What's your take ? I think more testing at this level is great, but might be better to look at working with upstreams to add tests that could run from %check... > ( Adding Tim Flink to CC to answer from the infrastructure side. ) > > > * Are there any volunteers to join me in planning and coordinating > this project? We need to somehow prioritize which packages need > inspection and working on, count the available test cases, report > bugs if missing, produce patches, etc. It will be a long run one and > needs lots of work just because the great number of packages. > > > * Who else should I be talking to ? If you want to involve fedora infrastructure, the infrastructure list might be good... kevin signature.asc Description: PGP signature -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Thoughts about Travis-CI integration
On mið 18.des 2013 15:52, Alexander Todorov wrote: * Are there any volunteers to join me in planning and coordinating this project? We need to somehow prioritize which packages need inspection and working on, count the available test cases, report bugs if missing, produce patches, etc. It will be a long run one and needs lots of work just because the great number of packages. Count me it inn JBG -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
