Re: (re)introducing - fedora-review - tool to help with package reviews
Excerpts from Brendan Jones's message of Fri Dec 16 00:32:24 +0100 2011: On 12/15/2011 09:57 PM, Brendan Jones wrote: On 11/21/2011 02:14 PM, Stanislav Ochotnicky wrote: Hello fellow devs, I am sure quite a few of you have done some reviews and thought Hey, a,b,c and d could be automated. For E I could use some more information that can be automatically gathered. Some of you even wrote your own tools to do some of these things. Hi Stan, great idea. Will try to use this prior to any forthcoming reviews. I find the most time consuming task in the review process is the license check. I use a combination of find/head/grep commands to try and determine if some of the source files have differing licenses to the stated one in the spec. None of my methods guarantee 100% license detection, given the sheer number of licenses out there, although if we could consolidate all of the methods reviewers use for this we would have a nifty tool indeed. Not sure if this is something which should be part of this package or another entirely? regards, Brendan The guys on the packaging list enlightened me on the existence of licensecheck from rpmdevtools. From my brief tests it does a good job but have not used it against cornercases Hi, I planned to add running of licensecheck already, so now I created a feature request in our trac[1]. We certainly will not reimplement it. We will use it though :-) I can tell you right now that cornercases will never be caught with tools like this. Licensecheck only looks at headers/comments, whereas licensing depends on many things and can be quite confusing. I am sure that rpmdevtools maintainers would be happy to accept improvements though. [1] https://fedorahosted.org/FedoraReview/ticket/22 -- Stanislav Ochotnicky sochotni...@redhat.com Software Engineer - Base Operating Systems Brno PGP: 7B087241 Red Hat Inc. http://cz.redhat.com signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: (re)introducing - fedora-review - tool to help with package reviews
On 2011-12-19 10:32, Stanislav Ochotnicky wrote: I can tell you right now that cornercases will never be caught with tools like this. Licensecheck only looks at headers/comments, whereas licensing depends on many things and can be quite confusing. I am sure that rpmdevtools maintainers would be happy to accept improvements though. Actually I'd be happier if patches were sent to licensecheck upstream which is Debian's devscripts package/team. http://packages.qa.debian.org/d/devscripts.html -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: (re)introducing - fedora-review - tool to help with package reviews
On 16.12.2011 00:32, Brendan Jones wrote: Not sure if this is something which should be part of this package or another entirely? Sure, we will reinvent the wheel yet again (see http://www.fossology.org/ ... yes it would probably require some fedora-wide server, or maybe not, I don't know enough about it). The guys on the packaging list enlightened me on the existence of licensecheck from rpmdevtools. From my brief tests it does a good job but have not used it against cornercases Oh, we do already? :( Matěj -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: (re)introducing - fedora-review - tool to help with package reviews
On 11/21/2011 02:14 PM, Stanislav Ochotnicky wrote: Hello fellow devs, I am sure quite a few of you have done some reviews and thought Hey, a,b,c and d could be automated. For E I could use some more information that can be automatically gathered. Some of you even wrote your own tools to do some of these things. Hi Stan, great idea. Will try to use this prior to any forthcoming reviews. I find the most time consuming task in the review process is the license check. I use a combination of find/head/grep commands to try and determine if some of the source files have differing licenses to the stated one in the spec. None of my methods guarantee 100% license detection, given the sheer number of licenses out there, although if we could consolidate all of the methods reviewers use for this we would have a nifty tool indeed. Not sure if this is something which should be part of this package or another entirely? regards, Brendan -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: (re)introducing - fedora-review - tool to help with package reviews
On 12/15/2011 09:57 PM, Brendan Jones wrote: On 11/21/2011 02:14 PM, Stanislav Ochotnicky wrote: Hello fellow devs, I am sure quite a few of you have done some reviews and thought Hey, a,b,c and d could be automated. For E I could use some more information that can be automatically gathered. Some of you even wrote your own tools to do some of these things. Hi Stan, great idea. Will try to use this prior to any forthcoming reviews. I find the most time consuming task in the review process is the license check. I use a combination of find/head/grep commands to try and determine if some of the source files have differing licenses to the stated one in the spec. None of my methods guarantee 100% license detection, given the sheer number of licenses out there, although if we could consolidate all of the methods reviewers use for this we would have a nifty tool indeed. Not sure if this is something which should be part of this package or another entirely? regards, Brendan The guys on the packaging list enlightened me on the existence of licensecheck from rpmdevtools. From my brief tests it does a good job but have not used it against cornercases -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: (re)introducing - fedora-review - tool to help with package reviews
On 11/21/2011 02:14 PM, Stanislav Ochotnicky wrote: - Any ideas, bugreports etc will be much appreciated I would be very interested in feature, which instead of mock, will run Koji scratch build and then will download resulting packages from Koji. -- Miroslav Suchy Red Hat Satellite Engineering -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: (re)introducing - fedora-review - tool to help with package reviews
On Mon, 2011-12-05 at 17:06 +0100, Miroslav Suchý wrote: On 11/21/2011 02:14 PM, Stanislav Ochotnicky wrote: - Any ideas, bugreports etc will be much appreciated I would be very interested in feature, which instead of mock, will run Koji scratch build and then will download resulting packages from Koji. I have been thinking about this but as an option which would not be the default (imho). Could you open a ticket on the trac ? Thanks, Pierre -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
(re)introducing - fedora-review - tool to help with package reviews
Hello fellow devs, I am sure quite a few of you have done some reviews and thought Hey, a,b,c and d could be automated. For E I could use some more information that can be automatically gathered. Some of you even wrote your own tools to do some of these things. Yet there is no unified tool, nor format for package reviews. There are more reasons, but I guess biggest one is there are just too many guidelines and there is probably no one who knows all of them. So few of us got together and hopefully created something that can be used by everyone. fedora-review[1] is now in updates-testing. It provides several checks: 66 generic tests (licensing, md5sum sources, bundling, etc.) 9 c/c++ specific checks (static libs, ldconfig, headers, rpath etc.) 13 java specific checks (javadoc, depmaps, jpackage-utils reqs) 8 R specific checks There are still many more checks waiting to be written. I'd like to see Perl, Python and Ruby checks, though I am not *that* familiar with their guidelines. I think the important thing here is that checks can be written in basically any language. We have simple JSON api[2] using stdin/stdout for communication. There is an example external plugin in documentation in perl and shell (though that's just mock-check). Our goal is for each language SIG (or other specific package group) to maintain their own checks together with their guidelines. * How you can help * - Tell us what checks are missing - Tell us if you think checks are doing it wrong - Create new checks (in language of your choice - we have JSON api) - do this even if the test cannot be automated. At least it will appear on the checklist for your packages! - Any ideas, bugreports etc will be much appreciated [1] https://fedorahosted.org/FedoraReview [2] https://fedorahosted.org/FedoraReview/browser/api/README -- Stanislav Ochotnicky sochotni...@redhat.com Software Engineer - Base Operating Systems Brno PGP: 7B087241 Red Hat Inc. http://cz.redhat.com signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: (re)introducing - fedora-review - tool to help with package reviews
On 11/21/2011 01:14 PM, Stanislav Ochotnicky wrote: Hello fellow devs, I am sure quite a few of you have done some reviews and thought Hey, a,b,c and d could be automated. For E I could use some more information that can be automatically gathered. Some of you even wrote your own tools to do some of these things. Yet there is no unified tool, nor format for package reviews. There are more reasons, but I guess biggest one is there are just too many guidelines and there is probably no one who knows all of them. So few of us got together and hopefully created something that can be used by everyone. fedora-review[1] is now in updates-testing. It provides several checks: 66 generic tests (licensing, md5sum sources, bundling, etc.) 9 c/c++ specific checks (static libs, ldconfig, headers, rpath etc.) 13 java specific checks (javadoc, depmaps, jpackage-utils reqs) 8 R specific checks There are still many more checks waiting to be written. I'd like to see Perl, Python and Ruby checks, though I am not *that* familiar with their guidelines. I think the important thing here is that checks can be written in basically any language. We have simple JSON api[2] using stdin/stdout for communication. There is an example external plugin in documentation in perl and shell (though that's just mock-check). Our goal is for each language SIG (or other specific package group) to maintain their own checks together with their guidelines. * How you can help * - Tell us what checks are missing - Tell us if you think checks are doing it wrong - Create new checks (in language of your choice - we have JSON api) - do this even if the test cannot be automated. At least it will appear on the checklist for your packages! - Any ideas, bugreports etc will be much appreciated [1] https://fedorahosted.org/FedoraReview [2] https://fedorahosted.org/FedoraReview/browser/api/README Is this not something that releng/autoqa could use as well as in run against all already existing specs and automatically file bugs if they aren't ( and to keep things ) up to guidelines? JBG -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: (re)introducing - fedora-review - tool to help with package reviews
On Mon, 2011-11-21 at 13:25 +, Jóhann B. Guðmundsson wrote: On 11/21/2011 01:14 PM, Stanislav Ochotnicky wrote: Hello fellow devs, I am sure quite a few of you have done some reviews and thought Hey, a,b,c and d could be automated. For E I could use some more information that can be automatically gathered. Some of you even wrote your own tools to do some of these things. Yet there is no unified tool, nor format for package reviews. There are more reasons, but I guess biggest one is there are just too many guidelines and there is probably no one who knows all of them. So few of us got together and hopefully created something that can be used by everyone. fedora-review[1] is now in updates-testing. It provides several checks: 66 generic tests (licensing, md5sum sources, bundling, etc.) 9 c/c++ specific checks (static libs, ldconfig, headers, rpath etc.) 13 java specific checks (javadoc, depmaps, jpackage-utils reqs) 8 R specific checks There are still many more checks waiting to be written. I'd like to see Perl, Python and Ruby checks, though I am not *that* familiar with their guidelines. I think the important thing here is that checks can be written in basically any language. We have simple JSON api[2] using stdin/stdout for communication. There is an example external plugin in documentation in perl and shell (though that's just mock-check). Our goal is for each language SIG (or other specific package group) to maintain their own checks together with their guidelines. * How you can help * - Tell us what checks are missing - Tell us if you think checks are doing it wrong - Create new checks (in language of your choice - we have JSON api) - do this even if the test cannot be automated. At least it will appear on the checklist for your packages! - Any ideas, bugreports etc will be much appreciated [1] https://fedorahosted.org/FedoraReview [2] https://fedorahosted.org/FedoraReview/browser/api/README Is this not something that releng/autoqa could use as well as in run against all already existing specs and automatically file bugs if they aren't ( and to keep things ) up to guidelines? As more and more extended tests become available one could think of this. However there are a number of cases where packages go against guideline (especially the rpmlint output must be clean)* for valid reasons. So running blindly the tool without a human look to the output to fill bug reports might not be such a good idea. Pierre * Ok, here we could use the integration between fedpkg lint and the .rpmlint file in the repo -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: (re)introducing - fedora-review - tool to help with package reviews
Excerpts from Jóhann B. Guðmundsson's message of Mon Nov 21 14:25:22 +0100 2011: [1] https://fedorahosted.org/FedoraReview [2] https://fedorahosted.org/FedoraReview/browser/api/README Is this not something that releng/autoqa could use as well as in run against all already existing specs and automatically file bugs if they aren't ( and to keep things ) up to guidelines? I wouldn't be against it, but I can imagine all the shouting on the bugzilla and mailing lists this would cause(i.e. But my package is working!!). Plus output of our tool is more verbose than rpmlint (thought it does more as well). There will always be tests that cannot be automated for one reason or the other. In cases like that we have ways to add helpful information to the template (for example output of licensecheck run on all files in tarball could be added to licensing part). This helpful output would be considered noise for a lot of packagers I guess. And there will always be false positives. I would hope none of our checks will have false negative (i.e. check will report A-OK, but the guidelines would be broken). All that said: If our QA/releng guys find some part of fedora-review needs some tweaks to be usable for tasks they have to do: file issues in our trac and we'll do our best. -- Stanislav Ochotnicky sochotni...@redhat.com Software Engineer - Base Operating Systems Brno PGP: 7B087241 Red Hat Inc. http://cz.redhat.com signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: (re)introducing - fedora-review - tool to help with package reviews
Well there may be a chance this tool may eventually become officially adopted by QA after it gets tested and used long enough to consider it safe/stable. On Mon, Nov 21, 2011 at 7:48 AM, Stanislav Ochotnicky sochotni...@redhat.com wrote: Excerpts from Jóhann B. Guðmundsson's message of Mon Nov 21 14:25:22 +0100 2011: [1] https://fedorahosted.org/FedoraReview [2] https://fedorahosted.org/FedoraReview/browser/api/README Is this not something that releng/autoqa could use as well as in run against all already existing specs and automatically file bugs if they aren't ( and to keep things ) up to guidelines? I wouldn't be against it, but I can imagine all the shouting on the bugzilla and mailing lists this would cause(i.e. But my package is working!!). Plus output of our tool is more verbose than rpmlint (thought it does more as well). There will always be tests that cannot be automated for one reason or the other. In cases like that we have ways to add helpful information to the template (for example output of licensecheck run on all files in tarball could be added to licensing part). This helpful output would be considered noise for a lot of packagers I guess. And there will always be false positives. I would hope none of our checks will have false negative (i.e. check will report A-OK, but the guidelines would be broken). All that said: If our QA/releng guys find some part of fedora-review needs some tweaks to be usable for tasks they have to do: file issues in our trac and we'll do our best. -- Stanislav Ochotnicky sochotni...@redhat.com Software Engineer - Base Operating Systems Brno PGP: 7B087241 Red Hat Inc. http://cz.redhat.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
