Bug#732835: Provide build tools with more information
On Tue, Dec 24, 2013 at 01:18:46AM +0100, Guillem Jover wrote: On Mon, 2013-12-23 at 11:36:19 +0100, Guido Günther wrote: [..snip..] The current unstructured output of dpkg-buildpackage leaves us only with parsing the logs and looking at the exit status. Since the exit status seems to be that of the invoked command mostly (except for dpkg-checkbuilddeps) we can't infere what failed. So there are two parts: 1.) progress report (which step are we currently at) 2.) error report (which step failed and why) While 2.) can be fixed via more detailed exit codes 1.) can't. Ok, I'll take a look at this while I'm finishing up the dpkg-buildpackage rework for 1.17.6, as at least the first is trivial now that the required foundation for the shell hooks is there. Great! For example for 1.17.6 I'm adding hooks support, which can be useful for wrappers. Hooks are nice and can be useful but they can also be confusing: gbp (hooks) - pbuilder (hooks) - dpkg-buildpackage (hooks) If dpkg-buildpackage doesn't want to suck in the functionality of the other two a nice way to propagate information like progress, errors, build results up the chain would be really nice. Just to be clear, I was not suggesting to use (shell) hooks as a replacement for something like a --status-fd, that might end up being very cumbersome, just an example of things that a frontend/wrapper might need that are only possible if supported by the tool. OTOH something else that I could envision are perl hooks, possibly in addition to the shell ones, so that other tools could be implemented as modules extending dpkg-buildpackage functionality, instead of programs wrapping it. And one could, for example, do something like: $ dpkg-buildpackage --modules schroot,git And get the Dpkg::*::Schroot.pm and Dpkg::*::Git.pm modules loaded and specific hooks run with enough config and information, at the key stage points. I see. This would move dpkg-buildpackage up the stack and turn it into kind of a frontend by itself. This might work out o.k. although I'm quiet happy with dpkg-buildpackage being a bit minimalistic (and therefore very reliable) like it's atm. I wonder if this could hinder dpkg-buildpackage's development speed since we'd have to care about API stability to lots of lots of modules as well? This report is triggered by #732678 where gbp failed to find the generated changes file for a architecture independent package build since it didn't look at the options passed to dpkg-buildpackage until recently. For the problem described in the bug report, I think a better way to solve this is to run lintian directly from dpkg-buildpackage, which will be possible with dpkg 1.17.6, by using the new check-command support. See the following commit for further details: http://anonscm.debian.org/gitweb/?p=dpkg/dpkg.git;a=commitdiff;h=1cef5694 Or do you need the .changes file for something else? (Rahpael pointed this out already) If one may want the build environment to be as minimalistic as possible and not want to have lintian or other verifiers in there. We can also use the changes file to find out about the other build artifacts to upload them to a temp repository or some such. So invoking lintian is not the only usage and Right. I'd rather not see every usage scenario crammed into dpkg-buildpackage itself. Oh, me neither (at least not directly), I was just trying to gather what you had in mind to see how that could be best placed in dpkg-buildpackage (or other tools). Thank you for your feedback! Cheers, -- Guido -- To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#732835: Provide build tools with more information
Hi, On Mon, Dec 23, 2013 at 05:23:42AM +0100, Guillem Jover wrote: Hi! On Sun, 2013-12-22 at 10:18:32 +0100, Guido Günther wrote: I't be nice if dpkg-buildpackage would report build information (e.g. things like the name of the generated changes file) to build-tools invoking it. Doing this via a status fd would be nice[1] since we could then implement more nice things like progress information (we'd then be able to detect easily which dpkg-buildpackage's steps failed) without parsing the full build output. [1] The status fd has the disadvantage that we need to pass it through tools like pbuilder and sbuild as well so a file at a well known location might be simpler. How do you see that progress information being used? Or to report what? The progress information could be useful on the buildds and in CI systems like e.g. Jenkins. It would allow us to get a better idea at what stage we're currently at (like the steps 1-9 from dpkg-buildpackage's manapage). It will also help us to report more detailed what step failed so we can see at a glance if we had unmet build-deps or if the binary build failed. I'm not saying a --status-fd kind of option might not be useful, just interested to know, if maybe there's something else that needs fixing instead in dpkg-buildpackage, now that I'm reworking it. For example if its error reporting leaves to be desired, then I'd rather improve that, rather than adding support for wrappers to workaround it. :) The current unstructured output of dpkg-buildpackage leaves us only with parsing the logs and looking at the exit status. Since the exit status seems to be that of the invoked command mostly (except for dpkg-checkbuilddeps) we can't infere what failed. So there are two parts: 1.) progress report (which step are we currently at) 2.) error report (which step failed and why) While 2.) can be fixed via more detailed exit codes 1.) can't. For example for 1.17.6 I'm adding hooks support, which can be useful for wrappers. Hooks are nice and can be useful but they can also be confusing: gbp (hooks) - pbuilder (hooks) - dpkg-buildpackage (hooks) If dpkg-buildpackage doesn't want to suck in the functionality of the other two a nice way to propagate information like progress, errors, build results up the chain would be really nice. This report is triggered by #732678 where gbp failed to find the generated changes file for a architecture independent package build since it didn't look at the options passed to dpkg-buildpackage until recently. For the problem described in the bug report, I think a better way to solve this is to run lintian directly from dpkg-buildpackage, which will be possible with dpkg 1.17.6, by using the new check-command support. See the following commit for further details: http://anonscm.debian.org/gitweb/?p=dpkg/dpkg.git;a=commitdiff;h=1cef5694 Or do you need the .changes file for something else? (Rahpael pointed this out already) If one may want the build environment to be as minimalistic as possible and not want to have lintian or other verifiers in there. We can also use the changes file to find out about the other build artifacts to upload them to a temp repository or some such. So invoking lintian is not the only usage and I'd rather not see every usage scenario crammed into dpkg-buildpackage itself. Cheers and thanks for your quick response, -- Guido -- To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#732835: Provide build tools with more information
On Mon, 2013-12-23 at 11:36:19 +0100, Guido Günther wrote: On Mon, Dec 23, 2013 at 05:23:42AM +0100, Guillem Jover wrote: On Sun, 2013-12-22 at 10:18:32 +0100, Guido Günther wrote: I't be nice if dpkg-buildpackage would report build information (e.g. things like the name of the generated changes file) to build-tools invoking it. Doing this via a status fd would be nice[1] since we could then implement more nice things like progress information (we'd then be able to detect easily which dpkg-buildpackage's steps failed) without parsing the full build output. [1] The status fd has the disadvantage that we need to pass it through tools like pbuilder and sbuild as well so a file at a well known location might be simpler. How do you see that progress information being used? Or to report what? The progress information could be useful on the buildds and in CI systems like e.g. Jenkins. It would allow us to get a better idea at what stage we're currently at (like the steps 1-9 from dpkg-buildpackage's manapage). It will also help us to report more detailed what step failed so we can see at a glance if we had unmet build-deps or if the binary build failed. Sure. I'm not saying a --status-fd kind of option might not be useful, just interested to know, if maybe there's something else that needs fixing instead in dpkg-buildpackage, now that I'm reworking it. For example if its error reporting leaves to be desired, then I'd rather improve that, rather than adding support for wrappers to workaround it. :) The current unstructured output of dpkg-buildpackage leaves us only with parsing the logs and looking at the exit status. Since the exit status seems to be that of the invoked command mostly (except for dpkg-checkbuilddeps) we can't infere what failed. So there are two parts: 1.) progress report (which step are we currently at) 2.) error report (which step failed and why) While 2.) can be fixed via more detailed exit codes 1.) can't. Ok, I'll take a look at this while I'm finishing up the dpkg-buildpackage rework for 1.17.6, as at least the first is trivial now that the required foundation for the shell hooks is there. For example for 1.17.6 I'm adding hooks support, which can be useful for wrappers. Hooks are nice and can be useful but they can also be confusing: gbp (hooks) - pbuilder (hooks) - dpkg-buildpackage (hooks) If dpkg-buildpackage doesn't want to suck in the functionality of the other two a nice way to propagate information like progress, errors, build results up the chain would be really nice. Just to be clear, I was not suggesting to use (shell) hooks as a replacement for something like a --status-fd, that might end up being very cumbersome, just an example of things that a frontend/wrapper might need that are only possible if supported by the tool. OTOH something else that I could envision are perl hooks, possibly in addition to the shell ones, so that other tools could be implemented as modules extending dpkg-buildpackage functionality, instead of programs wrapping it. And one could, for example, do something like: $ dpkg-buildpackage --modules schroot,git And get the Dpkg::*::Schroot.pm and Dpkg::*::Git.pm modules loaded and specific hooks run with enough config and information, at the key stage points. This report is triggered by #732678 where gbp failed to find the generated changes file for a architecture independent package build since it didn't look at the options passed to dpkg-buildpackage until recently. For the problem described in the bug report, I think a better way to solve this is to run lintian directly from dpkg-buildpackage, which will be possible with dpkg 1.17.6, by using the new check-command support. See the following commit for further details: http://anonscm.debian.org/gitweb/?p=dpkg/dpkg.git;a=commitdiff;h=1cef5694 Or do you need the .changes file for something else? (Rahpael pointed this out already) If one may want the build environment to be as minimalistic as possible and not want to have lintian or other verifiers in there. We can also use the changes file to find out about the other build artifacts to upload them to a temp repository or some such. So invoking lintian is not the only usage and Right. I'd rather not see every usage scenario crammed into dpkg-buildpackage itself. Oh, me neither (at least not directly), I was just trying to gather what you had in mind to see how that could be best placed in dpkg-buildpackage (or other tools). Thanks, Guillem -- To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#732835: Provide build tools with more information
Package: dpkg-dev Version: 1.17.1 Severity: wishlist Hi, I't be nice if dpkg-buildpackage would report build information (e.g. things like the name of the generated changes file) to build-tools invoking it. Doing this via a status fd would be nice[1] since we could then implement more nice things like progress information (we'd then be able to detect easily which dpkg-buildpackage's steps failed) without parsing the full build output. This report is triggered by #732678 where gbp failed to find the generated changes file for a architecture independent package build since it didn't look at the options passed to dpkg-buildpackage until recently. Cheers, -- Guido [1] The status fd has the disadvantage that we need to pass it through tools like pbuilder and sbuild as well so a file at a well known location might be simpler. -- To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#732835: Provide build tools with more information
Hi! On Sun, 2013-12-22 at 10:18:32 +0100, Guido Günther wrote: I't be nice if dpkg-buildpackage would report build information (e.g. things like the name of the generated changes file) to build-tools invoking it. Doing this via a status fd would be nice[1] since we could then implement more nice things like progress information (we'd then be able to detect easily which dpkg-buildpackage's steps failed) without parsing the full build output. [1] The status fd has the disadvantage that we need to pass it through tools like pbuilder and sbuild as well so a file at a well known location might be simpler. How do you see that progress information being used? Or to report what? I'm not saying a --status-fd kind of option might not be useful, just interested to know, if maybe there's something else that needs fixing instead in dpkg-buildpackage, now that I'm reworking it. For example if its error reporting leaves to be desired, then I'd rather improve that, rather than adding support for wrappers to workaround it. :) For example for 1.17.6 I'm adding hooks support, which can be useful for wrappers. This report is triggered by #732678 where gbp failed to find the generated changes file for a architecture independent package build since it didn't look at the options passed to dpkg-buildpackage until recently. For the problem described in the bug report, I think a better way to solve this is to run lintian directly from dpkg-buildpackage, which will be possible with dpkg 1.17.6, by using the new check-command support. See the following commit for further details: http://anonscm.debian.org/gitweb/?p=dpkg/dpkg.git;a=commitdiff;h=1cef5694 Or do you need the .changes file for something else? Thanks, Guillem -- To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#732835: Provide build tools with more information
On Mon, 23 Dec 2013, Guillem Jover wrote: Doing this via a status fd would be nice[1] since we could then implement more nice things like progress information (we'd then be able to detect easily which dpkg-buildpackage's steps failed) without parsing the full build output. [1] The status fd has the disadvantage that we need to pass it through tools like pbuilder and sbuild as well so a file at a well known location might be simpler. How do you see that progress information being used? Or to report what? Being able to programmatically detect in what step the failure happened is cleary useful, in particular for mass rebuilds. Differentiating failures in the source rebuild or the binary rebuild is already significant. I'm not saying a --status-fd kind of option might not be useful, just interested to know, if maybe there's something else that needs fixing instead in dpkg-buildpackage, now that I'm reworking it. For example if its error reporting leaves to be desired, then I'd rather improve that, rather than adding support for wrappers to workaround it. :) For example for 1.17.6 I'm adding hooks support, which can be useful for wrappers. Hooks are certainly useful but when dpkg-buildpackage is really wrapped in something much larger then it's somewhat counter-intuitive to have to include all the post-build processing in the final dpkg-buildpackage hook. I really believe that we ought to communicate back the generated changes file. For the problem described in the bug report, I think a better way to solve this is to run lintian directly from dpkg-buildpackage, which will be possible with dpkg 1.17.6, by using the new check-command support. See the following commit for further details: This is not so clear cut. Letting dpkg-buildpackage call lintian means that lintian must be installed in the build environment and that might not be the case in minimal build environment. You might want to call lintian outside of it (and thus outside of dpkg-buildpackage). Or do you need the .changes file for something else? There a lots of things that you might want to do with a built package besides checking it: installing it for tests, uploading it, installing it in a repository, modifying it (in the case of dgit), etc. Cheers, -- Raphaël Hertzog ◈ Debian Developer Discover the Debian Administrator's Handbook: → http://debian-handbook.info/get/ -- To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
