Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Peter Volkov wrote: > This means that every ebuild which uses 'unpack ${A}' should have in > DEPEND a program which is selected by filename extension of sources. So > as I understood your initial suggestion was to make this happen > automatically. And this is very logical as makes ebuilds cleaner and > more terse. So why did you changed your mind and try to write another > eclass (which then should sit in the tree forever), to create duplicate > unpack function instead of just making step you suggested initially? Is > there any intension to remove unpack logic from package manager? And if > yes, why? ++ I also was wondering this. And if we add "unpack2()", which could be a little hard to explain in the documentation, it will need to be there at least until ebuilds stop using it (when portage gets this functionality ultimately). For my uses, I'd rather do deps manually and use the official portage unpack() until portage has such features in order to keep my ebuilds looking a bit more clean. -Joe
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Sun, 20 Jul 2008 17:41:58 +0400 Peter Volkov <[EMAIL PROTECTED]> wrote: > В Чтв, 17/07/2008 в 04:51 +0200, Marius Mauch пишет: > > At dev.gentoo.org/~genone/unpack.eclass is the draft for an eclass > > to implement this feature. > > Marius, although it's possible to do this things in eclass why is > eclass better? As I see portage's unpack() already has extension <-> > "program to unpack" relation. Basically unpack() in ebuild.sh has the > following code: > > unpack() { > [snip] > case "${x##*.}" in > tar) > tar xof "${srcdir}${x}" ${tar_opts} || die "$myfail" > ;; > tgz) > tar xozf "${srcdir}${x}" ${tar_opts} || die "$myfail" > [snip and so on...] > > This means that every ebuild which uses 'unpack ${A}' should have in > DEPEND a program which is selected by filename extension of sources. > So as I understood your initial suggestion was to make this happen > automatically. And this is very logical as makes ebuilds cleaner and > more terse. So why did you changed your mind and try to write another > eclass (which then should sit in the tree forever), to create > duplicate unpack function instead of just making step you suggested > initially? Is there any intension to remove unpack logic from package > manager? And if yes, why? The eclass is provided as a proof-of-concept implementation that can be used without adding additional infrastructure and implementing and defining a new EAPI version. PM-based solutions would be nicer long-term, but are also more tricky to "get right" (mainly a question of where to inject the deps). Also the eclass approach has the benefit that the whole unpack logic can be maintained in one location vs. being split between multiple places in the package manager and the tree, and is easier to extend (no need for an EAPI change just to add a new unpack format). And as you ask, there have been plans to move quite a bit of the bash code from portage into the tree (transparently to ebuilds though) as that's the better place to maintain many helper functions like do* or new* that are only used by ebuilds/eclasses and not portage. But those plans have already existed for years (long before EAPI existed) without anything happening, and they've never been very specific wrt what exactly could be moved. Even with a PM-based solution it might be useful, depending on the actual implementation, to have the unpack() funtion in the tree, for the reasons outlined in the first paragraph. Marius
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
В Чтв, 17/07/2008 в 04:51 +0200, Marius Mauch пишет: > At dev.gentoo.org/~genone/unpack.eclass is the draft for an eclass > to implement this feature. Marius, although it's possible to do this things in eclass why is eclass better? As I see portage's unpack() already has extension <-> "program to unpack" relation. Basically unpack() in ebuild.sh has the following code: unpack() { [snip] case "${x##*.}" in tar) tar xof "${srcdir}${x}" ${tar_opts} || die "$myfail" ;; tgz) tar xozf "${srcdir}${x}" ${tar_opts} || die "$myfail" [snip and so on...] This means that every ebuild which uses 'unpack ${A}' should have in DEPEND a program which is selected by filename extension of sources. So as I understood your initial suggestion was to make this happen automatically. And this is very logical as makes ebuilds cleaner and more terse. So why did you changed your mind and try to write another eclass (which then should sit in the tree forever), to create duplicate unpack function instead of just making step you suggested initially? Is there any intension to remove unpack logic from package manager? And if yes, why? -- Peter.
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Marius Mauch kirjoitti: If someone wants to use it I can add it on the tree (after the normal review process and being better tested), but I'll only be doing it when there is an actual demand for it (no point in adding an eclass that nobody uses). I have been long thinking about adding support for .zip handling to java eclasses so let's get this in and it will get a user from java-pkg-2 and java-pkg-opt-2 eclasses at least. Regards, Petteri signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Thu, 17 Jul 2008 07:00:32 -0500 Joe Peterson <[EMAIL PROTECTED]> wrote: > Marius Mauch wrote: > > The eclass also contains it's own implementation of unpack (renamed > > to unpack2) and src_unpack so the logic which tools/packages are > > used for unpacking can be maintained in a single place instead of > > being splitted between the package manager and the tree. > > Marius, these look like nice functions. A couple of questions: > > 1) Since it requires altering an ebuild to use these features (i.e. no > automatic), what is the advantage over just including the dep the > usual way? You wouldn't have to maintain the relevant deps, e.g. when you change SRC_URI, when the unpack implementation changes (e.g. adding support for new unpackers) or keeping the conditionals in sync with SRC_URI. > 2) The name "unpack2" is intended to be temporary, right? ;) I'd > guess that after this functionality is integrated into portage, there > would be no need for the extra unpack func. But then we'd probably > have to keep "unpack2" for a long time as an alias. Any way to avoid > that? Well, it can't be named unpack to avoid conflicts with the internal portage function. If this functionality would be added on the PM side then there would be no more need for the eclass (you wouldn't want to use both implementations at the same time), and it would be opt-in via a new EAPI, so you'd have to change the ebuilds anyway to benefit from it. So I don't see a need to have a unpack2 alias outside the eclass. > 3) Same would go for the needed call for unpack_update_depend, > correct? I.e. it would not be needed after portage has this > functionality (and at that point, the unpack and dep code would be > unified cleanly). So this function would then become a stub, > correct? The only thing here is that ebuilds could linger for some > time without being "cleaned up". See above. Marius -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Marius Mauch wrote: > The eclass also contains it's own implementation of unpack (renamed to > unpack2) and src_unpack so the logic which tools/packages are used for > unpacking can be maintained in a single place instead of being > splitted between the package manager and the tree. Marius, these look like nice functions. A couple of questions: 1) Since it requires altering an ebuild to use these features (i.e. no automatic), what is the advantage over just including the dep the usual way? 2) The name "unpack2" is intended to be temporary, right? ;) I'd guess that after this functionality is integrated into portage, there would be no need for the extra unpack func. But then we'd probably have to keep "unpack2" for a long time as an alias. Any way to avoid that? 3) Same would go for the needed call for unpack_update_depend, correct? I.e. it would not be needed after portage has this functionality (and at that point, the unpack and dep code would be unified cleanly). So this function would then become a stub, correct? The only thing here is that ebuilds could linger for some time without being "cleaned up". -Joe -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Tue, 15 Jul 2008 04:14:18 +0200 Marius Mauch <[EMAIL PROTECTED]> wrote: > As a result of Cardoes earlier mail we talked a bit about possible > solutions in #gento-portage, and I suggested to let portage > automatically inject the deps based on SRC_URI pattern matching. > A mapping of extensions and their unpack deps would be kept in the > tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )' At dev.gentoo.org/~genone/unpack.eclass is the draft for an eclass to implement this feature. It doesn't modify DEPEND by itself, ebuild have to call unpack_update_depend on their own for that, as automatic updates are problematic when you have to take indirect and multiple inheritance of eclasses into account. As automatic updates aren't implemented there is also no need for a RESTRICT setting or similar, and also no requirements on the order of SRC_URI/RESTRICT/inherit. The eclass also contains it's own implementation of unpack (renamed to unpack2) and src_unpack so the logic which tools/packages are used for unpacking can be maintained in a single place instead of being splitted between the package manager and the tree. If someone wants to use it I can add it on the tree (after the normal review process and being better tested), but I'll only be doing it when there is an actual demand for it (no point in adding an eclass that nobody uses). It's still possible to come up with a package-manager-based solution in the future, but for the moment the eclass should help with the immideate problem (though the problem isn't exactly new). Marius -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Wed, 16 Jul 2008 00:43:28 +0200 Marius Mauch <[EMAIL PROTECTED]> wrote: > Interesting idea. Unfortunately our depstring parser doesn't like > empty parentheses (as they are usually problem indicators), so it > doesn't work out. Bleh. You can hack around that using a second (looping) pass then. I think this is right, assuming extglob: for p in $SRC_URI ; do if [[ "${p}" == "(" ]] || [[ "${p}" == ")" ]] || \ [[ "${p%\?}" != "${p}" ]] ; then UNPACK_DEPENDS="${UNPACK_DEPENDS} $p " elif [[ "${p%.zip}" != "${p}" ]] ; then UNPACK_DEPENDS="${UNPACK_DEPENDS} app-arch/unzip " elif [[ "${p%.bz2}" != "${p}" ]] ; then UNPACK_DEPENDS="${UNPACK_DEPENDS} app-arch/bzip2 " fi done while true ; do UNPACK_DEPENDS=" ${UNPACK_DEPENDS} " u="${UNPACK_DEPENDS}" UNPACK_DEPENDS="${UNPACK_DEPENDS//?(+( )+([^ ])\?)+( )(+( ))+( )/ }" [[ "$u" == "${UNPACK_DEPENDS}" ]] && break done Although... Allowing ( ) makes much more sense... -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Tue, 15 Jul 2008 22:34:33 +0100 Ciaran McCreesh <[EMAIL PROTECTED]> wrote: > On Tue, 15 Jul 2008 23:23:26 +0200 > Marius Mauch <[EMAIL PROTECTED]> wrote: > > Right, just I'd expect the parsing of SRC_URI (with conditionals) to > > be a bit tricky in bash, not something I'm going to work on. An > > eclass-based solution would have a few benefits though wrt the > > metadata cache. > > Well... You don't really have to parse it. > > for p in $SRC_URI ; do > if [[ "${p}" == "(" ]] || [[ "${p}" == ")" ]] || \ > [[ "${p%\?}" != "${p}" ]] ; then > UNPACK_DEPENDS="${UNPACK_DEPENDS} $p " > elif [[ "${p%.zip}" != "${p}" ]] ; then > UNPACK_DEPENDS="${UNPACK_DEPENDS} app-arch/unzip " > elif [[ "${p%.bz2}" != "${p}" ]] ; then > UNPACK_DEPENDS="${UNPACK_DEPENDS} app-arch/bzip2 " > fi > done > > Granted, it'll generate invalid output if SRC_URI is invalid (for > example, if SRC_URI has mismatched parens, the output will too), but I > can't think of any situation where breaking DEPEND if SRC_URI is > already broken is a problem. Interesting idea. Unfortunately our depstring parser doesn't like empty parentheses (as they are usually problem indicators), so it doesn't work out. Marius -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Tue, 15 Jul 2008 23:23:26 +0200 Marius Mauch <[EMAIL PROTECTED]> wrote: > Right, just I'd expect the parsing of SRC_URI (with conditionals) to > be a bit tricky in bash, not something I'm going to work on. An > eclass-based solution would have a few benefits though wrt the > metadata cache. Well... You don't really have to parse it. for p in $SRC_URI ; do if [[ "${p}" == "(" ]] || [[ "${p}" == ")" ]] || \ [[ "${p%\?}" != "${p}" ]] ; then UNPACK_DEPENDS="${UNPACK_DEPENDS} $p " elif [[ "${p%.zip}" != "${p}" ]] ; then UNPACK_DEPENDS="${UNPACK_DEPENDS} app-arch/unzip " elif [[ "${p%.bz2}" != "${p}" ]] ; then UNPACK_DEPENDS="${UNPACK_DEPENDS} app-arch/bzip2 " fi done Granted, it'll generate invalid output if SRC_URI is invalid (for example, if SRC_URI has mismatched parens, the output will too), but I can't think of any situation where breaking DEPEND if SRC_URI is already broken is a problem. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Tue, 15 Jul 2008 19:12:37 +0100 Ciaran McCreesh <[EMAIL PROTECTED]> wrote: > On Tue, 15 Jul 2008 04:14:18 +0200 > Marius Mauch <[EMAIL PROTECTED]> wrote: > > As a result of Cardoes earlier mail we talked a bit about possible > > solutions in #gento-portage, and I suggested to let portage > > automatically inject the deps based on SRC_URI pattern matching. > > A mapping of extensions and their unpack deps would be kept in the > > tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )' > > Could do it just as an eclass... > > inherit work-out-my-unpack-deps-for-me > > In principle, there's nothing that can't be done on the ebuild side > here. Right, just I'd expect the parsing of SRC_URI (with conditionals) to be a bit tricky in bash, not something I'm going to work on. An eclass-based solution would have a few benefits though wrt the metadata cache. Marius -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
If an ebuild lists bzip2 in DEPEND, the package manager has to bring it in. The proposed method would only add automatically determined dependencies, not remove what you listed in DEPEND. A hypothetical problem is; If a package source file has a bz extension but does not need bzip2 in any way, an extra dependency is in effect. But can this situation ever happen ? I guess not. Rémi Cardona demis ki:: Fabian Groffen wrote: Manual override as in "emerge --nodeps" or something. No, manual override as in "the ebuild turns off auto-detection and specifically asks for app-arch/{bzip2,gzip,tar,whatever} using DEPEND" Cheers, Rémi -- Gokdeniz Karadag -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Tue, 15 Jul 2008 22:15:16 +0300 Petteri Räty <[EMAIL PROTECTED]> wrote: > > Could do it just as an eclass... > > > > inherit work-out-my-unpack-deps-for-me > > > > In principle, there's nothing that can't be done on the ebuild side > > here. > > Wouldn't this also require having a variable like SRC_URI or > AUTODEP_SRC_URI above inherit? Yep. Although, you could do something like: inherit normal-eclass-stuff SRC_URI="blah" inherit fancy-extra-eclass-stuff -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Ciaran McCreesh kirjoitti: On Tue, 15 Jul 2008 04:14:18 +0200 Marius Mauch <[EMAIL PROTECTED]> wrote: As a result of Cardoes earlier mail we talked a bit about possible solutions in #gento-portage, and I suggested to let portage automatically inject the deps based on SRC_URI pattern matching. A mapping of extensions and their unpack deps would be kept in the tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )' Could do it just as an eclass... inherit work-out-my-unpack-deps-for-me In principle, there's nothing that can't be done on the ebuild side here. Wouldn't this also require having a variable like SRC_URI or AUTODEP_SRC_URI above inherit? Regards, Petteri signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Ciaran McCreesh wrote: On Tue, 15 Jul 2008 04:14:18 +0200 Marius Mauch <[EMAIL PROTECTED]> wrote: As a result of Cardoes earlier mail we talked a bit about possible solutions in #gento-portage, and I suggested to let portage automatically inject the deps based on SRC_URI pattern matching. A mapping of extensions and their unpack deps would be kept in the tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )' Could do it just as an eclass... inherit work-out-my-unpack-deps-for-me In principle, there's nothing that can't be done on the ebuild side here. For the future EAPI side, you could require package managers to define super-magic/package-manager-unpack-bzip2 etc for whatever they use to do the unpacking, and to make it work with existing EAPIs, use || deps with what existing package managers happen to use as the second item. For current EAPIs it's no worse than the existing situation, where ebuilds have to guess that package managers use 'app-arch/unzip' to unpack zip files. This is pretty close to what I had brought up in #gentoo-portage yesterday as well and seems like the best step forward. -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Fabian Groffen wrote: Manual override as in "emerge --nodeps" or something. No, manual override as in "the ebuild turns off auto-detection and specifically asks for app-arch/{bzip2,gzip,tar,whatever} using DEPEND" Cheers, Rémi -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Tue, 15 Jul 2008 04:14:18 +0200 Marius Mauch <[EMAIL PROTECTED]> wrote: > As a result of Cardoes earlier mail we talked a bit about possible > solutions in #gento-portage, and I suggested to let portage > automatically inject the deps based on SRC_URI pattern matching. > A mapping of extensions and their unpack deps would be kept in the > tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )' Could do it just as an eclass... inherit work-out-my-unpack-deps-for-me In principle, there's nothing that can't be done on the ebuild side here. For the future EAPI side, you could require package managers to define super-magic/package-manager-unpack-bzip2 etc for whatever they use to do the unpacking, and to make it work with existing EAPIs, use || deps with what existing package managers happen to use as the second item. For current EAPIs it's no worse than the existing situation, where ebuilds have to guess that package managers use 'app-arch/unzip' to unpack zip files. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On 15-07-2008 19:53:47 +0200, Rémi Cardona wrote: > Marius Mauch wrote: >> Potential problems: >> - might cause trouble for some packages that use custom code for >> unpacking, or due to circular deps, this could simply be solved with a >> new RESTRICT value though. > > As long as this is done to allow a 100% manual override, then this is a > _very_ good idea. Manual override as in "emerge --nodeps" or something. I have no examples at hand, but during bootstrapping I have more than often a problem with dependencies, which requires careful manual emerging of packages (often ignoring deps) because those deps cannot be compiled or emerged yet. -- Fabian Groffen Gentoo on a different level -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Marius Mauch wrote: Potential problems: - might cause trouble for some packages that use custom code for unpacking, or due to circular deps, this could simply be solved with a new RESTRICT value though. As long as this is done to allow a 100% manual override, then this is a _very_ good idea. Rémi -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Marius Mauch kirjoitti: So, is this something ebuild maintainers would like in general, or does such a feature cause you nightmares? I have actually been thinking about writing support for this into the Java eclasses because most Java upstreams use .zip files and we need app-arch/unzip in DEPEND for that. Regards, Petteri signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Marius Mauch wrote: As a result of Cardoes earlier mail we talked a bit about possible solutions in #gento-portage, and I suggested to let portage automatically inject the deps based on SRC_URI pattern matching. A mapping of extensions and their unpack deps would be kept in the tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )' Benefits: - simplified depstrings for most packages (I assume >90% of the tree) - easier maintenance if dependencies are changing, or additional implementations become supported (this could also be achieved with virtuals though) - potentially more accurate dependencies, as some common errors would be eliminated (e.g. proper treatment of use-conditionals, not adding unpack deps to RDEPEND) - long-term adds the possibility to remove bzip2, gzip and tar from @system Potential problems: - might cause trouble for some packages that use custom code for unpacking, or due to circular deps, this could simply be solved with a new RESTRICT value though. - automagic deps could be confusing to devs/users - not available for existing EAPIs (due to the mentioned problems) - slightly slower dep calculations due to the extra processing - possible match errors So, is this something ebuild maintainers would like in general, or does such a feature cause you nightmares? Marius I'm in favor of this. -- gentoo-dev@lists.gentoo.org mailing list
[gentoo-dev] RFC: auto-detection of unpack dependencies
As a result of Cardoes earlier mail we talked a bit about possible solutions in #gento-portage, and I suggested to let portage automatically inject the deps based on SRC_URI pattern matching. A mapping of extensions and their unpack deps would be kept in the tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )' Benefits: - simplified depstrings for most packages (I assume >90% of the tree) - easier maintenance if dependencies are changing, or additional implementations become supported (this could also be achieved with virtuals though) - potentially more accurate dependencies, as some common errors would be eliminated (e.g. proper treatment of use-conditionals, not adding unpack deps to RDEPEND) - long-term adds the possibility to remove bzip2, gzip and tar from @system Potential problems: - might cause trouble for some packages that use custom code for unpacking, or due to circular deps, this could simply be solved with a new RESTRICT value though. - automagic deps could be confusing to devs/users - not available for existing EAPIs (due to the mentioned problems) - slightly slower dep calculations due to the extra processing - possible match errors So, is this something ebuild maintainers would like in general, or does such a feature cause you nightmares? Marius -- gentoo-dev@lists.gentoo.org mailing list