Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
В Чтв, 17/07/2008 в 04:51 +0200, Marius Mauch пишет:
At dev.gentoo.org/~genone/unpack.eclass is the draft for an eclass
to implement this feature.
Marius, although it's possible to do this things in eclass why is eclass
better? As I see portage's unpack() already has extension - program
to unpack relation. Basically unpack() in ebuild.sh has the following
code:
unpack() {
[snip]
case ${x##*.} in
tar)
tar xof ${srcdir}${x} ${tar_opts} || die $myfail
;;
tgz)
tar xozf ${srcdir}${x} ${tar_opts} || die $myfail
[snip and so on...]
This means that every ebuild which uses 'unpack ${A}' should have in
DEPEND a program which is selected by filename extension of sources. So
as I understood your initial suggestion was to make this happen
automatically. And this is very logical as makes ebuilds cleaner and
more terse. So why did you changed your mind and try to write another
eclass (which then should sit in the tree forever), to create duplicate
unpack function instead of just making step you suggested initially? Is
there any intension to remove unpack logic from package manager? And if
yes, why?
--
Peter.
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Sun, 20 Jul 2008 17:41:58 +0400
Peter Volkov [EMAIL PROTECTED] wrote:
В Чтв, 17/07/2008 в 04:51 +0200, Marius Mauch пишет:
At dev.gentoo.org/~genone/unpack.eclass is the draft for an eclass
to implement this feature.
Marius, although it's possible to do this things in eclass why is
eclass better? As I see portage's unpack() already has extension -
program to unpack relation. Basically unpack() in ebuild.sh has the
following code:
unpack() {
[snip]
case ${x##*.} in
tar)
tar xof ${srcdir}${x} ${tar_opts} || die $myfail
;;
tgz)
tar xozf ${srcdir}${x} ${tar_opts} || die $myfail
[snip and so on...]
This means that every ebuild which uses 'unpack ${A}' should have in
DEPEND a program which is selected by filename extension of sources.
So as I understood your initial suggestion was to make this happen
automatically. And this is very logical as makes ebuilds cleaner and
more terse. So why did you changed your mind and try to write another
eclass (which then should sit in the tree forever), to create
duplicate unpack function instead of just making step you suggested
initially? Is there any intension to remove unpack logic from package
manager? And if yes, why?
The eclass is provided as a proof-of-concept implementation that can be
used without adding additional infrastructure and implementing and
defining a new EAPI version. PM-based solutions would be nicer
long-term, but are also more tricky to get right (mainly a question
of where to inject the deps). Also the eclass approach has the benefit
that the whole unpack logic can be maintained in one location vs. being
split between multiple places in the package manager and the tree, and
is easier to extend (no need for an EAPI change just to add a new unpack
format).
And as you ask, there have been plans to move quite a bit of
the bash code from portage into the tree (transparently to ebuilds
though) as that's the better place to maintain many helper functions
like do* or new* that are only used by ebuilds/eclasses and not portage.
But those plans have already existed for years (long before EAPI
existed) without anything happening, and they've never been very
specific wrt what exactly could be moved.
Even with a PM-based solution it might be useful, depending on the
actual implementation, to have the unpack() funtion in the tree, for
the reasons outlined in the first paragraph.
Marius
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Peter Volkov wrote:
This means that every ebuild which uses 'unpack ${A}' should have in
DEPEND a program which is selected by filename extension of sources. So
as I understood your initial suggestion was to make this happen
automatically. And this is very logical as makes ebuilds cleaner and
more terse. So why did you changed your mind and try to write another
eclass (which then should sit in the tree forever), to create duplicate
unpack function instead of just making step you suggested initially? Is
there any intension to remove unpack logic from package manager? And if
yes, why?
++
I also was wondering this. And if we add unpack2(), which could be a
little hard to explain in the documentation, it will need to be there at
least until ebuilds stop using it (when portage gets this functionality
ultimately).
For my uses, I'd rather do deps manually and use the official portage
unpack() until portage has such features in order to keep my ebuilds
looking a bit more clean.
-Joe
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Marius Mauch kirjoitti: If someone wants to use it I can add it on the tree (after the normal review process and being better tested), but I'll only be doing it when there is an actual demand for it (no point in adding an eclass that nobody uses). I have been long thinking about adding support for .zip handling to java eclasses so let's get this in and it will get a user from java-pkg-2 and java-pkg-opt-2 eclasses at least. Regards, Petteri signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Marius Mauch wrote: The eclass also contains it's own implementation of unpack (renamed to unpack2) and src_unpack so the logic which tools/packages are used for unpacking can be maintained in a single place instead of being splitted between the package manager and the tree. Marius, these look like nice functions. A couple of questions: 1) Since it requires altering an ebuild to use these features (i.e. no automatic), what is the advantage over just including the dep the usual way? 2) The name unpack2 is intended to be temporary, right? ;) I'd guess that after this functionality is integrated into portage, there would be no need for the extra unpack func. But then we'd probably have to keep unpack2 for a long time as an alias. Any way to avoid that? 3) Same would go for the needed call for unpack_update_depend, correct? I.e. it would not be needed after portage has this functionality (and at that point, the unpack and dep code would be unified cleanly). So this function would then become a stub, correct? The only thing here is that ebuilds could linger for some time without being cleaned up. -Joe -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Thu, 17 Jul 2008 07:00:32 -0500 Joe Peterson [EMAIL PROTECTED] wrote: Marius Mauch wrote: The eclass also contains it's own implementation of unpack (renamed to unpack2) and src_unpack so the logic which tools/packages are used for unpacking can be maintained in a single place instead of being splitted between the package manager and the tree. Marius, these look like nice functions. A couple of questions: 1) Since it requires altering an ebuild to use these features (i.e. no automatic), what is the advantage over just including the dep the usual way? You wouldn't have to maintain the relevant deps, e.g. when you change SRC_URI, when the unpack implementation changes (e.g. adding support for new unpackers) or keeping the conditionals in sync with SRC_URI. 2) The name unpack2 is intended to be temporary, right? ;) I'd guess that after this functionality is integrated into portage, there would be no need for the extra unpack func. But then we'd probably have to keep unpack2 for a long time as an alias. Any way to avoid that? Well, it can't be named unpack to avoid conflicts with the internal portage function. If this functionality would be added on the PM side then there would be no more need for the eclass (you wouldn't want to use both implementations at the same time), and it would be opt-in via a new EAPI, so you'd have to change the ebuilds anyway to benefit from it. So I don't see a need to have a unpack2 alias outside the eclass. 3) Same would go for the needed call for unpack_update_depend, correct? I.e. it would not be needed after portage has this functionality (and at that point, the unpack and dep code would be unified cleanly). So this function would then become a stub, correct? The only thing here is that ebuilds could linger for some time without being cleaned up. See above. Marius -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Tue, 15 Jul 2008 04:14:18 +0200 Marius Mauch [EMAIL PROTECTED] wrote: As a result of Cardoes earlier mail we talked a bit about possible solutions in #gento-portage, and I suggested to let portage automatically inject the deps based on SRC_URI pattern matching. A mapping of extensions and their unpack deps would be kept in the tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )' At dev.gentoo.org/~genone/unpack.eclass is the draft for an eclass to implement this feature. It doesn't modify DEPEND by itself, ebuild have to call unpack_update_depend on their own for that, as automatic updates are problematic when you have to take indirect and multiple inheritance of eclasses into account. As automatic updates aren't implemented there is also no need for a RESTRICT setting or similar, and also no requirements on the order of SRC_URI/RESTRICT/inherit. The eclass also contains it's own implementation of unpack (renamed to unpack2) and src_unpack so the logic which tools/packages are used for unpacking can be maintained in a single place instead of being splitted between the package manager and the tree. If someone wants to use it I can add it on the tree (after the normal review process and being better tested), but I'll only be doing it when there is an actual demand for it (no point in adding an eclass that nobody uses). It's still possible to come up with a package-manager-based solution in the future, but for the moment the eclass should help with the immideate problem (though the problem isn't exactly new). Marius -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Marius Mauch wrote: As a result of Cardoes earlier mail we talked a bit about possible solutions in #gento-portage, and I suggested to let portage automatically inject the deps based on SRC_URI pattern matching. A mapping of extensions and their unpack deps would be kept in the tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )' Benefits: - simplified depstrings for most packages (I assume 90% of the tree) - easier maintenance if dependencies are changing, or additional implementations become supported (this could also be achieved with virtuals though) - potentially more accurate dependencies, as some common errors would be eliminated (e.g. proper treatment of use-conditionals, not adding unpack deps to RDEPEND) - long-term adds the possibility to remove bzip2, gzip and tar from @system Potential problems: - might cause trouble for some packages that use custom code for unpacking, or due to circular deps, this could simply be solved with a new RESTRICT value though. - automagic deps could be confusing to devs/users - not available for existing EAPIs (due to the mentioned problems) - slightly slower dep calculations due to the extra processing - possible match errors So, is this something ebuild maintainers would like in general, or does such a feature cause you nightmares? Marius I'm in favor of this. -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Marius Mauch kirjoitti: So, is this something ebuild maintainers would like in general, or does such a feature cause you nightmares? I have actually been thinking about writing support for this into the Java eclasses because most Java upstreams use .zip files and we need app-arch/unzip in DEPEND for that. Regards, Petteri signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Marius Mauch wrote: Potential problems: - might cause trouble for some packages that use custom code for unpacking, or due to circular deps, this could simply be solved with a new RESTRICT value though. As long as this is done to allow a 100% manual override, then this is a _very_ good idea. Rémi -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On 15-07-2008 19:53:47 +0200, Rémi Cardona wrote: Marius Mauch wrote: Potential problems: - might cause trouble for some packages that use custom code for unpacking, or due to circular deps, this could simply be solved with a new RESTRICT value though. As long as this is done to allow a 100% manual override, then this is a _very_ good idea. Manual override as in emerge --nodeps or something. I have no examples at hand, but during bootstrapping I have more than often a problem with dependencies, which requires careful manual emerging of packages (often ignoring deps) because those deps cannot be compiled or emerged yet. -- Fabian Groffen Gentoo on a different level -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Tue, 15 Jul 2008 04:14:18 +0200 Marius Mauch [EMAIL PROTECTED] wrote: As a result of Cardoes earlier mail we talked a bit about possible solutions in #gento-portage, and I suggested to let portage automatically inject the deps based on SRC_URI pattern matching. A mapping of extensions and their unpack deps would be kept in the tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )' Could do it just as an eclass... inherit work-out-my-unpack-deps-for-me In principle, there's nothing that can't be done on the ebuild side here. For the future EAPI side, you could require package managers to define super-magic/package-manager-unpack-bzip2 etc for whatever they use to do the unpacking, and to make it work with existing EAPIs, use || deps with what existing package managers happen to use as the second item. For current EAPIs it's no worse than the existing situation, where ebuilds have to guess that package managers use 'app-arch/unzip' to unpack zip files. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Fabian Groffen wrote:
Manual override as in emerge --nodeps or something.
No, manual override as in the ebuild turns off auto-detection and
specifically asks for app-arch/{bzip2,gzip,tar,whatever} using DEPEND
Cheers,
Rémi
--
gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
Ciaran McCreesh wrote: On Tue, 15 Jul 2008 04:14:18 +0200 Marius Mauch [EMAIL PROTECTED] wrote: As a result of Cardoes earlier mail we talked a bit about possible solutions in #gento-portage, and I suggested to let portage automatically inject the deps based on SRC_URI pattern matching. A mapping of extensions and their unpack deps would be kept in the tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )' Could do it just as an eclass... inherit work-out-my-unpack-deps-for-me In principle, there's nothing that can't be done on the ebuild side here. For the future EAPI side, you could require package managers to define super-magic/package-manager-unpack-bzip2 etc for whatever they use to do the unpacking, and to make it work with existing EAPIs, use || deps with what existing package managers happen to use as the second item. For current EAPIs it's no worse than the existing situation, where ebuilds have to guess that package managers use 'app-arch/unzip' to unpack zip files. This is pretty close to what I had brought up in #gentoo-portage yesterday as well and seems like the best step forward. -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Tue, 15 Jul 2008 22:15:16 +0300 Petteri Räty [EMAIL PROTECTED] wrote: Could do it just as an eclass... inherit work-out-my-unpack-deps-for-me In principle, there's nothing that can't be done on the ebuild side here. Wouldn't this also require having a variable like SRC_URI or AUTODEP_SRC_URI above inherit? Yep. Although, you could do something like: inherit normal-eclass-stuff SRC_URI=blah inherit fancy-extra-eclass-stuff -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
If an ebuild lists bzip2 in DEPEND, the package manager has to bring it in.
The proposed method would only add automatically determined dependencies, not
remove what you listed in DEPEND.
A hypothetical problem is; If a package source file has a bz extension but does
not need bzip2 in any way, an extra dependency is in effect. But can this
situation ever happen ? I guess not.
Rémi Cardona demis ki::
Fabian Groffen wrote:
Manual override as in emerge --nodeps or something.
No, manual override as in the ebuild turns off auto-detection and
specifically asks for app-arch/{bzip2,gzip,tar,whatever} using DEPEND
Cheers,
Rémi
--
Gokdeniz Karadag
--
gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Tue, 15 Jul 2008 19:12:37 +0100 Ciaran McCreesh [EMAIL PROTECTED] wrote: On Tue, 15 Jul 2008 04:14:18 +0200 Marius Mauch [EMAIL PROTECTED] wrote: As a result of Cardoes earlier mail we talked a bit about possible solutions in #gento-portage, and I suggested to let portage automatically inject the deps based on SRC_URI pattern matching. A mapping of extensions and their unpack deps would be kept in the tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )' Could do it just as an eclass... inherit work-out-my-unpack-deps-for-me In principle, there's nothing that can't be done on the ebuild side here. Right, just I'd expect the parsing of SRC_URI (with conditionals) to be a bit tricky in bash, not something I'm going to work on. An eclass-based solution would have a few benefits though wrt the metadata cache. Marius -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Tue, 15 Jul 2008 23:23:26 +0200
Marius Mauch [EMAIL PROTECTED] wrote:
Right, just I'd expect the parsing of SRC_URI (with conditionals) to
be a bit tricky in bash, not something I'm going to work on. An
eclass-based solution would have a few benefits though wrt the
metadata cache.
Well... You don't really have to parse it.
for p in $SRC_URI ; do
if [[ ${p} == ( ]] || [[ ${p} == ) ]] || \
[[ ${p%\?} != ${p} ]] ; then
UNPACK_DEPENDS=${UNPACK_DEPENDS} $p
elif [[ ${p%.zip} != ${p} ]] ; then
UNPACK_DEPENDS=${UNPACK_DEPENDS} app-arch/unzip
elif [[ ${p%.bz2} != ${p} ]] ; then
UNPACK_DEPENDS=${UNPACK_DEPENDS} app-arch/bzip2
fi
done
Granted, it'll generate invalid output if SRC_URI is invalid (for
example, if SRC_URI has mismatched parens, the output will too), but I
can't think of any situation where breaking DEPEND if SRC_URI is
already broken is a problem.
--
Ciaran McCreesh
signature.asc
Description: PGP signature
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Tue, 15 Jul 2008 22:34:33 +0100
Ciaran McCreesh [EMAIL PROTECTED] wrote:
On Tue, 15 Jul 2008 23:23:26 +0200
Marius Mauch [EMAIL PROTECTED] wrote:
Right, just I'd expect the parsing of SRC_URI (with conditionals) to
be a bit tricky in bash, not something I'm going to work on. An
eclass-based solution would have a few benefits though wrt the
metadata cache.
Well... You don't really have to parse it.
for p in $SRC_URI ; do
if [[ ${p} == ( ]] || [[ ${p} == ) ]] || \
[[ ${p%\?} != ${p} ]] ; then
UNPACK_DEPENDS=${UNPACK_DEPENDS} $p
elif [[ ${p%.zip} != ${p} ]] ; then
UNPACK_DEPENDS=${UNPACK_DEPENDS} app-arch/unzip
elif [[ ${p%.bz2} != ${p} ]] ; then
UNPACK_DEPENDS=${UNPACK_DEPENDS} app-arch/bzip2
fi
done
Granted, it'll generate invalid output if SRC_URI is invalid (for
example, if SRC_URI has mismatched parens, the output will too), but I
can't think of any situation where breaking DEPEND if SRC_URI is
already broken is a problem.
Interesting idea. Unfortunately our depstring parser doesn't like empty
parentheses (as they are usually problem indicators), so it doesn't
work out.
Marius
--
gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] RFC: auto-detection of unpack dependencies
On Wed, 16 Jul 2008 00:43:28 +0200
Marius Mauch [EMAIL PROTECTED] wrote:
Interesting idea. Unfortunately our depstring parser doesn't like
empty parentheses (as they are usually problem indicators), so it
doesn't work out.
Bleh. You can hack around that using a second (looping) pass then. I
think this is right, assuming extglob:
for p in $SRC_URI ; do
if [[ ${p} == ( ]] || [[ ${p} == ) ]] || \
[[ ${p%\?} != ${p} ]] ; then
UNPACK_DEPENDS=${UNPACK_DEPENDS} $p
elif [[ ${p%.zip} != ${p} ]] ; then
UNPACK_DEPENDS=${UNPACK_DEPENDS} app-arch/unzip
elif [[ ${p%.bz2} != ${p} ]] ; then
UNPACK_DEPENDS=${UNPACK_DEPENDS} app-arch/bzip2
fi
done
while true ; do
UNPACK_DEPENDS= ${UNPACK_DEPENDS}
u=${UNPACK_DEPENDS}
UNPACK_DEPENDS=${UNPACK_DEPENDS//?(+( )+([^ ])\?)+( )(+( ))+( )/ }
[[ $u == ${UNPACK_DEPENDS} ]] break
done
Although... Allowing ( ) makes much more sense...
--
Ciaran McCreesh
signature.asc
Description: PGP signature
