Re: [gentoo-dev] Addressing GLEP-62 itself
On Tue, 25 Sep 2012 12:54:39 -0700 Brian Harring ferri...@gmail.com wrote: On Tue, Sep 25, 2012 at 08:58:07PM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 14:47:33 -0400 Ian Stakenvicius a...@gentoo.org wrote: Based on the above I do expect the reference implementation would also need to change. I expect, for instance, that the PM's metadata-handling would need to occur as normal even though none of the package's phase functions would run, that is, *DEPEND (realistically RDEPEND as that should be the only one affected here, maybe PDEPEND too) and USE/PKGUSE would get updated. Since portage would not be re-emerging the package from the tree the original ebuild would remain. Yes, unless I'm missing something that's the intent. I will re-read and update the GLEP a bit sometime this week. There's a fairly strong user interaction component here, along w/ potential nastyness for ebuilds (the proposal assume that a flag will be toggable in all cases within an ebuild if IUSE_RUNTIME specified; I guarantee instances where that fails can be found in the tree if a basic audit was done). Additionally, this *is* useless if it's done in a form the UI an't display/handle; Ciaran may bitch about REQUIRED_USE's UI (which I knew going in was going to be problematic, just to be clear), but he's right on that front. Additionally, this needs to be thought out how it'll interact with eclasses; what stacking, etc. It doesn't cover the basics there to say the least. The proposal didn't cover eclasses at all. Is there a need to do so or are we chasing some kind of perfection based on filling all unused slots? Pretty much, this needs an implementation, partial conversion of the tree to demonstrate it. Just to prove that fricking point; if you had tried implementing this, a full investigation of what's involved alone, you'd have spotted that the core of the proposal is based on a wrong assumption. Portage doesn't write unfinalized DEPEND/RDEPEND/PDEPEND to the VDB. There's a footnote there, saying: The package manager has to ensure that all relevant information is stored in the installed package metadata. I expect, as a corollary to this, that a rebuild would be necessary if (on-disk-IUSE_RUNTIME xor in-ebuild-IUSE_RUNTIME) was non-empty (--newuse) or resulted in any flags that are in USE (--reinstall=changed-use). IMO this would be necessary to ensure the local ebuild copy and all related metadata for it gets updated in vdb. I think that's a common package manager logic and it's out of scope of the GLEP. Portage doesn't do physical updates last I saw- if it did, well, that's fairly dangerous. Only thing stored in the VDB is the ebuild and the environment dump, and the environment dump is what's ran from. You cannot sanely pick part the dump and try to intermix current ebuilds; rephrasing, I'll kick in the head anyone who tries that. What are you talking about? As far as I can see, we are talking here about *full rebuild*. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] [RFC] Multiple ABI support through package appending/partial removal
On Tue, 25 Sep 2012 13:12:56 -0700 Brian Harring ferri...@gmail.com wrote: On Mon, Sep 24, 2012 at 12:09:49AM +0200, Micha?? G??rny wrote: Hello, Since my previous idea of DYNAMIC_SLOTS proved too complex to design and implement, I would like to offer an another idea, based partially on what Ciaran mentioned. Before I start getting into details, I'd like to know your opinions, and what possible problems am I missing. To keep it clean, I will focus on Python ABIs but other languages and multilib could be handled in a similar manner. The problem === Right now, building packages for multiple Python ABIs is done using USE_EXPAND-based useflags. This is a working solution but it requires rebuilding the package for all ABIs whenever the chosen ABI list changes. While it may be not that important for most of the Python packages, it becomes such when it comes to things like boost or -- if we'd extend that to multilib -- say, llvm. In that case, whenever a newly-installed package requests a specific ABI, user has to spend twice as much time to rebuild the same version. The general idea While not getting too deep into ebuild syntax, the core part of the idea is to mark some of the USE_EXPAND variables 'special'. In this particular example, such a special flag group would be 'PYTHON_TARGETS'. Now, let's consider user installs a new package with one python_targets_python2_7 enabled. The package is built and installed like usual but aside to regular vdb files an additional file is introduced, listing all the installed files as 'belonging' to python_targets_python2_7. If user enables python_targets_python3_2 on the same package, the PM doesn't trigger a full rebuild. Instead, it builds the package with the new flag being the only flag in PYTHON_TARGETS. The new files are installed over the installed package (and added to CONTENTS in vdb), and the files in install image are listed in vdb as 'belonging' to python_targets_python3_2. What you're proposing would liter the ebuild/eclass with has_version checks; in brain dead simple cases, you can replace parts of the pkg as you're proposing there. However if it installs scripts, things start getting more complex; needs to vary how it installs if it's overlaying part of itself. That's the idea. You're given a tool, now thinking twice before using it. This proposal also doesn't work in the phase of := slot deps either, not unless you've got a way to ensure, potentially weeks/months after the first build, that the node locks to the same slotting. A rebuild then? Whenever files from two ABIs collide, package manager either replaces the installed files if the 'new' ABI is considered 'better' than the old one or preserves it. This follows the current behavior when multiple ABIs are built, and later builds overwrite files from earlier ones. This is handwavey and kind of crackadled; the PM has no way of knowing which USE_EXPAND target is considered 'best', so in the case of multilib (say 64b and 32b subversion) there isn't any way to which svn binary should be there- 64b or 32b. Best I can tell, your proposal winds up just being last one to merge wins which isn't acceptable. This is just an early idea. Details like precedence weren't converted into any syntax yet. At the point, the additional file contains something like (ugly pseudo-syntax): /usr/lib64/python2.7/foo.py python_targets_python2_7 /usr/lib64/python3.2/foo.py python_targets_python3_2 /usr/share/doc/foo-1.2.3/README.bz2 python_targets_python2_7 \ python_targets_python3_2 Now, if user requests disabling python_targets_python2_7 on the package, the package manager may not rebuild it as well. Instead, it removes python_targets_python2_7 from the above list, and unmerges the files which don't belong into any other ABI. If we're going to do sub-packaging... which is what you're attempting here... the VDB backend for it minimally cannot be a one off USE_EXPAND hack. That'll just back us into a corner- which the vdb already does quite heavily. Any subpackaging content tracking needs to be generic and usable. Really is that simple. Give a better explanation if you want me to follow your thoughts. Sadly, this will not 'downgrade' common files to another ABI but I believe that it is not really a killer-feature. Installing new packages and upgrading existing == Whenever a new package is to be built and multiple ABIs are requested, the package manager should split the build process between particular ABIs. Preferably, it should build all of them one-by-one, recording the 'belongs' entries from the image and then install them as a single package. And how does the package know that it's being targetted at
Re: [gentoo-dev] GLEP: gentoo sync based unified deps proposal
On Sun, 16 Sep 2012 06:52:11 -0700 Brian Harring ferri...@gmail.com wrote: Keeping it short and quick, a basic glep has been written for what I'm proposing for DEPENDENCIES enhancement. The live version of the doc is available at http://dev.gentoo.org/~ferringb/unified-dependencies/extensible_dependencies.html One more question -- are we going to keep 'foo,bar?' syntax as a special case applying only to dependency atoms or are we going to extend it to USE flags? -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] Re: Lastrite: net-misc/mediatomb and www-apache/mod_musicindex
On 26/09/12 08:24, Ryan Hill wrote: On Tue, 25 Sep 2012 18:17:03 +0300 Samuli Suominen ssuomi...@gentoo.org wrote: # Samuli Suominen ssuomi...@gentoo.org (25 Sep 2012) # Multiple build failures: #435394, #423991 and #425806 # Other bugs: #270830, #368409 # Unmasking would require addressing the build failure bugs # Removal in 30 days net-misc/mediatomb I use this daily so I'll have a look if no one else does. Thanks!
Re: [gentoo-dev] Addressing GLEP-62 itself
On Wed, Sep 26, 2012 at 08:02:44AM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 12:54:39 -0700 Brian Harring ferri...@gmail.com wrote: On Tue, Sep 25, 2012 at 08:58:07PM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 14:47:33 -0400 Ian Stakenvicius a...@gentoo.org wrote: Based on the above I do expect the reference implementation would also need to change. I expect, for instance, that the PM's metadata-handling would need to occur as normal even though none of the package's phase functions would run, that is, *DEPEND (realistically RDEPEND as that should be the only one affected here, maybe PDEPEND too) and USE/PKGUSE would get updated. Since portage would not be re-emerging the package from the tree the original ebuild would remain. Yes, unless I'm missing something that's the intent. I will re-read and update the GLEP a bit sometime this week. There's a fairly strong user interaction component here, along w/ potential nastyness for ebuilds (the proposal assume that a flag will be toggable in all cases within an ebuild if IUSE_RUNTIME specified; I guarantee instances where that fails can be found in the tree if a basic audit was done). Additionally, this *is* useless if it's done in a form the UI an't display/handle; Ciaran may bitch about REQUIRED_USE's UI (which I knew going in was going to be problematic, just to be clear), but he's right on that front. ^^^ This point still needs addressing. Additionally, this needs to be thought out how it'll interact with eclasses; what stacking, etc. It doesn't cover the basics there to say the least. The proposal didn't cover eclasses at all. Is there a need to do so or are we chasing some kind of perfection based on filling all unused slots? Eclass stacking here matters; if it's stacked, it means ebuilds have to use out of bound (ie, other vars) to tell the eclass when it shouldn't mark a flag as runtime toggable. If it's not stacked by the pm, then they have to manually stack; that differs from the norm and makes it easier to screwup; however; does allow for them to filter, albeit a slight pain in the ass doing so. There's a choice there, and the answer matters, so yes, you should actually have a complete glep before trying to shove it up to the council and extract a vote out of them. Lest the intention is to just have them kick it back to the curb... Pretty much, this needs an implementation, partial conversion of the tree to demonstrate it. Just to prove that fricking point; if you had tried implementing this, a full investigation of what's involved alone, you'd have spotted that the core of the proposal is based on a wrong assumption. Portage doesn't write unfinalized DEPEND/RDEPEND/PDEPEND to the VDB. There's a footnote there, saying: The package manager has to ensure that all relevant information is stored in the installed package metadata. Frankly I don't fully buy that you were aware of this issue from the start of the proposal; the wording partially covers it however. Ddoesn't call it out, but via tha req it dumps it on the package manager developers heads to sort it- which already is the case. Binpkgs minimally weren't addressed which is why I still don't think this was actually spotted up front. Were it, the performance impact should've been mentioned, and quantified; con's are part of a normal glep. Meaning.. wait for it... writing a prototype is required to get those stats. ;) Based on the above I do expect the reference implementation would alsoneed to change. I expect, for instance, that the PM's metadata-handling would need to occur as normal even though none of the package's phase functions would run, that is, *DEPEND (realistically RDEPEND as that should be the only one affected here, maybe PDEPEND too) and USE/PKGUSE would get updated. Since portage would not be re-emerging the package from the tree the original ebuild would remain. I expect, as a corollary to this, that a rebuild would be necessary if (on-disk-IUSE_RUNTIME xor in-ebuild-IUSE_RUNTIME) was non-empty (--newuse) or resulted in any flags that are in USE (--reinstall=changed-use). IMO this would be necessary to ensure the local ebuild copy and all related metadata for it gets updated in vdb. I think that's a common package manager logic and it's out of scope of the GLEP. Portage doesn't do physical updates last I saw- if it did, well, that's fairly dangerous. Only thing stored in the VDB is the ebuild and the environment dump, and the environment dump is what's ran from. You cannot sanely pick part the dump and try to intermix current ebuilds; rephrasing, I'll kick in the head anyone who tries that. What are you talking about? As far as I can see, we are talking here about *full rebuild*. Added the full quote
Re: [gentoo-dev] GLEP: gentoo sync based unified deps proposal
On Wed, Sep 26, 2012 at 08:58:54AM +0200, Micha?? G??rny wrote: On Sun, 16 Sep 2012 06:52:11 -0700 Brian Harring ferri...@gmail.com wrote: Keeping it short and quick, a basic glep has been written for what I'm proposing for DEPENDENCIES enhancement. The live version of the doc is available at http://dev.gentoo.org/~ferringb/unified-dependencies/extensible_dependencies.html One more question -- are we going to keep 'foo,bar?' syntax as a special case applying only to dependency atoms or are we going to extend it to USE flags? It's fairly dep specific; that's effectivelly foo|bar when you think about it; it multiplies out to dep:foo? ( that block ) dep:bar? ( that block ) in a dumb PM (smarter one just leaves the tree collapsed and filters as it goes). Phrased another way, I'm not sure we really need shorthand for the following: x? ( blah ) y? ( blah ) z? ( blah ) into x,y,z? ( blah ) It's a rare case; I could only foresee that potentially being of use for arch flags; ie, amd64,x86? ( blah ); which I'd write as arch:amd64,x86? ( blah ) personally since I don't like the notion of introducing ',' into raw, non use group flags. That said, I don't hugely care; people think it's useful, then have at it. ~harring
Re: [gentoo-dev] [RFC] Multiple ABI support through package appending/partial removal
On Wed, Sep 26, 2012 at 08:35:37AM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 13:12:56 -0700 Brian Harring ferri...@gmail.com wrote: On Mon, Sep 24, 2012 at 12:09:49AM +0200, Micha?? G??rny wrote: Hello, Since my previous idea of DYNAMIC_SLOTS proved too complex to design and implement, I would like to offer an another idea, based partially on what Ciaran mentioned. Before I start getting into details, I'd like to know your opinions, and what possible problems am I missing. To keep it clean, I will focus on Python ABIs but other languages and multilib could be handled in a similar manner. The problem === Right now, building packages for multiple Python ABIs is done using USE_EXPAND-based useflags. This is a working solution but it requires rebuilding the package for all ABIs whenever the chosen ABI list changes. While it may be not that important for most of the Python packages, it becomes such when it comes to things like boost or -- if we'd extend that to multilib -- say, llvm. In that case, whenever a newly-installed package requests a specific ABI, user has to spend twice as much time to rebuild the same version. The general idea While not getting too deep into ebuild syntax, the core part of the idea is to mark some of the USE_EXPAND variables 'special'. In this particular example, such a special flag group would be 'PYTHON_TARGETS'. Now, let's consider user installs a new package with one python_targets_python2_7 enabled. The package is built and installed like usual but aside to regular vdb files an additional file is introduced, listing all the installed files as 'belonging' to python_targets_python2_7. If user enables python_targets_python3_2 on the same package, the PM doesn't trigger a full rebuild. Instead, it builds the package with the new flag being the only flag in PYTHON_TARGETS. The new files are installed over the installed package (and added to CONTENTS in vdb), and the files in install image are listed in vdb as 'belonging' to python_targets_python3_2. What you're proposing would liter the ebuild/eclass with has_version checks; in brain dead simple cases, you can replace parts of the pkg as you're proposing there. However if it installs scripts, things start getting more complex; needs to vary how it installs if it's overlaying part of itself. That's the idea. You're given a tool, now thinking twice before using it. This statement doesn't make sense. Clarify it. This proposal also doesn't work in the phase of := slot deps either, not unless you've got a way to ensure, potentially weeks/months after the first build, that the node locks to the same slotting. A rebuild then? More logic to dump on the PM. Yes, you can hack around it- I pointed it out because the existence of that corner case says something about the proposal. Whenever files from two ABIs collide, package manager either replaces the installed files if the 'new' ABI is considered 'better' than the old one or preserves it. This follows the current behavior when multiple ABIs are built, and later builds overwrite files from earlier ones. This is handwavey and kind of crackadled; the PM has no way of knowing which USE_EXPAND target is considered 'best', so in the case of multilib (say 64b and 32b subversion) there isn't any way to which svn binary should be there- 64b or 32b. Best I can tell, your proposal winds up just being last one to merge wins which isn't acceptable. This is just an early idea. Details like precedence weren't converted into any syntax yet. Precedence is a core requirement of anything that's more than just a non lib; python would've already been required to be addressed if we didn't have that damn wrapper in place (when a multislotting proposal goes through, that wrapper should die in the process). At the point, the additional file contains something like (ugly pseudo-syntax): /usr/lib64/python2.7/foo.py python_targets_python2_7 /usr/lib64/python3.2/foo.py python_targets_python3_2 /usr/share/doc/foo-1.2.3/README.bz2 python_targets_python2_7 \ python_targets_python3_2 Now, if user requests disabling python_targets_python2_7 on the package, the package manager may not rebuild it as well. Instead, it removes python_targets_python2_7 from the above list, and unmerges the files which don't belong into any other ABI. If we're going to do sub-packaging... which is what you're attempting here... the VDB backend for it minimally cannot be a one off USE_EXPAND hack. That'll just back us into a corner- which the vdb already does quite heavily. Any subpackaging content tracking needs to be generic and usable. Really is that simple. Give a better
Re: [gentoo-dev] Addressing GLEP-62 itself
On Wed, 26 Sep 2012 03:29:17 -0700 Brian Harring ferri...@gmail.com wrote: On Wed, Sep 26, 2012 at 08:02:44AM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 12:54:39 -0700 Brian Harring ferri...@gmail.com wrote: On Tue, Sep 25, 2012 at 08:58:07PM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 14:47:33 -0400 Ian Stakenvicius a...@gentoo.org wrote: Based on the above I do expect the reference implementation would also need to change. I expect, for instance, that the PM's metadata-handling would need to occur as normal even though none of the package's phase functions would run, that is, *DEPEND (realistically RDEPEND as that should be the only one affected here, maybe PDEPEND too) and USE/PKGUSE would get updated. Since portage would not be re-emerging the package from the tree the original ebuild would remain. Yes, unless I'm missing something that's the intent. I will re-read and update the GLEP a bit sometime this week. There's a fairly strong user interaction component here, along w/ potential nastyness for ebuilds (the proposal assume that a flag will be toggable in all cases within an ebuild if IUSE_RUNTIME specified; I guarantee instances where that fails can be found in the tree if a basic audit was done). Additionally, this *is* useless if it's done in a form the UI an't display/handle; Ciaran may bitch about REQUIRED_USE's UI (which I knew going in was going to be problematic, just to be clear), but he's right on that front. ^^^ This point still needs addressing. IUSE_RUNTIME is optional for PMs, why does the UI matter at all ? Also, the proposal doesn't assume flags are toggable at will, it assumes they are useflags and obey the same rules. Additionally, this needs to be thought out how it'll interact with eclasses; what stacking, etc. It doesn't cover the basics there to say the least. The proposal didn't cover eclasses at all. Is there a need to do so or are we chasing some kind of perfection based on filling all unused slots? Eclass stacking here matters; if it's stacked, it means ebuilds have to use out of bound (ie, other vars) to tell the eclass when it shouldn't mark a flag as runtime toggable. If it's not stacked by the pm, then they have to manually stack; that differs from the norm and makes it easier to screwup; however; does allow for them to filter, albeit a slight pain in the ass doing so. There's a choice there, and the answer matters, so yes, you should actually have a complete glep before trying to shove it up to the council and extract a vote out of them. Lest the intention is to just have them kick it back to the curb... It can't be stacked and it's not wise to do so; it is a simple bash variable like tons of others in eclasses: Package managers not implementing this GLEP will consider the ``IUSE_RUNTIME`` variable as an irrelevant bash variable 2. introduce additional ``IUSE_RUNTIME`` variable listing names of USE flags related to optional runtime dependencies (without prefixes related to IUSE defaults). Treating bash variables as bash variables is rather the norm, stacking is the exception. As I understand it, your only objection here is that you want to see written 'IUSE_RUNTIME gets no special treatment' in the proposal ? A.
[gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
A few months ago, I filed bug 423651 to ask that bzip2 on the install media be replaced with pbzip2. It was closed a short while later, telling me that it'd involve changing what's kept in @system, and that had to be discussed here, rather than in a bug report. Here's a detailed description of how pbzip2 operates, as described by a friend of mine: pbzip2's compression routine splits the input into blocks (with a default of 900,000 bytes), which it then feeds into the standard bzip2 compression routine. The output of the various calls to the bzip2 compression routine are then concatenated together. The end result is the same as if you had first used the split command on the input, run individual bzip2 commands on the split pieces, then recombined the individual bz2 files using cat. The down side to this is that you have multiple file headers, footers, and byte-align padding, plus the fact that bzip2 does a RLE compression stage to fill the buffer it feeds to the BWT, the main part of the compression routine. If you happen to have a section with 1MiB of the same byte, the pbzip2 front-end will split that into two blocks (at the default settings) and feed them to separate bzip2 compressors. bzip2 will then compress the first block down to a buffer of about 17kiB before passing it on to be compressed further, and the rest of the data would have fit within this block, if pbzip2 hadn't split it the way it had. As for decompression, pbzip2 can only really do parallel decompression of files that it created, since it seeks for the bz2 file header in order to split it to different workers. One reason for this is that the bz2 block header is not byte aligned. I really don't know how to carry this discussion any further than this; I'll answer any questions I can. -- :wq
Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol mike...@gmail.com wrote: A few months ago, I filed bug 423651 to ask that bzip2 on the install media be replaced with pbzip2. It was closed a short while later, telling me that it'd involve changing what's kept in @system, and that had to be discussed here, rather than in a bug report. If we're going to ship a parallel bzip2 implementation, it should be lbzip2 and not pbzip2. lbzip2 can decompress bz2 archives with multiple threads that haven't been compressed with lbzip2/pbzip2.
Re: [gentoo-dev] Addressing GLEP-62 itself
On Wed, Sep 26, 2012 at 02:38:02PM -0300, Alexis Ballier wrote: On Wed, 26 Sep 2012 03:29:17 -0700 Brian Harring ferri...@gmail.com wrote: On Wed, Sep 26, 2012 at 08:02:44AM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 12:54:39 -0700 Brian Harring ferri...@gmail.com wrote: On Tue, Sep 25, 2012 at 08:58:07PM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 14:47:33 -0400 Ian Stakenvicius a...@gentoo.org wrote: Based on the above I do expect the reference implementation would also need to change. I expect, for instance, that the PM's metadata-handling would need to occur as normal even though none of the package's phase functions would run, that is, *DEPEND (realistically RDEPEND as that should be the only one affected here, maybe PDEPEND too) and USE/PKGUSE would get updated. Since portage would not be re-emerging the package from the tree the original ebuild would remain. Yes, unless I'm missing something that's the intent. I will re-read and update the GLEP a bit sometime this week. There's a fairly strong user interaction component here, along w/ potential nastyness for ebuilds (the proposal assume that a flag will be toggable in all cases within an ebuild if IUSE_RUNTIME specified; I guarantee instances where that fails can be found in the tree if a basic audit was done). Additionally, this *is* useless if it's done in a form the UI an't display/handle; Ciaran may bitch about REQUIRED_USE's UI (which I knew going in was going to be problematic, just to be clear), but he's right on that front. ^^^ This point still needs addressing. IUSE_RUNTIME is optional for PMs, why does the UI matter at all ? Also, the proposal doesn't assume flags are toggable at will, it assumes they are useflags and obey the same rules. I truly hate claims like this; it's optional, so who cares if it's whacky. Think through the proposal; for it to work reliably, it's not optional. Same issue I've been y'all over the head with, rendered/finalized vs raw/unfinalized deps being stored in the VDB. All managers have to write unfinalized if that proposal goes through, even if they don't support the optional toggling after the fact. As for the UI... arguing but it's optional! doesn't give a blank check for the UI angle. What the plan, more colorization and a new char for emerge -vp? Because we're kind of running out of chars there... It's a simple enough request; one that wouldn't even need to be made if there was code backing this proposal; on a related note, hell yes I'm wary of having this dumped on manager authors heads and having to be told sort out the mess/make it so. So I'm asking y'all to at least put in an equivalent time investment doing a quick prototype. This isn't an unreasonable request, and has been the norm for most gleps for a long while. Additionally, this needs to be thought out how it'll interact with eclasses; what stacking, etc. It doesn't cover the basics there to say the least. The proposal didn't cover eclasses at all. Is there a need to do so or are we chasing some kind of perfection based on filling all unused slots? Eclass stacking here matters; if it's stacked, it means ebuilds have to use out of bound (ie, other vars) to tell the eclass when it shouldn't mark a flag as runtime toggable. If it's not stacked by the pm, then they have to manually stack; that differs from the norm and makes it easier to screwup; however; does allow for them to filter, albeit a slight pain in the ass doing so. There's a choice there, and the answer matters, so yes, you should actually have a complete glep before trying to shove it up to the council and extract a vote out of them. Lest the intention is to just have them kick it back to the curb... It can't be stacked and it's not wise to do so; it is a simple bash variable like tons of others in eclasses: It cannot be stacked because y'all are trying to shove this in as an optional; unlike it's brother IUSE, which stacks. As for ons of others that don't stack; very few actually influence the package manager; ~14 roughly, minimally 5 of those stack (those that don't, basically aren't lists). Package managers not implementing this GLEP will consider the ``IUSE_RUNTIME`` variable as an irrelevant bash variable 2. introduce additional ``IUSE_RUNTIME`` variable listing names of USE flags related to optional runtime dependencies (without prefixes related to IUSE defaults). Treating bash variables as bash variables is rather the norm, stacking is the exception. As I understand it, your only objection here is that you want to see written 'IUSE_RUNTIME gets no special treatment' in the proposal ? My objection is punting it to the council till it's actually nailed down/sane; having them mark it accepted
Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
Am 26.09.2012 22:43, schrieb Matt Turner: On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol mike...@gmail.com wrote: A few months ago, I filed bug 423651 to ask that bzip2 on the install media be replaced with pbzip2. It was closed a short while later, telling me that it'd involve changing what's kept in @system, and that had to be discussed here, rather than in a bug report. If we're going to ship a parallel bzip2 implementation, it should be lbzip2 and not pbzip2. lbzip2 can decompress bz2 archives with multiple threads that haven't been compressed with lbzip2/pbzip2. This seems relevant, especially comment 12ff: https://bugs.gentoo.org/show_bug.cgi?id=309683 For further anecdotal evidence: I've used pbzip2 with USE=symlink for several months now and never had trouble with it. Checking out lbzip2 now. I noticed it doesn't install a bunzip2 symlink. Regards, Florian Philipp signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Addressing GLEP-62 itself
On 09/26/2012 11:02 PM, Brian Harring wrote: So yes, I want it actually finalized. Bluntly; there's zero point having the council comment if it ain't finalized. I understand. We probably have to wait for it then. Meanwhile... is there an alternative approach?
Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
Michael Mol schrieb: A few months ago, I filed bug 423651 to ask that bzip2 on the install media be replaced with pbzip2. If I understand correctly, pbzip2 depends on bzip2. So what you are asking is that pbzip2 is preferred over bzip2 when both are installed, and that pbzip2 is installed by default? I have so far encountered only one anecdotal case in #gentoo IRC where pbzip2[symlink] caused problems in emerging a package. Disabling the symlink flag made the problem go away. However I can't point to the report right now, maybe someone with searchable backlog can uncover it. A different question is whether in the cases where parallel bzip2 makes sense, is it really the best solution? xz is outperforming bzip2's compression ratio for large files (for an informal comparison, see bug 434350). And xz is faster at decompression, which offsets the parallel advantage to some degree. Best regards, Chí-Thanh Christopher Nguyễn
Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
On Wed, Sep 26, 2012 at 5:27 PM, Florian Philipp li...@binarywings.net wrote: Am 26.09.2012 22:43, schrieb Matt Turner: On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol mike...@gmail.com wrote: A few months ago, I filed bug 423651 to ask that bzip2 on the install media be replaced with pbzip2. It was closed a short while later, telling me that it'd involve changing what's kept in @system, and that had to be discussed here, rather than in a bug report. If we're going to ship a parallel bzip2 implementation, it should be lbzip2 and not pbzip2. lbzip2 can decompress bz2 archives with multiple threads that haven't been compressed with lbzip2/pbzip2. This seems relevant, especially comment 12ff: https://bugs.gentoo.org/show_bug.cgi?id=309683 For further anecdotal evidence: I've used pbzip2 with USE=symlink for several months now and never had trouble with it. Checking out lbzip2 now. I noticed it doesn't install a bunzip2 symlink. Piotr Szymaniak asked me about lbzip2, and I bounced the question over to my friend. He didn't investigate it deeply; it crashed (OOM or something else, I don't know) when he tried it on a large file. Could have been from 2GB to 2TB, from what he has laying around. I don't know; I didn't get that one in writing. :) But if it proves to be stable for small and very large files, I'd have no complaint. :) -- :wq
Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
On Wed, Sep 26, 2012 at 5:49 PM, Chí-Thanh Christopher Nguyễn chith...@gentoo.org wrote: Michael Mol schrieb: A few months ago, I filed bug 423651 to ask that bzip2 on the install media be replaced with pbzip2. If I understand correctly, pbzip2 depends on bzip2. So what you are asking is that pbzip2 is preferred over bzip2 when both are installed, and that pbzip2 is installed by default? pbzip2 uses libbzip2, which I understand bzip2 to also be a wrapper around. I have so far encountered only one anecdotal case in #gentoo IRC where pbzip2[symlink] caused problems in emerging a package. Disabling the symlink flag made the problem go away. However I can't point to the report right now, maybe someone with searchable backlog can uncover it. pbzip2[symlink] is more or less the scenario I'd like to see. A different question is whether in the cases where parallel bzip2 makes sense, is it really the best solution? xz is outperforming bzip2's compression ratio for large files (for an informal comparison, see bug 434350). And xz is faster at decompression, which offsets the parallel advantage to some degree. xz is faster for decompression, by my inspiration case was during system installation, so compression. Last I looked, xz was still very slow for threaded compression. And it's not block-oriented, so I don't think that's really possible without loss of compression efficiency, anyway...and my use cases range from 4-8 physical cores. -- :wq
Re: [gentoo-dev] Addressing GLEP-62 itself
On Wed, 26 Sep 2012 14:02:57 -0700 Brian Harring ferri...@gmail.com wrote: On Wed, Sep 26, 2012 at 02:38:02PM -0300, Alexis Ballier wrote: On Wed, 26 Sep 2012 03:29:17 -0700 Brian Harring ferri...@gmail.com wrote: On Wed, Sep 26, 2012 at 08:02:44AM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 12:54:39 -0700 Brian Harring ferri...@gmail.com wrote: On Tue, Sep 25, 2012 at 08:58:07PM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 14:47:33 -0400 Ian Stakenvicius a...@gentoo.org wrote: Based on the above I do expect the reference implementation would also need to change. I expect, for instance, that the PM's metadata-handling would need to occur as normal even though none of the package's phase functions would run, that is, *DEPEND (realistically RDEPEND as that should be the only one affected here, maybe PDEPEND too) and USE/PKGUSE would get updated. Since portage would not be re-emerging the package from the tree the original ebuild would remain. Yes, unless I'm missing something that's the intent. I will re-read and update the GLEP a bit sometime this week. There's a fairly strong user interaction component here, along w/ potential nastyness for ebuilds (the proposal assume that a flag will be toggable in all cases within an ebuild if IUSE_RUNTIME specified; I guarantee instances where that fails can be found in the tree if a basic audit was done). Additionally, this *is* useless if it's done in a form the UI an't display/handle; Ciaran may bitch about REQUIRED_USE's UI (which I knew going in was going to be problematic, just to be clear), but he's right on that front. ^^^ This point still needs addressing. IUSE_RUNTIME is optional for PMs, why does the UI matter at all ? Also, the proposal doesn't assume flags are toggable at will, it assumes they are useflags and obey the same rules. I truly hate claims like this; it's optional, so who cares if it's whacky. Think through the proposal; for it to work reliably, it's not optional. Same issue I've been y'all over the head with, rendered/finalized vs raw/unfinalized deps being stored in the VDB. All managers have to write unfinalized if that proposal goes through, even if they don't support the optional toggling after the fact. Why? _Current_ PMs will rebuild the package. The point of this is just to give them a hint that they do not need to rebuild it. We already have an implementation actually: one that ignores the hint :) As for the UI... arguing but it's optional! doesn't give a blank check for the UI angle. What the plan, more colorization and a new char for emerge -vp? Because we're kind of running out of chars there... How is this relevant ? It's a simple enough request; one that wouldn't even need to be made if there was code backing this proposal; on a related note, hell yes I'm wary of having this dumped on manager authors heads and having to be told sort out the mess/make it so. So I'm asking y'all to at least put in an equivalent time investment doing a quick prototype. This isn't an unreasonable request, and has been the norm for most gleps for a long while. I guess people do not want to invest time in writing code for something doomed. The original request was just to have it 'accepted' so that an implementation can start. If the implementation is good then make it final, otherwise amend or reject the glep. This isn't unreasonable either. Additionally, this needs to be thought out how it'll interact with eclasses; what stacking, etc. It doesn't cover the basics there to say the least. The proposal didn't cover eclasses at all. Is there a need to do so or are we chasing some kind of perfection based on filling all unused slots? Eclass stacking here matters; if it's stacked, it means ebuilds have to use out of bound (ie, other vars) to tell the eclass when it shouldn't mark a flag as runtime toggable. If it's not stacked by the pm, then they have to manually stack; that differs from the norm and makes it easier to screwup; however; does allow for them to filter, albeit a slight pain in the ass doing so. There's a choice there, and the answer matters, so yes, you should actually have a complete glep before trying to shove it up to the council and extract a vote out of them. Lest the intention is to just have them kick it back to the curb... It can't be stacked and it's not wise to do so; it is a simple bash variable like tons of others in eclasses: It cannot be stacked because y'all are trying to shove this in as an optional; unlike it's brother IUSE, which stacks. As for ons of others that don't stack; very few actually influence the package manager; ~14 roughly, minimally 5 of those stack (those that
Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
On Wed, Sep 26, 2012 at 5:59 PM, Michael Mol mike...@gmail.com wrote: On Wed, Sep 26, 2012 at 5:49 PM, Chí-Thanh Christopher Nguyễn chith...@gentoo.org wrote: Michael Mol schrieb: A few months ago, I filed bug 423651 to ask that bzip2 on the install media be replaced with pbzip2. If I understand correctly, pbzip2 depends on bzip2. So what you are asking is that pbzip2 is preferred over bzip2 when both are installed, and that pbzip2 is installed by default? pbzip2 uses libbzip2, which I understand bzip2 to also be a wrapper around. libbz2 is built and installed by the app-arch/bzip2 package. Thus, app-arch/pbzip2 depends on app-arch/bzip2, unless someone rips libbz2 out into a separate ebuild.
Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
2012/9/26 Mike Gilbert flop...@gentoo.org: On Wed, Sep 26, 2012 at 5:59 PM, Michael Mol mike...@gmail.com wrote: On Wed, Sep 26, 2012 at 5:49 PM, Chí-Thanh Christopher Nguyễn chith...@gentoo.org wrote: Michael Mol schrieb: A few months ago, I filed bug 423651 to ask that bzip2 on the install media be replaced with pbzip2. If I understand correctly, pbzip2 depends on bzip2. So what you are asking is that pbzip2 is preferred over bzip2 when both are installed, and that pbzip2 is installed by default? pbzip2 uses libbzip2, which I understand bzip2 to also be a wrapper around. libbz2 is built and installed by the app-arch/bzip2 package. Thus, app-arch/pbzip2 depends on app-arch/bzip2, unless someone rips libbz2 out into a separate ebuild. That sound like a plan. Maybe bzip2 should become a virtual as busybox also provides an implementation. -- Christoph Junghans http://dev.gentoo.org/~ottxor/
Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
On Wed, Sep 26, 2012 at 6:57 PM, Christoph Junghans ott...@gentoo.org wrote: 2012/9/26 Mike Gilbert flop...@gentoo.org: On Wed, Sep 26, 2012 at 5:59 PM, Michael Mol mike...@gmail.com wrote: On Wed, Sep 26, 2012 at 5:49 PM, Chí-Thanh Christopher Nguyễn chith...@gentoo.org wrote: Michael Mol schrieb: A few months ago, I filed bug 423651 to ask that bzip2 on the install media be replaced with pbzip2. If I understand correctly, pbzip2 depends on bzip2. So what you are asking is that pbzip2 is preferred over bzip2 when both are installed, and that pbzip2 is installed by default? pbzip2 uses libbzip2, which I understand bzip2 to also be a wrapper around. libbz2 is built and installed by the app-arch/bzip2 package. Thus, app-arch/pbzip2 depends on app-arch/bzip2, unless someone rips libbz2 out into a separate ebuild. That sound like a plan. Maybe bzip2 should become a virtual as busybox also provides an implementation. This makes sense. And going back to my initial issue, I don't really care which implementation gets used on the bootable media, so long as it supports scaling to use my CPU cores. -- :wq
Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
On 26/09/2012 15:57, Christoph Junghans wrote: That sound like a plan. Maybe bzip2 should become a virtual as busybox also provides an implementation. No, just, no. -- Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/
