Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
On 1 November 2014 00:18, Rich Freeman ri...@gentoo.org wrote: So, if there is a better way, I'm all ears for constructive suggestions. By constructive I mean that somebody who comes up with a script that automatically retrieves build logs and attaches them to bugs is being more helpful than somebody who says that somebody else should come up with such a script, and so on. That doesn't mean that we can't talk about solutions before we build them - only that it isn't helpful when we basically demand that others build them for us. Let me try to be clear (once again) of what the problem is. Even just spending time rewriting this irks me because I have written, repeated, re-explained, described the problem a number of times, between bugs, this ML and my blog ( https://blog.flameeyes.eu/tag/tinderbox ). The problem with it's trivial to do that in python so just do it is that first of all Python is not my language of choice, so the whole infrastructure is currently not written in Python at all. And all the people, including Luca, who promised they can convert it to Python in no time, never delivered. Beside the point that, if it's so trivial for somebody, I would expect it'll take them less time to provide me with the tool, rather than complain about it. Another suggestion that happens fairly often is oh just do it after opening the bug with a script, and the answer is - no way. Because right now it's all self contained in a browser for good reasons: I open these bugs when I'm at work waiting for a meeting, at home waiting to sleep, or while watching TV. I do it from laptops and tablets, and if I have to start copy-pasting things between browser and console to run a script, I'd rather just leave things broken because it *is* too much work by the amount of bugs I open. And it's not something I can do on a tablet or at work. Which would mean that not only it'll take me more time , and I would also have less time available. Not a good deal. Since the bugs are opened through a pre-filled form template, it's not easy for anything else to know what the bug is to attach logs to. One alternative would be to have the app that I use to show me the logs file the bug entirely for me. Unfortunately that means I'll have to figure out how to secure the app, as right now, being just a display-only thing, it's completely open to the Internet. And I'm sure Infra would rather not have a tool open to the internet that can open bugs with logs of many MBs attached. It will also slightly lower the quality of bugs because either it needs to build its own form to fill in for summary and blocking bugs, or it'll just go with the two summaries I know for sure how to get from the log (fails to build, fails tests), so no more foo[doc] fails to build or foo fails to build with ncurses[tinfo]. But let's reason a moment on the no-linked-logs policy: as Rich pointed out already, the policy is there for a reason and that reason is that we don't want people to submit bugs with pastebins or home server logs because they are bound to go away. I link logs to Amazon, which I pay for. Mike says it's unreasonable to expect me to pay Amazon for them forever... I pay $1/mo for storing viewing an adding the logs. And even that turned out to be actually an excess as I was paying between $.12 and $.20 for storing EBS snapshots from long ago that I never ended up removing. It's not even a rounding error at this point. Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
On 1 November 2014 10:47, Diego Elio Pettenò flamee...@flameeyes.eu wrote: But let's reason a moment on the no-linked-logs policy: as Rich pointed out already, the policy is there for a reason and that reason is that we don't want people to submit bugs with pastebins or home server logs because they are bound to go away. I link logs to Amazon, which I pay for. Mike says it's unreasonable to expect me to pay Amazon for them forever... I pay $1/mo for storing viewing an adding the logs. And even that turned out to be actually an excess as I was paying between $.12 and $.20 for storing EBS snapshots from long ago that I never ended up removing. It's not even a rounding error at this point. And I forgot to add, to run the tinderbox, hosting the server costs me $600/mo — and sure, I use it to stage the packages for a bunch of my private servers too, and it runs fate instances for libav. But you can probably note how the two numbers don't even come *close* to each other. Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/
Re: [gentoo-dev] Removing 'selinux? ( sec-policy/selinux-*)' from DEPEND
On Mon, Sep 01, 2014 at 11:26:49PM +0200, Tom Wijsman wrote: [...] With this change, we implement the same end result (correctly labeled files after installation) while removing the need for the DEPEND dependency. After all, this was not a build-time dependency but a merge-time one, which we abused a bit to make things work. With this change in place, we can now update the tree (at least, for those packages that do not have other SELinux related dependency requirements - those that link with libselinux still need it in DEPEND of course) to remove the USE=selinux conditional dependency from DEPEND. Given the discussion on dynamic dependencies and so, I am thinking about doing this as follows: 1. Create a tracker with separate bugs for every package where this change can be made 2. Give developers time to apply this (simple) change together with whatever other changes they were planning. 3. After 6 months or so, do the change myself (with revbump) [...] Is this a good approach to take? [...] LGTM; we should avoid unnecessary bumps rebuilds for trivial changes, especially when a USE flag based dependency line is removed from DEPEND. Michał Górny told me on IRC that I might be approaching this incorrectly (or at least, inefficiently). I was working on the massive bug-spree (right now stopped around 22% of the packages to investigate) so I'm temporarily holding off until I'm certain. The only change I want to instill on packages is to remove the USE=selinux specific dependency to a sec-policy/selinux-* package from the DEPEND variable. So something like: DEPEND= foo - bar - selinux? ( sec-policy/selinux-bez ) + bar If I am allowed to do this change without revbumping, I can just stop making massive bug reports and do the change(s) myself... Someone? Pretty-please? Wkr, Sven Vermeulen
Re: [gentoo-dev] Removing 'selinux? ( sec-policy/selinux-*)' from DEPEND
Dnia 2014-11-01, o godz. 12:17:22 Sven Vermeulen sw...@gentoo.org napisał(a): On Mon, Sep 01, 2014 at 11:26:49PM +0200, Tom Wijsman wrote: [...] With this change, we implement the same end result (correctly labeled files after installation) while removing the need for the DEPEND dependency. After all, this was not a build-time dependency but a merge-time one, which we abused a bit to make things work. With this change in place, we can now update the tree (at least, for those packages that do not have other SELinux related dependency requirements - those that link with libselinux still need it in DEPEND of course) to remove the USE=selinux conditional dependency from DEPEND. Given the discussion on dynamic dependencies and so, I am thinking about doing this as follows: 1. Create a tracker with separate bugs for every package where this change can be made 2. Give developers time to apply this (simple) change together with whatever other changes they were planning. 3. After 6 months or so, do the change myself (with revbump) [...] Is this a good approach to take? [...] LGTM; we should avoid unnecessary bumps rebuilds for trivial changes, especially when a USE flag based dependency line is removed from DEPEND. Michał Górny told me on IRC that I might be approaching this incorrectly (or at least, inefficiently). I was working on the massive bug-spree (right now stopped around 22% of the packages to investigate) so I'm temporarily holding off until I'm certain. The only change I want to instill on packages is to remove the USE=selinux specific dependency to a sec-policy/selinux-* package from the DEPEND variable. So something like: DEPEND= foo - bar - selinux? ( sec-policy/selinux-bez ) + bar If I am allowed to do this change without revbumping, I can just stop making massive bug reports and do the change(s) myself... You should have emphasized that the dependency will still be in RDEPEND. As I said with QA hat on, such a change is fine since it affects build-time dependencies only. People who installed the package already are not affected. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
El sáb, 01-11-2014 a las 10:47 +, Diego Elio Pettenò escribió: On 1 November 2014 00:18, Rich Freeman ri...@gentoo.org wrote: So, if there is a better way, I'm all ears for constructive suggestions. By constructive I mean that somebody who comes up with a script that automatically retrieves build logs and attaches them to bugs is being more helpful than somebody who says that somebody else should come up with such a script, and so on. That doesn't mean that we can't talk about solutions before we build them - only that it isn't helpful when we basically demand that others build them for us. Let me try to be clear (once again) of what the problem is. Even just spending time rewriting this irks me because I have written, repeated, re-explained, described the problem a number of times, between bugs, this ML and my blog ( https://blog.flameeyes.eu/tag/tinderbox ). [...] Would be possible to change the script opening the bug reports to CC somebody to the bugs assigned to vapier? For example, when metadata.xml causes the bug assigned to him, the script could additionally CC other people (like me) that could handle the build.log stuff
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
On 11/01/2014 01:18 AM, Rich Freeman wrote: What we can't do is force somebody to contribute. If somebody says that if we don't do multilib their way, they'll stop being the only libreoffice maintainer, and nobody else wants to maintain libreoffice, then we are left in a hard place (completely contrived scenario). Someone who doesn't want to contribute shouldn't have commit access in the first place. And I think that's currently not the case. However, people who regularly block or even just ignore contribution processes/channels are a serious problem. More so if they are in non-trivial positions. Whether they do it on purpose or not is secondary. But this is definitely not about technicalities (like should lua have a pkgconfig file or not). Every time this gets brought up someone says oh, but we can't do anything, because someone might be sad about it and quit gentoo. Uhm, right. That's already happening, all the time. What is worse is taking this argument as an excuse to not fix our contribution culture, which is very bad on a lot of fronts. And every time an incident like this happens... it gets even worse. To be blunt with you, I think this is a matter of people having so much respect for someone that they almost tolerate any misbehaviour. And this behaviour spreads. It's a phenomenon we know from people like Linus. Because he is smart after all and his insults are pretty funny. It's not easy to emancipate oneself from the negativity in the opensource world. It isn't caused by a particular group of people, but by tolerating certain patterns of behaviour. But there will be no improvement if we don't take such issues more seriously. I don't really see that happening. It's something the oldtimers have more power over than the council.
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
On Sat, Nov 1, 2014 at 11:38 AM, hasufell hasuf...@gentoo.org wrote: But there will be no improvement if we don't take such issues more seriously. I don't really see that happening. It's something the oldtimers have more power over than the council. In a community project, the folks with power are those doing the work. The newest member of Gentoo can have more power to direct the course of the distro than every oldtimer or council member there is, if they just contribute more than them. If the maintainer of package A or provider of service B is a pain to work with, all it takes is for somebody else who is easier to work with to maintain package A or provide service B. In the absence of this, the only thing the Council can do is ask infra to revoke somebody's commit/bugzilla/whatever access, which basically leaves us in the same place that we'd be if folks quit, but I suppose with the emotional satisfaction of being the one pushing everybody around. So, I'm more than happy when somebody steps up like axs and volunteers to start converting links to attachments, or when somebody steps up to take care of some @system package that only has one maintainer. Otherwise, the best the council can really hope to do is keep everybody out of each other's way, and we do that reasonably well. After the last flare-up like this the Council voted that anybody can maintain games without making them a part of the games herd. So, there are no barriers, but that isn't going to magically make a bunch of games show up in the tree. A year ago the Council voted that maintainers have to stay out of the way if the systemd team wants to add units to packages, and while it wasn't an issue the same would be true about anybody wanting to add an openrc init.d script to a package. That doesn't magically make units or init.d scripts show up, but it clears away blockades. At some point when behavior is egregious you have to say enough is enough, but somebody simply being unwilling to fix a bug or perform a free service for Gentoo isn't really an offense in and of itself - we get far more noise from the quarterly systemd flamewar. Personally, I'd prefer less drama, but this is the distro which was among the first to support systemd, and the only one (AFAIK) to make eudev an option. If we're going to be about choice, then to some extent we need to deal with the natural conflict that leads to. Hopefully we can all see that we're better off giving an inch so that we can reap the benefits of each other's contributions... -- Rich
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
On 01/11/14 11:47, Diego Elio Pettenò wrote: The problem with it's trivial to do that in python so just do it is that first of all Python is not my language of choice, so the whole infrastructure is currently not written in Python at all. And all the people, including Luca, who promised they can convert it to Python in no time, never delivered. Beside the point that, if it's so trivial for somebody, I would expect it'll take them less time to provide me with the tool, rather than complain about it. Sorry I hadn't time to do that and everything fell low in the todolist since the tinderbox was put on hold. That said, we have enough people with python skills could we just pick few days and come up with a good solution? lu
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
The newest member of Gentoo can have more power to direct the course of the distro than every oldtimer or council member there is, if they just contribute more than them. If the maintainer of package A or provider of service B is a pain to work with, all it takes is for somebody else who is easier to work with to maintain package A or provide service B. Rich, I fully agree with the overall sentiment of the rest of your e-mail, but I think above statements are just not true. For both appropriate and inappropriate reasons. On one hand, if you are new to the project, nobody knows (and trusts) you yet. So, it's definitely not the best idea to immediately propose big plans to reform everything. You talk to people, show that you can handle things, earn trust, and one day some time later noone will complain if you alone properly and carefully handle restructuring the profile tree. On the other hand, except for peaceful and cooperative places (kde team comes to my mind since that's where I grew up as a Gentoo dev, but I'm sure there are more examples), if as a newbie you pick the wrong things to work on you might as well immediately retire again- you'll get blocked out by territoriality. If you try to push things, well there's always someone who has the idea to invoke QA or comrel. [Let's retire him, (he might be making sense but) he's making way too much noise. Luckily, that usually just does't happen.] This has become much better in the recent past, but it's not ideal yet. I would be very glad if all our technical geniuses working deep on the insides of Gentoo were to adopt new devs and teach them the intricacies. I would also be very glad if all our technical geniuses were to participate in IRC and mailing list discussions, so we can learn from them and understand why they do things. Yes I know it is occasionally a waste of time and I can well understand that after X years in Gentoo your enthusiasm for it disappears. No, that does not make it unnecessary. Yes, in my opinion we are scaring off people. No, adapting to all misbehaviour of current contributors is not an option. Noone is irreplaceable. -- Andreas K. Huettel Gentoo Linux developer dilfri...@gentoo.org http://www.akhuettel.de/ signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 31/10/14 10:37 PM, Ian Stakenvicius wrote: Sorry for top posting. I volunteer to write something to get the logs attached to bugs. I'll do it next week. Whether it be something the tinderbox can run or something separate that will use pybugz to find bugs and then attach whatever logs are pointed to by URLs, etc, I'll figure it out. OK, i've cobbled something together that looks like it'll work. I'll be putting this thing on a cronjob on one of my systems, and hopefully it'll be able to keep up with Diego's tinderbox runs and bug filing with as little delay as possible. I've already started running it on all bugs filed by Diego after October 20th. If anybody sees any issues with the attached build.log's, please let me know by adding the bug with the issue to a tracker I made for this purpose, bug 527870 Thanks, Ian -BEGIN PGP SIGNATURE- Version: GnuPG v2 iF4EAREIAAYFAlRVLHoACgkQ2ugaI38ACPDiJQD/SwRyd9AW0xtKxetZI44J4ifl aqCrIqZz979axZJeIWAA/iU11C28o2E9KeSK13CoRPW89+YSjk3bORlB2CMMJUxq =t8bX -END PGP SIGNATURE-
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
Ian Stakenvicius wrote: OK, i've cobbled something together that looks like it'll work. Cool! Thanks a lot for doing that. //Peter
Re: [gentoo-dev] Removing 'selinux? ( sec-policy/selinux-*)' from DEPEND
On Sat, Nov 01, 2014 at 01:52:57PM +0100, Michał Górny wrote: Michał Górny told me on IRC that I might be approaching this incorrectly (or at least, inefficiently). I was working on the massive bug-spree (right now stopped around 22% of the packages to investigate) so I'm temporarily holding off until I'm certain. The only change I want to instill on packages is to remove the USE=selinux specific dependency to a sec-policy/selinux-* package from the DEPEND variable. So something like: DEPEND= foo - bar - selinux? ( sec-policy/selinux-bez ) + bar If I am allowed to do this change without revbumping, I can just stop making massive bug reports and do the change(s) myself... You should have emphasized that the dependency will still be in RDEPEND. As I said with QA hat on, such a change is fine since it affects build-time dependencies only. People who installed the package already are not affected. Thanks. I'll do the necessary updates tomorrow then (without revbump) and invalidate the bug reports I already made. Wkr, Sven Vermeulen
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
El sáb, 01-11-2014 a las 14:54 -0400, Ian Stakenvicius escribió: On 31/10/14 10:37 PM, Ian Stakenvicius wrote: Sorry for top posting. I volunteer to write something to get the logs attached to bugs. I'll do it next week. Whether it be something the tinderbox can run or something separate that will use pybugz to find bugs and then attach whatever logs are pointed to by URLs, etc, I'll figure it out. OK, i've cobbled something together that looks like it'll work. I'll be putting this thing on a cronjob on one of my systems, and hopefully it'll be able to keep up with Diego's tinderbox runs and bug filing with as little delay as possible. I've already started running it on all bugs filed by Diego after October 20th. If anybody sees any issues with the attached build.log's, please let me know by adding the bug with the issue to a tracker I made for this purpose, bug 527870 Thanks, Ian At least for the bugs I have read, it looks to work nice. Thanks a lot :D Is the script placed somewhere? (for learning purposes :)) Thanks
Re: [gentoo-dev] Removing 'selinux? ( sec-policy/selinux-*)' from DEPEND
On 11/01/2014 03:36 PM, Sven Vermeulen wrote: On Sat, Nov 01, 2014 at 01:52:57PM +0100, Michał Górny wrote: Michał Górny told me on IRC that I might be approaching this incorrectly (or at least, inefficiently). I was working on the massive bug-spree (right now stopped around 22% of the packages to investigate) so I'm temporarily holding off until I'm certain. The only change I want to instill on packages is to remove the USE=selinux specific dependency to a sec-policy/selinux-* package from the DEPEND variable. So something like: DEPEND= foo - bar - selinux? ( sec-policy/selinux-bez ) + bar If I am allowed to do this change without revbumping, I can just stop making massive bug reports and do the change(s) myself... You should have emphasized that the dependency will still be in RDEPEND. As I said with QA hat on, such a change is fine since it affects build-time dependencies only. People who installed the package already are not affected. Thanks. I'll do the necessary updates tomorrow then (without revbump) and invalidate the bug reports I already made. hat=QA Just since you poked me on irc and I tend to yell at anyone who breaks to dep tree by making RDEPEND changes without revbump I agree with mgorny. I don't believe this change will cause any issues with the dep tree for people who aren't, or cannot, run dynamic deps. Please proceed to make your changes as desired, without revbump, and you may close your bugs. Thanks, Zero/hat Wkr, Sven Vermeulen signature.asc Description: OpenPGP digital signature
[gentoo-dev] [RFC] qt4-build-multilib.eclass
Hello fellow devs, Please review the attached qt4-build-multilib.eclass ASAP. It's basically the current qt4-build.eclass with some old cruft removed and multilib support added. Given the amount of changes needed for multilib, we chose to introduce a separate eclass rather than modifying the existing one, to avoid destabilizing current consumers and to maintain code readability. While at it, we also refactored the code quite a bit, and rewrote some functions using a more modern style. Thanks, Davide Pesavento qt4-build-multilib.eclass Description: Binary data
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 01/11/14 03:53 PM, Pacho Ramos wrote: El sáb, 01-11-2014 a las 14:54 -0400, Ian Stakenvicius escribió: On 31/10/14 10:37 PM, Ian Stakenvicius wrote: Sorry for top posting. I volunteer to write something to get the logs attached to bugs. I'll do it next week. Whether it be something the tinderbox can run or something separate that will use pybugz to find bugs and then attach whatever logs are pointed to by URLs, etc, I'll figure it out. OK, i've cobbled something together that looks like it'll work. I'll be putting this thing on a cronjob on one of my systems, and hopefully it'll be able to keep up with Diego's tinderbox runs and bug filing with as little delay as possible. I've already started running it on all bugs filed by Diego after October 20th. If anybody sees any issues with the attached build.log's, please let me know by adding the bug with the issue to a tracker I made for this purpose, bug 527870 Thanks, Ian At least for the bugs I have read, it looks to work nice. Thanks a lot :D Is the script placed somewhere? (for learning purposes :)) Thanks Not at the moment ; I'm going to want to do a fair bit of cleanup before i let anyone else's eyes on it.. :) It's just a bash script, though -- i use 'pybugz search' to get a list of the bugs Diego's filed (using --offset and --limit to restrict the list returned), get the content of each one, and if it has no attachments but has a URL with 'https://tinderboxlogs' in the comment, then i wget the URL to a temp file, strip the HTML out, and attach it (compressing if necessary to save space) to the bug with 'pybugz attach'. Add in a couple of files that lets me keep track of the current search offset i should be using as well as the bug#'s the script has processed and skipped, and that's about it. -BEGIN PGP SIGNATURE- Version: GnuPG v2 iF4EAREIAAYFAlRVhG8ACgkQ2ugaI38ACPBRIAD+J0eZvjhRBcQ2KfxWKs3AozBL dhCZQ+sUCwtIb5dpemgA/RiwAabfqsG41o8CsitjIbzz/Q2MBZC4r/3q/3rkgP4H =LHOX -END PGP SIGNATURE-
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
On Sat, Nov 1, 2014 at 2:30 PM, Andreas K. Huettel dilfri...@gentoo.org wrote: The newest member of Gentoo can have more power to direct the course of the distro than every oldtimer or council member there is, if they just contribute more than them. If the maintainer of package A or provider of service B is a pain to work with, all it takes is for somebody else who is easier to work with to maintain package A or provide service B. Rich, I fully agree with the overall sentiment of the rest of your e-mail, but I think above statements are just not true. For both appropriate and inappropriate reasons. Well, I also get what you're saying, but I'm not sure that this is the best place to draw the line... On the other hand, except for peaceful and cooperative places (kde team comes to my mind since that's where I grew up as a Gentoo dev, but I'm sure there are more examples), if as a newbie you pick the wrong things to work on you might as well immediately retire again- you'll get blocked out by territoriality. If you try to push things, well there's always someone who has the idea to invoke QA or comrel. [Let's retire him, (he might be making sense but) he's making way too much noise. Luckily, that usually just does't happen.] This has become much better in the recent past, but it's not ideal yet. Well, nothing is ever ideal, but as issues come up they are being dealt with now. I can't think of any situations where somebody has been able to block out new contributors in the last year or two. Sure, there have been a few attempts, but we've squashed them. There is a lot we can do in the case of territoriality. In such a case we have somebody who is contributing, and all we need to do is declare that their contributions are to be accepted. There really is nothing anybody can do to stop somebody from contributing except reverts/etc, and doing that after the council establishes policy is going to lead to losing commit privs. Fortunately, it hasn't come to that in quite a while. I think that when push comes to shove people who are standing in the way come to appreciate the situation they're trying to promote. However, this is not really the same sort of situation. If somebody was trying to submit their own tinderbox bugs and Diego was telling them that he alone is allowed to run a tinderbox, then that would be territoriality. Such a move would really be silly though - people build stuff and submit logs in bugs all the time, and a tinderbox is just doing that on a larger scale. Likewise, if somebody was offering Mike patches to fix the bugs Diego is reporting and Mike was unjustifiably turning them away, or especially if he was combative with other devs willing to support those patches (and the patches were reasonable), then that too would be territoriality, and all we need to do is get Mike to stand aside. Neither of these hypotheticals really pertains here. Diego isn't stopping anybody else from submitting bugs in whatever format they wish, and Mike isn't preventing anybody from fixing bugs. Their actual technical contributions in these cases are net-positive, or near-zero at worst (a dev closing a bug that they aren't obligated to fix doesn't actually harm anybody unless somebody else was going to come along and fix it). Socially it would be nice if we could all compromise, but that is harder to deal with. It looks like axs has a workaround nearly ready which is likely to make this issue somewhat moot. I do agree that nobody is indispensable. If there are specific situations that really stress people to the point of quitting I would like to hear about them BEFORE people throw in the towel. In the end, though, we do want to have a distro and not just a polite mailing list, and that means that we need to appreciate everybody's (often-silent) positive contributions, and not just focus on their role in some recent conflict. -- Rich
Re: [gentoo-portage-dev] Re: [PATCH] emerge --search: use description index
On 10/23/2014 02:22 AM, Zac Medico wrote: In the future, we might decide to rewrite the search class so that it processes the index as a stream, which will allow individual search results to be displayed as soon as they are located [1]. This rewrite will require a new index API. So, considering that we probably want a new index API in the future, we might save some unnecessary effort and stick with the special-purpose/minimalistic IndexedPortdb/Vardb adapters for now. [1] https://bugs.gentoo.org/show_bug.cgi?id=412471 I have a stream-based back end working in this branch: https://github.com/zmedico/portage/commits/bug_525718_stream Now all that's left to do is to convert IndexedPortdb to expose an iter_cp_all method, and to fix the search class to iterate over iter_cp_all and display results incrementally. Once that's done, I'll think about exposing useful pieces as public APIs. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] Add btrfs.* to default PORTAGE_XATTR_EXCLUDE.
X-Gentoo-Bug: 527636
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=527636
---
cnf/make.globals | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/cnf/make.globals b/cnf/make.globals
index 013c556..dd99618 100644
--- a/cnf/make.globals
+++ b/cnf/make.globals
@@ -122,9 +122,10 @@ PORTAGE_ELOG_MAILFROM=portage@localhost
# Signing command used by repoman
PORTAGE_GPG_SIGNING_COMMAND=gpg --sign --digest-algo SHA256 --clearsign --yes
--default-key \\${PORTAGE_GPG_KEY}\ --homedir \\${PORTAGE_GPG_DIR}\
\\${FILE}\
+# btrfs.* attributes are irrelevant, see bug #527636.
# Security labels are special, see bug #461868.
# system.nfs4_acl attributes are irrelevant, see bug #475496.
-PORTAGE_XATTR_EXCLUDE=security.* system.nfs4_acl
+PORTAGE_XATTR_EXCLUDE=btrfs.* security.* system.nfs4_acl
#*
#** DO NOT EDIT THIS FILE **
--
2.0.4
[gentoo-portage-dev] [PATCH 2/5] Add IndexStreamIterator and MultiIterGroupBy.
This IndexStreamIterator class can be used together with the
pkg_desc_index_line_read function to read and index file incrementally
as a stream.
The MultiIterGroupBy class can be used to iterate over multiple
IndexStreamIterator instances at once, incrementally grouping results
for a particular package from multiple indices, while limiting the
amount of any given index that must be in memory at once.
Both of these classes are used by the IndexedPortdb class in the next
patch of this series.
X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
---
pym/portage/cache/index/IndexStreamIterator.py | 27 +
pym/portage/util/iterators/MultiIterGroupBy.py | 82 ++
pym/portage/util/iterators/__init__.py | 2 +
3 files changed, 111 insertions(+)
create mode 100644 pym/portage/cache/index/IndexStreamIterator.py
create mode 100644 pym/portage/util/iterators/MultiIterGroupBy.py
create mode 100644 pym/portage/util/iterators/__init__.py
diff --git a/pym/portage/cache/index/IndexStreamIterator.py
b/pym/portage/cache/index/IndexStreamIterator.py
new file mode 100644
index 000..972aee1
--- /dev/null
+++ b/pym/portage/cache/index/IndexStreamIterator.py
@@ -0,0 +1,27 @@
+# Copyright 2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+class IndexStreamIterator(object):
+
+ def __init__(self, f, parser):
+
+ self.parser = parser
+ self._file = f
+
+ def close(self):
+
+ if self._file is not None:
+ self._file.close()
+ self._file = None
+
+ def __iter__(self):
+
+ try:
+
+ for line in self._file:
+ node = self.parser(line)
+ if node is not None:
+ yield node
+
+ finally:
+ self.close()
diff --git a/pym/portage/util/iterators/MultiIterGroupBy.py
b/pym/portage/util/iterators/MultiIterGroupBy.py
new file mode 100644
index 000..d4e62ad
--- /dev/null
+++ b/pym/portage/util/iterators/MultiIterGroupBy.py
@@ -0,0 +1,82 @@
+# Copyright 2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+class MultiIterGroupBy(object):
+
+ This class functions similarly to the itertools.groupby function,
+ except that it takes multiple source iterators as input. The source
+ iterators must yield objects in sorted order. A group is yielded as
+ soon as the progress of all iterators reaches a state which
+ guarantees that there can not be any remaining (unseen) elements of
+ the group. This is useful for incremental display of grouped search
+ results.
+
+
+ def __init__(self, iterators, key = None):
+ self._iterators = iterators
+ self._key = key
+
+ def __iter__(self):
+
+ progress = []
+ iterators = self._iterators[:]
+ for index in iterators:
+ progress.append(None)
+
+ key_map = {}
+ eof = []
+ key_getter = self._key
+ if key_getter is None:
+ key_getter = lambda x: x
+ max_progress = None
+
+ while iterators:
+ min_progress = None
+ for i, index in enumerate(iterators):
+
+ if max_progress is not None and \
+ max_progress == progress[i] and \
+ min_progress is not None and \
+ max_progress != min_progress:
+ # This one has the most progress,
+ # so allow the others to catch up.
+ continue
+
+ for entry in index:
+ progress[i] = key_getter(entry)
+ key_group =
key_map.get(key_getter(entry))
+ if key_group is None:
+ key_group = []
+ key_map[key_getter(entry)] =
key_group
+
+ key_group.append(entry)
+
+ if min_progress is None or \
+ key_getter(entry)
min_progress:
+ min_progress = key_getter(entry)
+
+ if max_progress is None or \
+ key_getter(entry) =
max_progress:
+ max_progress = key_getter(entry)
+
[gentoo-portage-dev] [PATCH 4/5] Add IndexedVardb class.
Searching of installed packages is optimized to take advantage of
vardbdbapi._aux_cache, which is backed by vdb_metadata.pickle.
This class only implements a subset of vardbapi functionality that is
useful for searching incrementally. For this reason, the cp_all method
returns an ordered iterator instead of a list, so that search results
can be displayed incrementally.
X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
---
pym/portage/dbapi/IndexedVardb.py | 87 +++
pym/portage/dbapi/vartree.py | 19 +++--
2 files changed, 102 insertions(+), 4 deletions(-)
create mode 100644 pym/portage/dbapi/IndexedVardb.py
diff --git a/pym/portage/dbapi/IndexedVardb.py
b/pym/portage/dbapi/IndexedVardb.py
new file mode 100644
index 000..b2d894b
--- /dev/null
+++ b/pym/portage/dbapi/IndexedVardb.py
@@ -0,0 +1,87 @@
+# Copyright 2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import portage
+from portage.dep import Atom
+from portage.versions import _pkg_str
+
+class IndexedVardb(object):
+
+ A vardbapi interface that sacrifices validation in order to
+ improve performance. It takes advantage of vardbdbapi._aux_cache,
+ which is backed by vdb_metadata.pickle. Since _aux_cache is
+ not updated for every single merge/unmerge (see
+ _aux_cache_threshold), the list of packages is obtained directly
+ from the real vardbapi instance. If a package is missing from
+ _aux_cache, then its metadata is obtained using the normal
+ (validated) vardbapi.aux_get method.
+
+ For performance reasons, the match method only supports package
+ name and version constraints.
+
+
+ _copy_attrs = ('cpv_exists',
+ '_aux_cache_keys', '_cpv_sort_ascending')
+
+ def __init__(self, vardb):
+ self._vardb = vardb
+
+ for k in self._copy_attrs:
+ setattr(self, k, getattr(vardb, k))
+
+ self._cp_map = None
+
+ def cp_all(self):
+
+ Returns an ordered iterator instead of a list, so that search
+ results can be displayed incrementally.
+
+ if self._cp_map is not None:
+ return iter(sorted(self._cp_map))
+
+ return self._iter_cp_all()
+
+ def _iter_cp_all(self):
+ self._cp_map = cp_map = {}
+ previous_cp = None
+ for cpv in self._vardb._iter_cpv_all(sort = True):
+ cp = portage.cpv_getkey(cpv)
+ if cp is not None:
+ cp_list = cp_map.get(cp)
+ if cp_list is None:
+ cp_list = []
+ cp_map[cp] = cp_list
+ cp_list.append(_pkg_str(cpv))
+ if previous_cp is not None and \
+ previous_cp != cp:
+ yield previous_cp
+ previous_cp = cp
+
+ if previous_cp is not None:
+ yield previous_cp
+
+ def match(self, atom):
+
+ For performance reasons, only package name and version
+ constraints are supported.
+
+ if not isinstance(atom, Atom):
+ atom = Atom(atom)
+ cp_list = self._cp_map.get(atom.cp)
+ if cp_list is None:
+ return []
+ self._vardb._cpv_sort_ascending(cp_list)
+ return portage.match_from_list(atom, cp_list)
+
+ def aux_get(self, cpv, attrs, myrepo = None):
+ pkg_data = self._vardb._aux_cache[packages].get(cpv)
+ if not isinstance(pkg_data, tuple) or \
+ len(pkg_data) != 2 or \
+ not isinstance(pkg_data[1], dict):
+ pkg_data = None
+ if pkg_data is None:
+ # It may be missing from _aux_cache due to
+ # _aux_cache_threshold.
+ return self._vardb.aux_get(cpv, attrs)
+ metadata = pkg_data[1]
+ return [metadata.get(k, ) for k in attrs]
diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
index e21135a..37504e8 100644
--- a/pym/portage/dbapi/vartree.py
+++ b/pym/portage/dbapi/vartree.py
@@ -422,6 +422,9 @@ class vardbapi(dbapi):
(generally this is only necessary in critical sections that
involve merge or unmerge of packages).
+ return list(self._iter_cpv_all(use_cache=use_cache))
+
+ def _iter_cpv_all(self, use_cache = True, sort = False):
returnme = []
[gentoo-portage-dev]
In addition to indexed search, this patch series implements stream-based incremental display of search results (fixing bug #412471). I'll be maintaining this patch series in the following branch: https://github.com/zmedico/portage/tree/bug_525718_stream The old non-incremental version of this patch series is still available here: https://github.com/zmedico/portage/tree/bug_525718
[gentoo-portage-dev] [PATCH 5/5] Add emerge --search-index option.
The new emerge --search-index option, which is enabled by default,
causes pkg_desc_index to be used for search optimization. The search
index needs to be regenerated by egencache after changes are made to
a repository (see the --update-pkg-desc-index action).
For users that would like to modify ebuilds in a repository without
running egencache afterwards, emerge --search-index=n can be used to
get non-indexed search. Alternatively, the user could simply remove
the stale index file, in order to disable the search index for a
particular repository.
In order to conserve memory, indices are read as streams, and
MultiIterGroupBy is used to group results from IndexedPortdb and
IndexedVardb. Stream-oriented search also makes it possible to
display search results incrementally (fixing bug #412471).
X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
---
man/emerge.1| 8
pym/_emerge/actions.py | 3 +-
pym/_emerge/depgraph.py | 2 +-
pym/_emerge/main.py | 5 +++
pym/_emerge/search.py | 112 ++--
5 files changed, 95 insertions(+), 35 deletions(-)
diff --git a/man/emerge.1 b/man/emerge.1
index bbe71ac..7bcdd9a 100644
--- a/man/emerge.1
+++ b/man/emerge.1
@@ -796,6 +796,14 @@ If ebuilds using EAPIs which \fIdo not\fR support
\fBHDEPEND\fR are built in
the same \fBemerge\fR run as those using EAPIs which \fIdo\fR support
\fBHDEPEND\fR, this option affects only the former.
.TP
+.BR \-\-search\-index y | n
+Enable or disable indexed search for search actions. This option is
+enabled by default. The search index needs to be regenerated by
+\fBegencache\fR(1) after changes are made to a repository (see the
+\fB\-\-update\-pkg\-desc\-index\fR action). This setting can be added
+to \fBEMERGE_DEFAULT_OPTS\fR (see \fBmake.conf\fR(5)) and later
+overridden via the command line.
+.TP
.BR \-\-select [ y | n ] (\-w short option)
Add specified packages to the world set (inverse of
\fB\-\-oneshot\fR). This is useful if you want to
diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py
index 48b0826..8a22ab5 100644
--- a/pym/_emerge/actions.py
+++ b/pym/_emerge/actions.py
@@ -2015,7 +2015,8 @@ def action_search(root_config, myopts, myfiles, spinner):
searchinstance = search(root_config,
spinner, --searchdesc in myopts,
--quiet not in myopts, --usepkg in myopts,
- --usepkgonly in myopts)
+ --usepkgonly in myopts,
+ search_index = myopts.get(--search-index, y) != n)
for mysearch in myfiles:
try:
searchinstance.execute(mysearch)
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index 78b9236..2fbb7ce 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -8596,7 +8596,7 @@ def ambiguous_package_name(arg, atoms, root_config,
spinner, myopts):
s = search(root_config, spinner, --searchdesc in myopts,
--quiet not in myopts, --usepkg in myopts,
- --usepkgonly in myopts)
+ --usepkgonly in myopts, search_index = False)
null_cp = portage.dep_getkey(insert_category_into_atom(
arg, null))
cat, atom_pn = portage.catsplit(null_cp)
diff --git a/pym/_emerge/main.py b/pym/_emerge/main.py
index cf7966c..c08e12a 100644
--- a/pym/_emerge/main.py
+++ b/pym/_emerge/main.py
@@ -616,6 +616,11 @@ def parse_opts(tmpcmdline, silent=False):
choices :(True, rdeps)
},
+ --search-index: {
+ help: Enable or disable indexed search (enabled by
default),
+ choices: y_or_n
+ },
+
--select: {
shortopt : -w,
help: add specified packages to the world set
+ \
diff --git a/pym/_emerge/search.py b/pym/_emerge/search.py
index 4b0fd9f..acde3bd 100644
--- a/pym/_emerge/search.py
+++ b/pym/_emerge/search.py
@@ -7,9 +7,12 @@ import re
import portage
from portage import os
from portage.dbapi.porttree import _parse_uri_map
+from portage.dbapi.IndexedPortdb import IndexedPortdb
+from portage.dbapi.IndexedVardb import IndexedVardb
from portage.localization import localized_size
from portage.output import bold, bold as white, darkgreen, green, red
from portage.util import writemsg_stdout
+from portage.util.iterators.MultiIterGroupBy import MultiIterGroupBy
from _emerge.Package import Package
@@ -25,15 +28,17 @@ class search(object):
# public interface
#
def __init__(self, root_config, spinner, searchdesc,
- verbose, usepkg, usepkgonly):
+ verbose, usepkg, usepkgonly, search_index = True):
Searches the available and installed packages for the
supplied search key.
The list
[gentoo-portage-dev] [PATCH 1/5] Add egencache --update-pkg-desc-index action.
This adds an egencache --update-pkg-desc-index action which generates
a plain-text index of package names, versions, and descriptions. The
index can then be used to optimize emerge --search / --searchdesc
actions.
X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
---
bin/egencache | 38 ++--
man/egencache.1 | 4 +++
man/portage.5 | 12
pym/portage/cache/index/__init__.py | 2 ++
pym/portage/cache/index/pkg_desc_index.py | 48 +++
5 files changed, 102 insertions(+), 2 deletions(-)
create mode 100644 pym/portage/cache/index/__init__.py
create mode 100644 pym/portage/cache/index/pkg_desc_index.py
diff --git a/bin/egencache b/bin/egencache
index e366058..f97432f 100755
--- a/bin/egencache
+++ b/bin/egencache
@@ -48,6 +48,7 @@ portage._internal_caller = True
from portage import os, _encodings, _unicode_encode, _unicode_decode
from _emerge.MetadataRegen import MetadataRegen
from portage.cache.cache_errors import CacheError, StatCollision
+from portage.cache.index.pkg_desc_index import pkg_desc_index_line_format
from portage.const import TIMESTAMP_FORMAT
from portage.manifest import guessManifestFileType
from portage.package.ebuild._parallel_manifest.ManifestScheduler import
ManifestScheduler
@@ -57,7 +58,7 @@ from portage.util._async.run_main_scheduler import
run_main_scheduler
from portage.util._eventloop.global_event_loop import global_event_loop
from portage import cpv_getkey
from portage.dep import Atom, isjustname
-from portage.versions import pkgsplit, vercmp
+from portage.versions import pkgsplit, vercmp, _pkg_str
try:
from xml.etree import ElementTree
@@ -91,6 +92,9 @@ def parse_args(args):
actions.add_argument(--update-changelogs,
action=store_true,
help=update the ChangeLog files from SCM logs)
+ actions.add_argument(--update-pkg-desc-index,
+ action=store_true,
+ help=update package description index)
actions.add_argument(--update-manifests,
action=store_true,
help=update manifests)
@@ -451,6 +455,29 @@ class GenCache(object):
if hasattr(trg_cache, '_prune_empty_dirs'):
trg_cache._prune_empty_dirs()
+class GenPkgDescIndex(object):
+ def __init__(self, portdb, output_file):
+ self.returncode = os.EX_OK
+ self._portdb = portdb
+ self._output_file = output_file
+
+ def run(self):
+
+ portage.util.ensure_dirs(os.path.dirname(self._output_file))
+ f = portage.util.atomic_ofstream(self._output_file,
+ encoding=_encodings[repo.content])
+
+ portdb = self._portdb
+ for cp in portdb.cp_all():
+ pkgs = portdb.cp_list(cp)
+ if not pkgs:
+ continue
+ desc, = portdb.aux_get(pkgs[-1], [DESCRIPTION])
+
+ f.write(pkg_desc_index_line_format(cp, pkgs, desc))
+
+ f.close()
+
class GenUseLocalDesc(object):
def __init__(self, portdb, output=None,
preserve_comments=False):
@@ -893,7 +920,8 @@ def egencache_main(args):
local_config=False, env=env)
if not (options.update or options.update_use_local_desc or
- options.update_changelogs or options.update_manifests):
+ options.update_changelogs or options.update_manifests or
+ options.update_pkg_desc_index):
parser.error('No action specified')
return 1
@@ -1057,6 +1085,12 @@ def egencache_main(args):
else:
ret.append(scheduler.returncode)
+ if options.update_pkg_desc_index:
+ gen_index = GenPkgDescIndex(portdb, os.path.join(
+ repo_config.location, metadata, pkg_desc_index))
+ gen_index.run()
+ ret.append(gen_index.returncode)
+
if options.update_use_local_desc:
gen_desc = GenUseLocalDesc(portdb,
output=options.uld_output,
diff --git a/man/egencache.1 b/man/egencache.1
index f71feb3..3a3197f 100644
--- a/man/egencache.1
+++ b/man/egencache.1
@@ -19,6 +19,10 @@ for the details on package atom syntax.
.BR \-\-update\-changelogs
Update the ChangeLog files from SCM logs (supported only in git repos).
.TP
+.BR \-\-update\-pkg\-desc\-index
+Update the package description index which is located at
+\fImetadata/pkg_desc_index\fR in the repository.
+.TP
.BR \-\-update\-use\-local\-desc
Update the \fIprofiles/use.local.desc\fR file from metadata.xml.
.TP
diff --git a/man/portage.5 b/man/portage.5
index 309e259..f2f5243 100644
---
[gentoo-portage-dev] [PATCH 3/5] Add IndexedPortdb class.
The IndexedPortdb class uses pkg_desc_index to optimize searchs for
package names and descriptions. If the package description index is
missing from a particular repository, then all metadata for that
repository is obtained using the normal pordbapi.aux_get method.
This class only implements a subset of portdbapi functionality that is
useful for searching pkg_desc_index incrementally. For this reason,
the cp_all method returns an ordered iterator instead of a list, so
that search results can be displayed incrementally.
X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
---
pym/portage/dbapi/IndexedPortdb.py | 151 +
1 file changed, 151 insertions(+)
create mode 100644 pym/portage/dbapi/IndexedPortdb.py
diff --git a/pym/portage/dbapi/IndexedPortdb.py
b/pym/portage/dbapi/IndexedPortdb.py
new file mode 100644
index 000..4fb2cf1
--- /dev/null
+++ b/pym/portage/dbapi/IndexedPortdb.py
@@ -0,0 +1,151 @@
+# Copyright 2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import errno
+import io
+import functools
+import operator
+import os
+
+import portage
+from portage import _encodings
+from portage.dep import Atom
+from portage.exception import FileNotFound
+from portage.cache.index.IndexStreamIterator import IndexStreamIterator
+from portage.cache.index.pkg_desc_index import pkg_desc_index_line_read
+from portage.util.iterators.MultiIterGroupBy import MultiIterGroupBy
+from portage.versions import _pkg_str
+
+class IndexedPortdb(object):
+
+ A portdbapi interface that uses a package description index to
+ improve performance. If the description index is missing for a
+ particular repository, then all metadata for that repository is
+ obtained using the normal pordbapi.aux_get method.
+
+ For performance reasons, the match method only supports package
+ name and version constraints. For the same reason, the xmatch
+ method is not implemented.
+
+
+ _copy_attrs = ('cpv_exists', 'findname', 'getFetchMap',
+ '_aux_cache_keys', '_cpv_sort_ascending',
+ '_have_root_eclass_dir')
+
+ def __init__(self, portdb):
+
+ self._portdb = portdb
+
+ for k in self._copy_attrs:
+ setattr(self, k, getattr(portdb, k))
+
+ self._desc_cache = None
+ self._cp_map = None
+
+ def _init_index(self):
+
+ cp_map = {}
+ desc_cache = {}
+ self._desc_cache = desc_cache
+ self._cp_map = cp_map
+
+ streams = []
+ for repo_path in self._portdb.porttrees:
+ outside_repo = os.path.join(self._portdb.depcachedir,
+ repo_path.lstrip(os.sep))
+ filenames = []
+ for parent_dir in (repo_path, outside_repo):
+ filenames.append(os.path.join(parent_dir,
+ metadata, pkg_desc_index))
+
+ repo_name = self._portdb.getRepositoryName(repo_path)
+
+ try:
+ f = None
+ for filename in filenames:
+ try:
+ f = io.open(filename,
+
encoding=_encodings[repo.content])
+ except IOError as e:
+ if e.errno not in
(errno.ENOENT, errno.ESTALE):
+ raise
+ else:
+ break
+
+ if f is None:
+ raise FileNotFound(filename)
+
+ streams.append(iter(IndexStreamIterator(f,
+
functools.partial(pkg_desc_index_line_read,
+ repo = repo_name
+ except FileNotFound:
+
+ # No descriptions index was found, so populate
+ # cp_map the slow way.
+ for cp in
self._portdb.cp_all(trees=[repo_path]):
+
+ cp_list = cp_map.get(cp)
+ if cp_list is None:
+ cp_list = []
+ cp_map[cp] = cp_list
+ for cpv in self._portdb.cp_list(
+ cp, mytree = repo_path):
+ cp_list.append(_pkg_str(cpv,
repo = repo_name))
+
+ # Create a
[gentoo-portage-dev] [PATCH 5/5] Add emerge --search-index option.
The new emerge --search-index option, which is enabled by default,
causes pkg_desc_index to be used for search optimization. The search
index needs to be regenerated by egencache after changes are made to
a repository (see the --update-pkg-desc-index action).
For users that would like to modify ebuilds in a repository without
running egencache afterwards, emerge --search-index=n can be used to
get non-indexed search. Alternatively, the user could simply remove
the stale index file, in order to disable the search index for a
particular repository.
In order to conserve memory, indices are read as streams, and
MultiIterGroupBy is used to group results from IndexedPortdb and
IndexedVardb. Stream-oriented search also makes it possible to
display search results incrementally (fixing bug #412471).
X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
---
This updated patch fixes the search.output method to reset the match
count after each search is performed, in case there are multiple
searches.
man/emerge.1| 8
pym/_emerge/actions.py | 3 +-
pym/_emerge/depgraph.py | 2 +-
pym/_emerge/main.py | 5 ++
pym/_emerge/search.py | 119 ++--
5 files changed, 102 insertions(+), 35 deletions(-)
diff --git a/man/emerge.1 b/man/emerge.1
index bbe71ac..7bcdd9a 100644
--- a/man/emerge.1
+++ b/man/emerge.1
@@ -796,6 +796,14 @@ If ebuilds using EAPIs which \fIdo not\fR support
\fBHDEPEND\fR are built in
the same \fBemerge\fR run as those using EAPIs which \fIdo\fR support
\fBHDEPEND\fR, this option affects only the former.
.TP
+.BR \-\-search\-index y | n
+Enable or disable indexed search for search actions. This option is
+enabled by default. The search index needs to be regenerated by
+\fBegencache\fR(1) after changes are made to a repository (see the
+\fB\-\-update\-pkg\-desc\-index\fR action). This setting can be added
+to \fBEMERGE_DEFAULT_OPTS\fR (see \fBmake.conf\fR(5)) and later
+overridden via the command line.
+.TP
.BR \-\-select [ y | n ] (\-w short option)
Add specified packages to the world set (inverse of
\fB\-\-oneshot\fR). This is useful if you want to
diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py
index 48b0826..8a22ab5 100644
--- a/pym/_emerge/actions.py
+++ b/pym/_emerge/actions.py
@@ -2015,7 +2015,8 @@ def action_search(root_config, myopts, myfiles, spinner):
searchinstance = search(root_config,
spinner, --searchdesc in myopts,
--quiet not in myopts, --usepkg in myopts,
- --usepkgonly in myopts)
+ --usepkgonly in myopts,
+ search_index = myopts.get(--search-index, y) != n)
for mysearch in myfiles:
try:
searchinstance.execute(mysearch)
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index 78b9236..2fbb7ce 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -8596,7 +8596,7 @@ def ambiguous_package_name(arg, atoms, root_config,
spinner, myopts):
s = search(root_config, spinner, --searchdesc in myopts,
--quiet not in myopts, --usepkg in myopts,
- --usepkgonly in myopts)
+ --usepkgonly in myopts, search_index = False)
null_cp = portage.dep_getkey(insert_category_into_atom(
arg, null))
cat, atom_pn = portage.catsplit(null_cp)
diff --git a/pym/_emerge/main.py b/pym/_emerge/main.py
index cf7966c..c08e12a 100644
--- a/pym/_emerge/main.py
+++ b/pym/_emerge/main.py
@@ -616,6 +616,11 @@ def parse_opts(tmpcmdline, silent=False):
choices :(True, rdeps)
},
+ --search-index: {
+ help: Enable or disable indexed search (enabled by
default),
+ choices: y_or_n
+ },
+
--select: {
shortopt : -w,
help: add specified packages to the world set
+ \
diff --git a/pym/_emerge/search.py b/pym/_emerge/search.py
index 4b0fd9f..1d710ee 100644
--- a/pym/_emerge/search.py
+++ b/pym/_emerge/search.py
@@ -7,9 +7,12 @@ import re
import portage
from portage import os
from portage.dbapi.porttree import _parse_uri_map
+from portage.dbapi.IndexedPortdb import IndexedPortdb
+from portage.dbapi.IndexedVardb import IndexedVardb
from portage.localization import localized_size
from portage.output import bold, bold as white, darkgreen, green, red
from portage.util import writemsg_stdout
+from portage.util.iterators.MultiIterGroupBy import MultiIterGroupBy
from _emerge.Package import Package
@@ -25,15 +28,17 @@ class search(object):
# public interface
#
def __init__(self, root_config, spinner, searchdesc,
- verbose, usepkg, usepkgonly):
+ verbose, usepkg,
[gentoo-portage-dev] [PATCH 2/5] Add IndexStreamIterator and MultiIterGroupBy.
This IndexStreamIterator class can be used together with the
pkg_desc_index_line_read function to read an index file incrementally
as a stream.
The MultiIterGroupBy class can be used to iterate over multiple
IndexStreamIterator instances at once, incrementally grouping results
for a particular package from multiple indices, while limiting the
amount of any given index that must be in memory at once.
Both of these classes are used by the IndexedPortdb class in the next
patch of this series.
X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
---
This updated patch includes a logic fix to ensure that all buffered
objects are yielded when all of the iterators are exhausted.
pym/portage/cache/index/IndexStreamIterator.py | 27 +
pym/portage/util/iterators/MultiIterGroupBy.py | 82 ++
pym/portage/util/iterators/__init__.py | 2 +
3 files changed, 111 insertions(+)
create mode 100644 pym/portage/cache/index/IndexStreamIterator.py
create mode 100644 pym/portage/util/iterators/MultiIterGroupBy.py
create mode 100644 pym/portage/util/iterators/__init__.py
diff --git a/pym/portage/cache/index/IndexStreamIterator.py
b/pym/portage/cache/index/IndexStreamIterator.py
new file mode 100644
index 000..972aee1
--- /dev/null
+++ b/pym/portage/cache/index/IndexStreamIterator.py
@@ -0,0 +1,27 @@
+# Copyright 2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+class IndexStreamIterator(object):
+
+ def __init__(self, f, parser):
+
+ self.parser = parser
+ self._file = f
+
+ def close(self):
+
+ if self._file is not None:
+ self._file.close()
+ self._file = None
+
+ def __iter__(self):
+
+ try:
+
+ for line in self._file:
+ node = self.parser(line)
+ if node is not None:
+ yield node
+
+ finally:
+ self.close()
diff --git a/pym/portage/util/iterators/MultiIterGroupBy.py
b/pym/portage/util/iterators/MultiIterGroupBy.py
new file mode 100644
index 000..ece7a4c
--- /dev/null
+++ b/pym/portage/util/iterators/MultiIterGroupBy.py
@@ -0,0 +1,82 @@
+# Copyright 2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+class MultiIterGroupBy(object):
+
+ This class functions similarly to the itertools.groupby function,
+ except that it takes multiple source iterators as input. The source
+ iterators must yield objects in sorted order. A group is yielded as
+ soon as the progress of all iterators reaches a state which
+ guarantees that there can not be any remaining (unseen) elements of
+ the group. This is useful for incremental display of grouped search
+ results.
+
+
+ def __init__(self, iterators, key = None):
+ self._iterators = iterators
+ self._key = key
+
+ def __iter__(self):
+
+ progress = []
+ iterators = self._iterators[:]
+ for index in iterators:
+ progress.append(None)
+
+ key_map = {}
+ eof = []
+ key_getter = self._key
+ if key_getter is None:
+ key_getter = lambda x: x
+ max_progress = None
+
+ while iterators:
+ min_progress = None
+ for i, index in enumerate(iterators):
+
+ if max_progress is not None and \
+ max_progress == progress[i] and \
+ min_progress is not None and \
+ max_progress != min_progress:
+ # This one has the most progress,
+ # so allow the others to catch up.
+ continue
+
+ for entry in index:
+ progress[i] = key_getter(entry)
+ key_group =
key_map.get(key_getter(entry))
+ if key_group is None:
+ key_group = []
+ key_map[key_getter(entry)] =
key_group
+
+ key_group.append(entry)
+
+ if min_progress is None or \
+ key_getter(entry)
min_progress:
+ min_progress = key_getter(entry)
+
+ if max_progress is None or \
+
Re: [gentoo-portage-dev] [PATCH] Add btrfs.* to default PORTAGE_XATTR_EXCLUDE.
On Sat, 1 Nov 2014 14:05:01 -0700
Zac Medico zmed...@gentoo.org wrote:
X-Gentoo-Bug: 527636
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=527636
Can you add a little more info why to the commit message please.
also can you add the bug # to the end of first line, it makes it easier
for tracking release news changes, etc.
Add btrfs.* to default PORTAGE_XATTR_EXCLUDE (527636)
otherwise is ok.
---
cnf/make.globals | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/cnf/make.globals b/cnf/make.globals
index 013c556..dd99618 100644
--- a/cnf/make.globals
+++ b/cnf/make.globals
@@ -122,9 +122,10 @@ PORTAGE_ELOG_MAILFROM=portage@localhost
# Signing command used by repoman
PORTAGE_GPG_SIGNING_COMMAND=gpg --sign --digest-algo SHA256
--clearsign --yes --default-key \\${PORTAGE_GPG_KEY}\ --homedir
\\${PORTAGE_GPG_DIR}\ \\${FILE}\
+# btrfs.* attributes are irrelevant, see bug #527636.
# Security
labels are special, see bug #461868. # system.nfs4_acl attributes are
irrelevant, see bug #475496. -PORTAGE_XATTR_EXCLUDE=security.*
system.nfs4_acl +PORTAGE_XATTR_EXCLUDE=btrfs.* security.*
system.nfs4_acl
#*
#** DO NOT EDIT THIS FILE **
--
Brian Dolbec dolsen
