Re: [HACKERS] 8.4 release planning
Merlin, Not completely. HS is in much better shape than win32 was when it was pulled from 7.4...the build system wasn't even in place yet nor any of the major challenges solved (like fork/exec). HS is working very well (Simon's ongoing work aside). I am pretty confident based on my personal testing that it would represent the project well if committed today. OK. I haven't tried it since early December; that's why I was just presenting alternatives. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, Jan 26, 2009 at 2:36 PM, Josh Berkus j...@agliodbs.com wrote: All, So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I'm convinced that if we took a staw poll, 80% of our users would be in favor of waiting for HS. This one feature will make more of a difference in the number of PG users than any feature since the Windows port. Maybe more. if we think we will need a lot of time to get this in form, there will be no difference in the time of the release just in the numbers in which HS comes SE-Linux: this patch has effectively been in development for 2 years ourside the core process before putting it in; the forked SEPostgres is in use in production. KaiGai has been available for 20 hours a week (or more) to troubleshoot issues and change APIs. I really don't see what the problem is with committing it. it hasn't been testing by ours in different platforms (ie: ubuntu has selinux and i want to give it a try, badly enough i have never used selinux so this is new to me)... nor we have any evidence that it doesn't affect to users that doesn't have any variant of selinux (ie: windows)... the real problem here is the base of users we enough knowledge of the tool to make some usefull tests == Regarding the Commitfests in general: we still haven't perfected this process. There are a number of issues with it which are hampered by technology, but a lot more by people. Here's my analysis of what's changed over the last year: 1) having the last CF on Nov. 1 was a mistake. That put us square in the path of the US Christian holidays during the critical integration phase .. which means we haven't really had 3 months of integration, we've had *two*. +1 -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
All, 1) having the last CF on Nov. 1 was a mistake. That put us square in the path of the US Christian holidays during the critical integration phase .. which means we haven't really had 3 months of integration, we've had *two*. Actually, I'm thinking about this again, and made a mistake about the mistake. The *original plan* was that we were not going to accept any new patches for Nov-CF. Just revised patches from eariler Fests. We didn't stick to that, which is mostly why we are still reviewing now. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Josh Berkus j...@agliodbs.com writes: So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I don't think this is correct. There are certainly a lot of users who would like an in-core replication solution, but HS by itself is not that --- you also need (near) real-time log shipping, which we have already decided to punt to 8.5. That being the case, I think the argument that HS is a must-have feature for 8.4 is actually rather weak. SE-Linux: this patch has effectively been in development for 2 years ourside the core process before putting it in; the forked SEPostgres is in use in production. KaiGai has been available for 20 hours a week (or more) to troubleshoot issues and change APIs. I really don't see what the problem is with committing it. The problem, in words of one syllable, is that we are not sure we want it. Do you see a user community clamoring for SEPostgres, or a hacker community that is willing or able to maintain it? If KaiGai-san got run over by a bus tomorrow, this patch would be a dead letter, because there just isn't anyone else who is taking sufficient (any?) interest in it. That doesn't bode well for its future viability. Compare the likely audience for it to previous patches of roughly similar complexity, such as integrated text search or the Windows port, and it's just not in the ballpark. The second problem is that we're not sure it's really the right thing, because we have no one who is competent to review the design from a security standpoint. But unless we get past the first problem the second one is moot. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Merlin Moncure mmonc...@gmail.com writes: HS is working very well (Simon's ongoing work aside). I am pretty confident based on my personal testing that it would represent the project well if committed today. I think a lot of people weren't aware there was anybody testing this patch other than Simon and Heikki -- I wasn't until just today. I wonder how many more people are trying it out? This is one of the changes for the better from the past. Though I still think a lot more would try it out once it's committed. Here's a thought experiment. If it was committable *today* would we be willing to go with it and plan to release with it? Assume that it would *still* mean a longer beta process, so it would still mean releasing in, say April instead of February or March. While personally I want us to switch to a mode where we commit large patches early in the development cycle I don't believe we would refuse to commit it today if it was ready. And I can't imagine two weeks would make the difference either. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL training! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On 1/26/09, Josh Berkus j...@agliodbs.com wrote: All, 1) having the last CF on Nov. 1 was a mistake. That put us square in the path of the US Christian holidays during the critical integration phase .. which means we haven't really had 3 months of integration, we've had *two*. Actually, I'm thinking about this again, and made a mistake about the mistake. The *original plan* was that we were not going to accept any new patches for Nov-CF. Just revised patches from eariler Fests. We didn't stick to that, which is mostly why we are still reviewing now. I don't recall us discussing that, but it sounds like it might help next cycle. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On 1/26/09, Gregory Stark st...@enterprisedb.com wrote: Merlin Moncure mmonc...@gmail.com writes: HS is working very well (Simon's ongoing work aside). I am pretty confident based on my personal testing that it would represent the project well if committed today. I think a lot of people weren't aware there was anybody testing this patch other than Simon and Heikki -- I wasn't until just today. I wonder how many more people are trying it out? This is one of the changes for the better from the past. Though I still think a lot more would try it out once it's committed. Here's a thought experiment. If it was committable *today* would we be willing to go with it and plan to release with it? Assume that it would *still* mean a longer beta process, so it would still mean releasing in, say April instead of February or March. I think the best person to answer that is Simon. He got a small reprieve the last time this discussion came up, but it's 'put up or shut up time'...time's up. Is the thing ready to go? While i'm 'pro HS', I'm also very much 'pro get 8.4 out asap'. I think both _may_ be possible. This is probably horribly optimistic. What I can do however is offer to summarize my own experiences testing and invite others to do the same. I'd also like to see Simon and or Heikki make a strong statement on where things stand _right now_ (not in two weeks) :-) merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Gregory Stark st...@enterprisedb.com writes: Here's a thought experiment. If it was committable *today* would we be willing to go with it and plan to release with it? Assume that it would *still* mean a longer beta process, so it would still mean releasing in, say April instead of February or March. April instead of February or March? If we cut off *all* further patch acceptance *today*, we might conceivably be ready to release May 1. It is not physically possible to get to beta from where we are before mid-February, given the open issues, lack of release notes, and already-scheduled back-branch updates that will be consuming some of core's time this week. And I doubt anyone thinks we need less than 2 months in beta. (Or did you mean April of 2010?) While personally I want us to switch to a mode where we commit large patches early in the development cycle I don't believe we would refuse to commit it today if it was ready. And I can't imagine two weeks would make the difference either. Well, if it were ready, we'd not be having this discussion. The problem is that it's not ready and no one is very sure about when it will be. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 20:12 +, Gregory Stark wrote: Merlin Moncure mmonc...@gmail.com writes: HS is working very well (Simon's ongoing work aside). I am pretty confident based on my personal testing that it would represent the project well if committed today. I think a lot of people weren't aware there was anybody testing this patch other than Simon and Heikki -- I wasn't until just today. I wonder how many more people are trying it out? OK, so maybe its been unclear: Gianni Ciolli has been leading a test team on this, with other people swapping in and out as they go to work on other things. There's been multiple test machines running various kinds of test on different OS. Gianni and I have done many midnight sessions together on this and his help and support has been invaluable. If some bugs have got passed that test screening in recent weeks, I think that's down to the sheer volume of tests and my own descriptions of how things ought to work and exactly where to focus. Gianni has located more issues than Heikki has, just. Hannu provided a lot of early input on the problems of conflict resolution and various alternatives. I'm the main developer, but I don't underrate their contributions. I've had specific bug reports from 6 people and test feedback/questions from about another 4-5 people; Mark isn't credited because I wasn't keeping track of them in earl stages of review. Recent refactoring has been a rich source of regressions and those don't go away overnight, but they don't take months either. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I don't think this is correct. There are certainly a lot of users who would like an in-core replication solution, but HS by itself is not that --- you also need (near) real-time log shipping, which we have already decided to punt to 8.5. That being the case, I think the argument that HS is a must-have feature for 8.4 is actually rather weak. SE-Linux: this patch has effectively been in development for 2 years ourside the core process before putting it in; the forked SEPostgres is in use in production. KaiGai has been available for 20 hours a week (or more) to troubleshoot issues and change APIs. I really don't see what the problem is with committing it. The problem, in words of one syllable, is that we are not sure we want it. Do you see a user community clamoring for SEPostgres, or a hacker community that is willing or able to maintain it? If KaiGai-san got run over by a bus tomorrow, this patch would be a dead letter, because there just isn't anyone else who is taking sufficient (any?) interest in it. That doesn't bode well for its future viability. Compare the likely audience for it to previous patches of roughly similar complexity, such as integrated text search or the Windows port, and it's just not in the ballpark. The second problem is that we're not sure it's really the right thing, because we have no one who is competent to review the design from a security standpoint. But unless we get past the first problem the second one is moot. I've never posted to this list before, but I am an SELinux upstream maintainer. I'd just like to interject here, we (the SELinux community) are very interested in KaiGai's work and have been looking forward to it being upstreamed for quite some time. While we haven't been able to analyze the patches directly to determine whether the security goals are indeed being met we have had much discussion and eventually community agreement on the security model being implemented. This happened years ago and has since been merged into the SELinux reference policy that practically all SELinux users use (distributions start with the reference policy and add rules/modules suitable for them). So the security model has been looked at, though not the implementation and we do have a community of developers, users and customers interested in this work. Joshua Brindle -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 15:47 -0500, Merlin Moncure wrote: I'd also like to see Simon and or Heikki make a strong statement on where things stand _right now_ (not in two weeks) :-) Well, we just found 2 bugs over the weekend, one of which is a regression from refactoring. The second bug is being fixed as part of requested refactoring from Heikki, discussed today. I've insisted on full visibility, so all known bugs have been recorded on Wiki as soon as they are reported, even internally located ones. Most have been fixed on same day found. Check out the Wiki, any day. I still have not completed Prepared Transaction support, but if I had then it just would have slowed down the earlier rework. This is probably the only hanging offence. Please remember that Heikki's rework proposals were still under discussion only 2 weeks ago. Personally, I've very happy with the code as stands, generally, but Heikki may wish to move a few things around yet. The rework has meant I've reread most of my own code, so I have found and fixed many of my own bugs and rewritten comments to better explain things. I found, reported and fixed the issue of drop tablespace for example. If cynical, I could have hidden that and waited a month for people to find it. Going Beta will reveal further feedback on usability, so we might expect a few tweaks there as people moan. We'll see. I won't put a time limit on this since people will just add their own fudge factors to that and we'll be no nearer to a true figure. Without everybody breathing down my neck it might be 2-3 days, but I'm spending time reading email and trying to keep my patches alive. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 15:49 -0500, Tom Lane wrote: The problem is that it's not ready and no one is very sure about when it will be. With respect, I've done more than any other developer has done to give you and the community full information on the patch as it develops. I'm sorry you don't believe what I tell you, but if you don't you should see the patch for yourself or ask the person reviewing it. The Wiki has been there for 4 months now, updated daily for last 8 weeks apart from illness. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Josh Berkus wrote: All, So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I'm convinced that if we took a staw poll, 80% of our users would be in favor of waiting for HS. This one feature will make more of a difference in the number of PG users than any feature since the Windows port. Maybe more. on the other hand: We held back version 4 months 7.4 for Windows, before it became apparent that there was at least a year more work to do. That was a mistake, and in many ways HS seems like a similar case. I can only confirm what Josh is saying here. We would also assume that 80% have been waiting for Simon's work for years. In fact, I have been dealing fulltime with PostgreSQL since 1999 and it has been a missing issue since than. Now that we are so close to fixing this issue for so many people out there, we should give it all the attention we have and support Simon + team wherever we can. I think Simon has responded to all question is almost realtime. We should take that into consideration. Also, Simon is focuing on a very open development model - this naturally means a lot of mailing list traffic. Isn't this what this project is all about? I am in favor of giving this patch a useful timeframe for completion. If people decide to give this patch a chance, we will definitely agree on putting some significant manpower in here as well. We are not the only ones who want to see that in. We already see people saying that they delay migrations because they are hoping for readable slaves to go in. Also, in the past 10 years I have been tortured with when can we have replication each and every day ... I am fed up :). i cannot hear it anymore (... but MySQL has replication *aargh*). best regards, hans -- -- Cybertec Schönig Schönig GmbH Professional PostgreSQL Consulting, Support, Training Gröhrmühlgasse 26, A-2700 Wiener Neustadt Web: www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Monday 26 January 2009 2:12:02 pm Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I don't think this is correct. There are certainly a lot of users who would like an in-core replication solution, but HS by itself is not that --- you also need (near) real-time log shipping, which we have already decided to punt to 8.5. Precisely. Thank you. That being the case, I think the argument that HS is a must-have feature for 8.4 is actually rather weak. I am just one person, representing one company...but I do agree, fwiw. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On 1/26/09 4:28 PM, Joshua Brindle met...@manicmethod.com wrote: Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: snip SE-Linux: this patch has effectively been in development for 2 years ourside the core process before putting it in; the forked SEPostgres is in use in production. KaiGai has been available for 20 hours a week (or more) to troubleshoot issues and change APIs. I really don't see what the problem is with committing it. The problem, in words of one syllable, is that we are not sure we want it. Do you see a user community clamoring for SEPostgres, or a hacker community that is willing or able to maintain it? If KaiGai-san got run over by a bus tomorrow, this patch would be a dead letter, because there just isn't anyone else who is taking sufficient (any?) interest in it. That doesn't bode well for its future viability. Compare the likely audience for it to previous patches of roughly similar complexity, such as integrated text search or the Windows port, and it's just not in the ballpark. The second problem is that we're not sure it's really the right thing, because we have no one who is competent to review the design from a security standpoint. But unless we get past the first problem the second one is moot. I've never posted to this list before, but I am an SELinux upstream maintainer. I'd just like to interject here, we (the SELinux community) are very interested in KaiGai's work and have been looking forward to it being upstreamed for quite some time. While we haven't been able to analyze the patches directly to determine whether the security goals are indeed being met we have had much discussion and eventually community agreement on the security model being implemented. This happened years ago and has since been merged into the SELinux reference policy that practically all SELinux users use (distributions start with the reference policy and add rules/modules suitable for them). So the security model has been looked at, though not the implementation and we do have a community of developers, users and customers interested in this work. I'd just like to echo Josh's sentiments. I'm also active in the SELinux community, and have been involved in several developments that really needed a database with mandatory access control mechanisms. Unfortunately, these developments have all had maintenance requirements that precluded using KaiGai's code as it was outside not in a commercial distribution. We've been waiting anxiously for it to be merged upstream. Additionally, I've talked to many other end users that really want to deploy LAPP stacks with these security features. They often came to us looking for us to help them build such systems, but we've had to turn them away as there was no supported way to build it. Thanks, Chad Sellers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Gregory Stark wrote: I think a lot of people weren't aware there was anybody testing this patch other than Simon and Heikki -- I wasn't until just today. I wonder how many more people are trying it out? I've been running the patch (I think since Jan 5) on a couple dev instances that were used primarily to test if our software worked with what we expected to turn into 8.4. I can't say I've really been testing the patch specifically, but rather testing my workplace's software against a version with this patch, and hadn't noticed any problems. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Gregory Stark wrote: I think a lot of people weren't aware there was anybody testing this patch ...I wonder how many more people are trying it out? I think I have an idea to improve this aspect for future commit fests. For a long time at each of my workplaces I've been running a development instance against CVS-HEAD just to make sure our software is more future-proof against up-and-coming releases. We run this system with -enable-debug, asserts, etc, and accept that it's just a development system not expected to be totally stable. If it were just as easy for us to pull from a all 'pending-patches' for-commit-fest-nov that pass regression tests branch, I'd happily pull from that instead. I realize in the current system (emailed patches), this would be a horrible pain to maintain such a branch; but perhaps some of the burden could be pushed down to the patch submitters (asking them to merge their own changes into this merged branch). And I hate bringing up the version control flame war again; but git really would make this easier. If all patches were on their own branches; the painful merges into this shared branch would be rare, as the source control system would remember the painful parts of the merges. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Ron Mayer rm...@cheapcomplexdevices.com writes: I realize in the current system (emailed patches), this would be a horrible pain to maintain such a branch; but perhaps some of the burden could be pushed down to the patch submitters (asking them to merge their own changes into this merged branch). I've considered maintaining such a repository a few times and dismissed it when I realized how much work it would be to maintain. And I hate bringing up the version control flame war again; but git really would make this easier. If all patches were on their own branches; the painful merges into this shared branch would be rare, as the source control system would remember the painful parts of the merges. We have git repositories, I still think maintaining a merged tree with dozens of patches would be a lot of work. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production Tuning -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
* Tom Lane (t...@sss.pgh.pa.us) wrote: Josh Berkus j...@agliodbs.com writes: Users: care about HS more than anything else in the world. I don't think this is correct. There are certainly a lot of users who would like an in-core replication solution, but HS by itself is not that --- you also need (near) real-time log shipping, which we have already decided to punt to 8.5. That being the case, I think the argument that HS is a must-have feature for 8.4 is actually rather weak. HS would still be really nice, and I'm a bit concerned that we'll end up in 8.5 with a similar discussion along the lines of well, we've only just committed HS, why don't we release 8.5 with that and wait on replication till 8.6. I agree that more folks are after replication than HS, but there is still a user base for it. SE-Linux: this patch has effectively been in development for 2 years ourside the core process before putting it in; the forked SEPostgres is in use in production. KaiGai has been available for 20 hours a week (or more) to troubleshoot issues and change APIs. I really don't see what the problem is with committing it. The problem, in words of one syllable, is that we are not sure we want it. Do you see a user community clamoring for SEPostgres, or a hacker community that is willing or able to maintain it? If KaiGai-san got run over by a bus tomorrow, this patch would be a dead letter, because there just isn't anyone else who is taking sufficient (any?) interest in it. I'm certainly interested in it and would like to see it in core. Would I use it immediately? No, because I've so far avoided having to have it for my systems. I don't expect that to last forever though, at some point I'll have to implement a system which requires the seperation provided by SELinux or move to something like Solaris Trusted Extensions and Oracle Label Security. That doesn't bode well for its future viability. Compare the likely audience for it to previous patches of roughly similar complexity, such as integrated text search or the Windows port, and it's just not in the ballpark. No, it doesn't have as large a user base as the Windows port or integrated text search. On the other hand, there *are* users out there, and hackers, who are willing and interested in it for PostgreSQL because it would give them an alternative to the de-facto standards. Perhaps it's just me, but historically it seems like there's also been a fair bit of aid given to trusted/SE implementations by various organizations who need it, both in terms of funding for others to work on it and in actual direct development. I realize this is a trivially defeated POV, but I really feel that 'if you build it, they will come' in this case. The second problem is that we're not sure it's really the right thing, because we have no one who is competent to review the design from a security standpoint. But unless we get past the first problem the second one is moot. I think the database design for SELinux is pretty well defined.. The implementation needs to be reviewed, but I think there's few who can do that, unfortunately. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
Stephen Frost sfr...@snowman.net writes: * Tom Lane (t...@sss.pgh.pa.us) wrote: The problem, in words of one syllable, is that we are not sure we want it. Do you see a user community clamoring for SEPostgres, or a hacker community that is willing or able to maintain it? No, it doesn't have as large a user base as the Windows port or integrated text search. On the other hand, there *are* users out there, and hackers, who are willing and interested in it for PostgreSQL because it would give them an alternative to the de-facto standards. Then why has *nobody* stepped up to review the design, much less the whole patch? The plain truth is that no one appears to care enough to expend any real effort. But this patch is far too large and invasive to accept on the basis that only one guy understands it and will/might continue to maintain it. I'll risk being rude to make my point: those who want SEPostgres in core need to put up or shut up. Now, not at some future time. We need people to sign off that this patch implements the features they want (not sounds roughly like some vague future need I might have) and does so correctly. An incorrect security feature is considerably worse than useless. And once it's in core we aren't going to have a whole lot of elbow room to change the definition later. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: The problem, in words of one syllable, is that we are not sure we want it. Do you see a user community clamoring for SEPostgres, or a hacker This is a chicken-and-egg type of problem. Security-conscious users, applications, hackers, and customers will flock towards whichever database product leads in that area. If some hypothetical database has only minimal security features, I imagine few security experts would spend a lot of time with the database. The second problem is that we're not sure it's really the right thing, because we have no one who is competent to review the design from a security standpoint. But unless we get past the first problem the second one is moot. Are we underestimating Kaigai Kohei? I seem to see him credited on the NSA's SELinux pages: http://www.nsa.gov/research/selinux/contrib.shtml and it seems his patches there related to postgresql were pretty widely discussed on the SELinux lists: http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Monday 26 January 2009 6:31:48 pm Ron Mayer wrote: Tom Lane wrote: [...snip...] The second problem is that we're not sure it's really the right thing, because we have no one who is competent to review the design from a security standpoint. But unless we get past the first problem the second one is moot. Are we underestimating Kaigai Kohei? I seem to see him credited on the NSA's SELinux pages: http://www.nsa.gov/research/selinux/contrib.shtml and it seems his patches there related to postgresql were pretty widely discussed on the SELinux lists: http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163 I'm pretty sure that when he says review that it is implied to be a peer review -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote: Then why has *nobody* stepped up to review the design, much less the whole patch? The plain truth is that no one appears to care enough to expend any real effort. But this patch is far too large and invasive to accept on the basis that only one guy understands it and will/might continue to maintain it. It was only today that I saw an announcement go out to our announce list to try and get people to pop their heads up and help. It looks like today is the day that someone actually reached out to the SELinux folks and asked for help. We really aren't very good at reaching out for help. We have a tendency to stick to our little pods, in this case -hackers. I'll risk being rude to make my point: those who want SEPostgres in core need to put up or shut up. Now, not at some future time. We need people to sign off that this patch implements the features they want (not sounds roughly like some vague future need I might have) and does so correctly. An incorrect security feature is considerably worse than useless. And once it's in core we aren't going to have a whole lot of elbow room to change the definition later. I think this is a fair statement to make. I would also note that at least one other SELinux person has offered today to review at least the SE portions of this. I quote: Yes, I will look at them to the extent I am able. As I am not familiar with the postgresql codebase I won't be able to assert the correctness of the hook placement (that is, where the security functions are called with respect to the data they are protecting being accessed). The postgresql community should be more familiar with the hook call sites and hopefully can assist there. I should be able to handle the security backend and determining whether it matches the security model we agreed on, but the hook placement is just as important since a misplaced or missing hook will allow access that should not be granted. Joshua Brindle I do think its important to recognize that SEPostgres is a specialty feature. Like full disjunctions was, it is a feature that very, very few will use but those that do really do want and will very much make use of it. If I put on my pragmatic helmet it is a high cost, low benefit feature for the vast majority of our community. If I put on my advocacy hat it is a, Hah take that you little blue mammal! and frankly There is no big O in this equation... if you want this kind of power you gotta ride the elephant!. There is a strategic element though. Adding features for the too smart for their own good lures the too smart for their own good to use (and enhance) a product. This could help us in the long run. Sincerely, Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Ron Mayer rm...@cheapcomplexdevices.com writes: Tom Lane wrote: The second problem is that we're not sure it's really the right thing, because we have no one who is competent to review the design from a security standpoint. Are we underestimating Kaigai Kohei? Perhaps he walks on water, but still I'd like to have more than one person who has confidence that this design and implementation are correct. and it seems his patches there related to postgresql were pretty widely discussed on the SELinux lists: http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163 Well, a quick look through that thread shows a lot of discussion of the selinux policy code that's in the patch, which is good as far as it goes because for sure there's no one in *this* list who understands a line of that stuff. But to be blunt there's no evidence there that anyone in that discussion has heard of a foreign key, much less understands why it might be an issue for this patch. I see a lot of reasoning by analogy to X servers, and little if any database-specific knowledge. Mind you, I'd like nothing better than to have some NSA database security experts (I'm sure there are some) show up here and tell us that this design is good, secure, and useful --- and why. But right now we have no evidence for that proposition. And we really need to understand *why* it's a useful design and what the critical security issues are, because otherwise we are 100% certain to break it in future maintenance (even granting the improbable supposition that there are no bugs in the patch today). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: Ron Mayer rm...@cheapcomplexdevices.com writes: Are we underestimating Kaigai Kohei? Perhaps he walks on water, but still I'd like to have more than one person who has confidence that this design and implementation are correct. Totally fair. I know I'm totally unqualified to review his buoyancy on water, though. and it seems his patches there related to postgresql were pretty widely discussed on the SELinux lists: http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163 Well, a quick look through that thread shows a lot of discussion of the selinux policy code that's in the patch, which is good as far as it goes because for sure there's no one in *this* list who understands a line of that stuff. Totally fair too. Mind you, I'd like nothing better than to have some NSA database security experts (I'm sure there are some) show up here and tell us that this design is good, secure, and useful --- and why. But right now we What's the right way for us to ask them? No doubt there are some, but how do we expect them to find join our email list? If we wanted more feedback would it make sense for someone who can speak for the project to call them and ask if they'd be interested in getting involved? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning (SE-Postgres)
* Ron Mayer (rm...@cheapcomplexdevices.com) wrote: What's the right way for us to ask them? No doubt there are some, but how do we expect them to find join our email list? If we wanted more feedback would it make sense for someone who can speak for the project to call them and ask if they'd be interested in getting involved? Sorry if it wasn't the 'right way', but I've forwarded Bruce's post to -announce (with some additional links to this discussion) to the mailing list up thread, to which I've also subscribed. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
Joshua D. Drake j...@commandprompt.com writes: On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote: Then why has *nobody* stepped up to review the design, much less the whole patch? It was only today that I saw an announcement go out to our announce list to try and get people to pop their heads up and help. It looks like today is the day that someone actually reached out to the SELinux folks and asked for help. Hmm, you think selinux people read pgsql-announce? But seriously, we *have* been trying to get people's attention for this patch, both inside and outside the postgres community, for well over a year now. The lack of response has been depressing and (IMHO) telling. Nowhere have we gotten anything more concrete than ooh, that's cool ... maybe I might use it someday, but I can't be bothered right now. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning (SE-Postgres)
On Mon, 2009-01-26 at 20:27 -0500, Stephen Frost wrote: * Ron Mayer (rm...@cheapcomplexdevices.com) wrote: What's the right way for us to ask them? No doubt there are some, but how do we expect them to find join our email list? If we wanted more feedback would it make sense for someone who can speak for the project to call them and ask if they'd be interested in getting involved? Sorry if it wasn't the 'right way', but I've forwarded Bruce's post to -announce (with some additional links to this discussion) to the mailing list up thread, to which I've also subscribed. As have I. Perhaps some humble requests for help that are directed at the pin point will help. Joshua D. Drake Thanks, Stephen -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: Hmm, you think selinux people read pgsql-announce? But seriously, we *have* been trying to get people's attention for this patch, both inside and outside the postgres community, for well over a year now. The lack of response has been depressing and (IMHO) telling. Nowhere have we gotten anything more concrete than ooh, that's cool ... maybe I might use it someday, but I can't be bothered right now. Ah! Then yes, that does say something about the lack of interest. It wasn't obvious to me that people were reaching out beyond these lists. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
* Ron Mayer (rm...@cheapcomplexdevices.com) wrote: Tom Lane wrote: Hmm, you think selinux people read pgsql-announce? But seriously, we *have* been trying to get people's attention for this patch, both inside and outside the postgres community, for well over a year now. The lack of response has been depressing and (IMHO) telling. Nowhere have we gotten anything more concrete than ooh, that's cool ... maybe I might use it someday, but I can't be bothered right now. Ah! Then yes, that does say something about the lack of interest. It wasn't obvious to me that people were reaching out beyond these lists. Where were we reaching outside the postgres community..? Just curious what other SELinux or related lists are out there. As mentioned up thread, there was interest for a couple of months this past summer on the NSA SELinux mailing list. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 20:28 -0500, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote: Then why has *nobody* stepped up to review the design, much less the whole patch? It was only today that I saw an announcement go out to our announce list to try and get people to pop their heads up and help. It looks like today is the day that someone actually reached out to the SELinux folks and asked for help. Hmm, you think selinux people read pgsql-announce? *shrug* they very well might if they are also postgres people. But seriously, we *have* been trying to get people's attention for this patch, both inside and outside the postgres community, for well over a year now. The lack of response has been depressing and (IMHO) telling. Nowhere have we gotten anything more concrete than ooh, that's cool ... maybe I might use it someday, but I can't be bothered right now. Well that is certainly unfortunate. If the very community that invented the tech isn't willing to help :( Joshua D. Drake regards, tom lane -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: Ron Mayer rm...@cheapcomplexdevices.com writes: Tom Lane wrote: The second problem is that we're not sure it's really the right thing, because we have no one who is competent to review the design from a security standpoint. Are we underestimating Kaigai Kohei? Perhaps he walks on water, but still I'd like to have more than one person who has confidence that this design and implementation are correct. and it seems his patches there related to postgresql were pretty widely discussed on the SELinux lists: http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163 Well, a quick look through that thread shows a lot of discussion of the selinux policy code that's in the patch, which is good as far as it goes because for sure there's no one in *this* list who understands a line of that stuff. But to be blunt there's no evidence there that anyone in that discussion has heard of a foreign key, much less understands why it might be an issue for this patch. I see a lot of reasoning by analogy to X servers, and little if any database-specific knowledge. http://marc.info/?l=selinuxm=115762285013528w=2 Is the original discussion thread for the security model used in the sepostgresql work. Hopefully you'll see some of the evidence you speak of there. For some additional resources there is a good chapter on database security (specifically multilevel databases) in Pfleeger's Security in Computing: http://www.amazon.com/Security-Computing-Second-Charles-Pfleeger/dp/0133374866/ref=si3_rdr_bb_product Mind you, I'd like nothing better than to have some NSA database security experts (I'm sure there are some) show up here and tell us that this design is good, secure, and useful --- and why. But right now we have no evidence for that proposition. And we really need to understand *why* it's a useful design and what the critical security issues are, because otherwise we are 100% certain to break it in future maintenance (even granting the improbable supposition that there are no bugs in the patch today). This capability puts PostgreSQL in a unique position. Not only is the security model more fine grained than any previous trusted database but it also allows interesting things like trusted stored procedures that can access data (and do any necessary fuzzing, modifying or filtering) that the client himself cannot read. It also integrates well with the labeled networking supplied by SELinux, as outlined on KaiGai's wiki page. There can be multiple clients running on one machine (such as a web server) that have different levels of database access, in a more fine grained and flexible way than using different database credentials allows. I'm happy to discuss this at length if it will help the patches get upstreamed. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Stephen Frost sfr...@snowman.net writes: * Ron Mayer (rm...@cheapcomplexdevices.com) wrote: Ah! Then yes, that does say something about the lack of interest. It wasn't obvious to me that people were reaching out beyond these lists. Where were we reaching outside the postgres community..? Well, I've been trying to get Red Hat to interest their NSA contacts in it, and Bruce and Josh B have been making efforts via EDB and Sun that I don't have details about, but nothing much has come of any of that. Maybe waving a red flag in front of the selinux list will have some effect. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: Stephen Frost sfr...@snowman.net writes: * Tom Lane (t...@sss.pgh.pa.us) wrote: The problem, in words of one syllable, is that we are not sure we want it. Do you see a user community clamoring for SEPostgres, or a hacker community that is willing or able to maintain it? No, it doesn't have as large a user base as the Windows port or integrated text search. On the other hand, there *are* users out there, and hackers, who are willing and interested in it for PostgreSQL because it would give them an alternative to the de-facto standards. Then why has *nobody* stepped up to review the design, much less the whole patch? The plain truth is that no one appears to care enough to expend any real effort. But this patch is far too large and invasive to accept on the basis that only one guy understands it and will/might continue to maintain it. The matter we're currently faced can be called as like a disconnection between OSS communities. At least, as several folks introduced in this thread, security focused people are strongly waiting for SE-PostgreSQL feature upstreamed. However, we have a wall to be overed, if they join to review the patches, because most of security experts are not database experts (familiar to its internal architectures). In addition, I have hesitated to involve security experts due to the discussion will need deep knowledge about its internal architectures. But I think Bruce's suggestion is whorthwhile. At least, it is a case we need cross-community discussion. I'll risk being rude to make my point: those who want SEPostgres in core need to put up or shut up. Now, not at some future time. We need people to sign off that this patch implements the features they want (not sounds roughly like some vague future need I might have) and does so correctly. An incorrect security feature is considerably worse than useless. And once it's in core we aren't going to have a whole lot of elbow room to change the definition later. At least, the security design of SE-PostgreSQL has been accepted for two years in SELinux community. An evidence is its upstreamed security policy (reference policy) contains rules for SE-PostgreSQL. http://oss.tresys.com/repos/refpolicy/trunk/policy/modules/services/postgresql.te Thanks, -- OSS Platform Development Division, NEC KaiGai Kohei kai...@ak.jp.nec.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Jaime Casanova wrote: SE-Linux: this patch has effectively been in development for 2 years ourside the core process before putting it in; the forked SEPostgres is in use in production. KaiGai has been available for 20 hours a week (or more) to troubleshoot issues and change APIs. I really don't see what the problem is with committing it. it hasn't been testing by ours in different platforms (ie: ubuntu has selinux and i want to give it a try, badly enough i have never used selinux so this is new to me)... nor we have any evidence that it doesn't affect to users that doesn't have any variant of selinux (ie: windows)... the real problem here is the base of users we enough knowledge of the tool to make some usefull tests It is obvious, if you see the code. When SELinux is disabled by platform or GUC option, security hooks works nothing. Thanks, -- OSS Platform Development Division, NEC KaiGai Kohei kai...@ak.jp.nec.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
* Tom Lane (t...@sss.pgh.pa.us) wrote: Well, I've been trying to get Red Hat to interest their NSA contacts in it, and Bruce and Josh B have been making efforts via EDB and Sun that I don't have details about, but nothing much has come of any of that. It's certainly frustrating to hear that Red Hat, who have been pretty big proponents of SELinux, not showing interest in having it supported in PostgreSQL.. :/ Maybe waving a red flag in front of the selinux list will have some effect. I certainly hope so. I'm planning to review the patch and documentation and set up an environment and run it through its paces this week. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
Joshua Brindle wrote: While we haven't been able to analyze the patches directly to determine whether the security goals are indeed being met we have had much discussion and eventually community agreement on the security model being implemented. This happened years ago and has since been merged into the SELinux reference policy that practically all SELinux users use (distributions start with the reference policy and add rules/modules suitable for them). So the security model has been looked at, though not the implementation and we do have a community of developers, users and customers interested in this work. That is good to know. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: Ron Mayer rm...@cheapcomplexdevices.com writes: Tom Lane wrote: The second problem is that we're not sure it's really the right thing, because we have no one who is competent to review the design from a security standpoint. Are we underestimating Kaigai Kohei? Perhaps he walks on water, but still I'd like to have more than one person who has confidence that this design and implementation are correct. and it seems his patches there related to postgresql were pretty widely discussed on the SELinux lists: http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163 Well, a quick look through that thread shows a lot of discussion of the selinux policy code that's in the patch, which is good as far as it goes because for sure there's no one in *this* list who understands a line of that stuff. But to be blunt there's no evidence there that anyone in that discussion has heard of a foreign key, much less understands why it might be an issue for this patch. I see a lot of reasoning by analogy to X servers, and little if any database-specific knowledge. I believe they understand the issues related to covert channels (via PK/FK) and polyinstantiation, though it is not talked on the thread. (NOTE: The origin of poluinstantiation is previous security research!) Yes, there is indeed no evidence. If so, it is a good idea to ask their opinion with SELinux folks (expect for me?). As I noted before several times, there are several classes in security requirements. Historically, TSCEC (Trusted Computer System Evaluation Criteria, Orange-Book) is a representative evaluation criteria for security features in IT producets. Now it is inherited to ISO/IEC15408 called as CC (Common Criteria). We can also consider CC as a set of security functional requirements, and it defined various kind of requirements. An interaction between foreign keys and invisible rows is a kind of covert channel issue. Indeed, it has a possibility users to infer existance of invisible tuples. In ISO/IEC15408, FDP_IFF (Information Flow Function, Functionality of Data Protection) section describes about related requirements. It defines various classes of requirements. One highest stuff requires to eliminate all the information-flows via covert-channel, but others does not require to eliminate at all, or does not mention about it. Here, it is important what requirement are choosen depends on a sort of evaluated product, environment, estimated-threat and so on. NOTE: Oracle Label Security does not care about covert-channels, because it may also compound for FK/PK issues and row-level security. http://www.commoncriteriaportal.org/files/epfiles/20080306_0402b.pdf In the previous discussion, we (including me) decided that SE-PostgreSQL also does not care about the FK/PK issues, because polyinstantiation feature needs unacceptable widespread reworks on PostgreSQL, so I added an explicit text in the documentation to notice its restriction for end-users. At least, earlier commercial database (Oracle) does not care, and it can follow ISO/IEC15408 manner, so I think it is fair enough approach. It is a summary of previous discussion. Joshua, Chad, please comment anything, if my understanding was incorrect. BTW, I have not walked on water yet. Thanks, -- OSS Platform Development Division, NEC KaiGai Kohei kai...@ak.jp.nec.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, 2009-01-27 at 12:30 +0900, KaiGai Kohei wrote: BTW, I have not walked on water yet. I have but I always end up wet. :) Joshua D. Drake Thanks, -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers- ow...@postgresql.org] On Behalf Of Joshua D. Drake Sent: Monday, January 26, 2009 7:42 PM To: KaiGai Kohei Cc: Tom Lane; Ron Mayer; Josh Berkus; Robert Haas; Merlin Moncure; Jonah H. Harris; Gregory Stark; Simon Riggs; Bruce Momjian; Bernd Helmle; Peter Eisentraut; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] 8.4 release planning On Tue, 2009-01-27 at 12:30 +0900, KaiGai Kohei wrote: BTW, I have not walked on water yet. I have but I always end up wet. :) I find that it helps to freeze the water first. ;-) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Dann Corbit wrote: -Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers- ow...@postgresql.org] On Behalf Of Joshua D. Drake Sent: Monday, January 26, 2009 7:42 PM To: KaiGai Kohei Cc: Tom Lane; Ron Mayer; Josh Berkus; Robert Haas; Merlin Moncure; Jonah H. Harris; Gregory Stark; Simon Riggs; Bruce Momjian; Bernd Helmle; Peter Eisentraut; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] 8.4 release planning On Tue, 2009-01-27 at 12:30 +0900, KaiGai Kohei wrote: BTW, I have not walked on water yet. I have but I always end up wet. :) I find that it helps to freeze the water first. ;-) I'm sorry for the improper humor. Shall we return to the discussion? -- OSS Platform Development Division, NEC KaiGai Kohei kai...@ak.jp.nec.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Joshua Brindle met...@manicmethod.com writes: http://marc.info/?l=selinuxm=115762285013528w=2 Is the original discussion thread for the security model used in the sepostgresql work. Hopefully you'll see some of the evidence you speak of there. Thanks for the link. I took a look through that thread and saw a lot of discussion about issues like how to relate the database-side and client-side permissions, which is all good stuff but mostly outside my purview as a database geek. I didn't find anything about the stuff that is really bothering me, which I think can be broken down into two main categories: 1. Silently filtering out rows according to an arbitrary security policy can break a bunch of fundamental SQL semantics, the most obvious being foreign key constraints --- an application might be able to see a dependent row that apparently has no referenced row, or might get an update or delete failure for a row that is unreferenced as far as it can see. Things get worse if an application can insert, update or delete rows that it can't select. The only answer I've been able to get about what SEPostgres will do about that is we really don't care that we're breaking SQL semantics. I don't find that to be a satisfactory answer. The security-geek reason why not is that it represents a huge information leak. The database-geek reason why not is that this will permanently destroy our ability to do a lot of important optimizations, eg join removal on the basis of foreign key constraints. (There are probably other good reasons, but that one will do for starters.) Perhaps this is fixable by constraining what a security policy is allowed to do, or in some other way that I don't know about, but I've seen no serious discussion about that. 2. I don't understand where to draw the dividing line between database system accesses (which can't be security constrained, at least not without breaking things entirely --- eg it will do you little good to imagine that you can hide rows in pg_security from the security-enforcement code ;-)) and user accesses that should be security-constrained. I am certain that the line is muddied beyond usability in the current system; there are a lot of user-exposed functions that are making use of the same infrastructure that core system accesses do. As an example, some of the people who think they want this feature would like to use it to hide the bodies of their user-defined functions from other people whom they don't wish to see their code. But pg_get_functiondef() uses the same catcache infrastructure as the code that would be called on to actually execute their function, so there is no reasonable place to prevent the function body from being exposed through that inquiry function or others of its ilk. This problem gets rapidly worse when you consider that Postgres is designed to be a very extensible system. It's not clear how to classify add-on code, and it is clear that any of it could unintentionally introduce a security hole. The only solution I can see is we stop guaranteeing that SEPostgres is good for anything the moment you load even one extension module, and that isn't a very satisfactory answer either. Even accepting such a restriction, there's too much code in core Postgres to let anyone feel very good about keeping the core free of security leaks. There are some other problems, like the rather frightening thought that a database dump might silently omit critical data (remember pg_dump is just another client). But I think the two categories above cover the issues that are making me seriously leery of this patch. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
OK, time for me to chime in. I think the outstanding commit-fest items can be broken down into four sections: o Log streaming o Hot standby o SE-PostgreSQL o Others I think we all agree that log streaming is not ready for 8.4, and that delaying for this feature would lead to an indeterminate delay. I have already talked to the NTT folks, and as painful as it is, I think they have accepted this --- many thanks to them. Hot standby seems to be having a lot of code churn. I am not sure if it is because the patch is being polished for completion or if lots of new problems are being discovered and fixed. Tom and I have both given up trying to track this patch. Fortunately, Heikki has been following it closely, so I think Heikki is going to make the final decision if hot standby is ready for 8.4 (because he is going to have to stand behind it as a committer). SE-PostgreSQL has been in steady development for a year so this is the time to decide about it. My feeling is if we don't accept it now, we are never going to have SE-Linux or row-level security. The next week should show us the right direction when we start discussion on Wednesday, noon GMT. The other patches are 1-2 weeks work and will be dealt with like patches from previous commit-fests. FYI, I think delaying a major release to get feature X always going to be counter-productive because it requires restarting development with no known completion time, and they typical behavior is just 2 more weeks, just two more weeks, over and over again, so no one knows how much time they have to develop stuff and things stall badly. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
All, FWIW, I'll comment that what we're seeing here is nothing new. We have: --One invasive patch which everybody (myself included) procrastinated on reviewing even though we got it early, and: --One must have big complex patch which arrived very late in the development cycle. These are our two reasons for every release delay I can think of. In prior releases, HOT, PITR, Windows, etc., all presented the same issues. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Josh Berkus j...@agliodbs.com writes: FWIW, I'll comment that what we're seeing here is nothing new. Certainly the Hot Standby situation is the same old song, different verse. (I'm personally of the opinion that the project has usually been better served when we decided not to postpone a release to wait for a specific feature.) SEPostgres seems qualitatively different to me, though. I think PG people have avoided reviewing it because (a) they weren't interested in it and (b) they knew they were unqualified to review it. Meanwhile it's emerging that the selinux people don't feel qualified to review it either. I'm not quite sure what to do about that. But throw it in there on faith doesn't sound like an appealing answer, and I've got no idea how long it will take to work out a non-faith-based answer. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 23:39 -0500, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: FWIW, I'll comment that what we're seeing here is nothing new. Meanwhile it's emerging that the selinux people don't feel qualified to review it either. I'm not quite sure what to do about that. But throw it in there on faith doesn't sound like an appealing answer, and I've got no idea how long it will take to work out a non-faith-based answer. O.k. maybe it is time to consider something non-traditional. What about two 8.4 releases? 8.4-stable 8.4-experimental stable is everything that stable is. PostgreSQL at its best. experimental is the day after we branch. Same catalog version but contains say SEPostgres and Hot Standby etc... 8.4-experimental gives people a stable same version compatible version to test at their leisure. We support it until 8.5 comes out. At 8.5, those features are merged (or ripped out) into 8.5-stable. Yes I know this is non-standard for our community but maybe it is time to crank it up a notch? Sincerely, Joshua D. Drake regards, tom lane -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
SEPostgres seems qualitatively different to me, though. I think PG people have avoided reviewing it because (a) they weren't interested in it and (b) they knew they were unqualified to review it. I think that you are off-base here. As I've pointed out previously, nobody was ever ASSIGNED to review SE-PostgreSQL. http://archives.postgresql.org/message-id/603c8f07090132m2a595ec0g677762c1fff58...@mail.gmail.com The reviewing that happened during this CommitFest did not happen on the basis of who was interested in which patches. There was a bit of that, but for the most part people reviewed the patches that they were asked to review. I assumed (am I the only one?) that the REASON why we were not asked to review SE-PostgreSQL or Hot Standby is because the committers were planning to do that themselves due to the complexity of the patches. Now, apparently, that assumption was totally wrong. But this doesn't seem complicated to me. If we bump SE-PostgreSQL to the next CommitFest and assign reviewers, they will review it. Maybe their review will not be 100% perfect, but that is why we have committers. If we continue to NOT assign reviewers, reviewers are unlikely to crawl out of the woodwork, just as they (mostly) didn't crawl out of the woodwork for any other patches (HS/SR being a notable exception). ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert, The reviewing that happened during this CommitFest did not happen on the basis of who was interested in which patches. There was a bit of that, but for the most part people reviewed the patches that they were asked to review. I assumed (am I the only one?) that the REASON why we were not asked to review SE-PostgreSQL or Hot Standby is because the committers were planning to do that themselves due to the complexity of the patches. Actually, I did assign someone to do a build and specification review. But yes, I expected that the code review would *have* to be done by a long-term committer. I pretty much assume that of anything over 300 lines. The idea behind having new reviewers take on all the small patches, was, of course, to give the main committers more time with patches like SEPostgres. It worked with other stuff (like Windowing and CTE). --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
The reviewing that happened during this CommitFest did not happen on the basis of who was interested in which patches. There was a bit of that, but for the most part people reviewed the patches that they were asked to review. I assumed (am I the only one?) that the REASON why we were not asked to review SE-PostgreSQL or Hot Standby is because the committers were planning to do that themselves due to the complexity of the patches. Actually, I did assign someone to do a build and specification review. But yes, I expected that the code review would *have* to be done by a long-term committer. I pretty much assume that of anything over 300 lines. The idea behind having new reviewers take on all the small patches, was, of course, to give the main committers more time with patches like SEPostgres. It worked with other stuff (like Windowing and CTE). Right, so, then I'm not sure why Tom is taking the lack of review as a sign of lack of interest. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
2009/1/27 Joshua D. Drake j...@commandprompt.com: On Mon, 2009-01-26 at 23:39 -0500, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: FWIW, I'll comment that what we're seeing here is nothing new. Meanwhile it's emerging that the selinux people don't feel qualified to review it either. I'm not quite sure what to do about that. But throw it in there on faith doesn't sound like an appealing answer, and I've got no idea how long it will take to work out a non-faith-based answer. O.k. maybe it is time to consider something non-traditional. What about two 8.4 releases? 8.4-stable 8.4-experimental stable is everything that stable is. PostgreSQL at its best. I dislike this idea - it's same like short processed 8.5 - that is more simple. Actually current base 8.4 has lot of features, enough for people, so it could be released. 8.5 should be implemented in shorted cycle - only one commitfest, that is enough (+3 month) for well completing SE and replication patches. In czech, big users started testing and using 8.3, and propably skip 8.4. For small projects should be 8.4 available early. regards Pavel Stehule experimental is the day after we branch. Same catalog version but contains say SEPostgres and Hot Standby etc... 8.4-experimental gives people a stable same version compatible version to test at their leisure. We support it until 8.5 comes out. At 8.5, those features are merged (or ripped out) into 8.5-stable. Yes I know this is non-standard for our community but maybe it is time to crank it up a notch? Sincerely, Joshua D. Drake regards, tom lane -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, Jan 26, 2009 at 6:07 PM, Gregory Stark st...@enterprisedb.com wrote: I realize in the current system (emailed patches), this would be a horrible pain to maintain such a branch; but perhaps some of the burden could be pushed down to the patch submitters (asking them to merge their own changes into this merged branch). I've considered maintaining such a repository a few times and dismissed it when I realized how much work it would be to maintain. I don't think it would be too bad if everyone were using git. And in fact, if people weren't using git, but we had some kind of a system that could (1) return a list of all of the current patches and (2) return for any given patch the message-ids of the messages wherein the various versions of the patch were submitted, it would be harder, but possibly managable. You might have trouble with really big patches creating lots of merge conflicts, but even if you just merged all the smaller patches into one tree and push it out for people to test against, that might still have some real value if a decent number of people did testing against that tree. I think that it would probably be pretty easy to write a webapp to replace the CommitFest web page that basically did the same thing but with a bit more structure around it - with database tables like commitfest, patch, patch_version, patch_comment, and patch_review. I think I might even be willing to write such a webapp if someone would be willing to provide the infrastructure. The CommitFest web page was really useful this time around, but it's not conducive to any kind of automated pull. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, Jan 27, 2009 at 12:40 AM, Pavel Stehule pavel.steh...@gmail.com wrote: so it could be released. 8.5 should be implemented in shorted cycle - only one commitfest, that is enough (+3 month) for well completing SE and replication patches. we tried this before (8.2 to 8.3 i think), the idea was that the next release should be in 6 months... we release at least 6 months later... ATM that a new release cycle starts new patch will arrive and there will be no way to get the shorted release in time... -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
2009/1/27 Jaime Casanova jcasa...@systemguards.com.ec: On Tue, Jan 27, 2009 at 12:40 AM, Pavel Stehule pavel.steh...@gmail.com wrote: so it could be released. 8.5 should be implemented in shorted cycle - only one commitfest, that is enough (+3 month) for well completing SE and replication patches. we tried this before (8.2 to 8.3 i think), the idea was that the next release should be in 6 months... we release at least 6 months later... ATM that a new release cycle starts new patch will arrive and there will be no way to get the shorted release in time... I remember it. Solution is - don't accept new patches for next commitfest. Pavel -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Pavel Stehule wrote: 2009/1/27 Jaime Casanova jcasa...@systemguards.com.ec: On Tue, Jan 27, 2009 at 12:40 AM, Pavel Stehule pavel.steh...@gmail.com wrote: so it could be released. 8.5 should be implemented in shorted cycle - only one commitfest, that is enough (+3 month) for well completing SE and replication patches. we tried this before (8.2 to 8.3 i think), the idea was that the next release should be in 6 months... we release at least 6 months later... ATM that a new release cycle starts new patch will arrive and there will be no way to get the shorted release in time... I remember it. Solution is - don't accept new patches for next commitfest. I don't think that'll work. People will still keep writing patches. Or if they don't, that's even worse! Either people will work on patches that they're interested in, or they'll go away and do something else. Only very few will drop their pet projects for the common good and help with the review instead. We can adjust the length of the release cycle by adjusting the number of commitfests. I think the normal ~ 1 year cycle is quite optimal, though. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Sorry for long description. Tom Lane wrote: Joshua Brindle met...@manicmethod.com writes: http://marc.info/?l=selinuxm=115762285013528w=2 Is the original discussion thread for the security model used in the sepostgresql work. Hopefully you'll see some of the evidence you speak of there. Thanks for the link. I took a look through that thread and saw a lot of discussion about issues like how to relate the database-side and client-side permissions, which is all good stuff but mostly outside my purview as a database geek. I didn't find anything about the stuff that is really bothering me, which I think can be broken down into two main categories: 1. Silently filtering out rows according to an arbitrary security policy can break a bunch of fundamental SQL semantics, the most obvious being foreign key constraints --- an application might be able to see a dependent row that apparently has no referenced row, or might get an update or delete failure for a row that is unreferenced as far as it can see. Things get worse if an application can insert, update or delete rows that it can't select. The only answer I've been able to get about what SEPostgres will do about that is we really don't care that we're breaking SQL semantics. I don't find that to be a satisfactory answer. As I repeated it several times, it is a covert channel issue. An important thing is what the limilation is well documented and allows end-users to choose the option. As I summarized in another message, it is not a must-requirement for security evaluation criteria (ISO/IEC15408, CC). The criteria allows to include a feature to eliminate covert channels, but it also allows not to include ones to eliminate them. http://wiki.postgresql.org/wiki/SEPostgreSQL#Covert_channels For example, Oracle Label Security which is certified as EAL4+ class and also provides row-level mandatory access controls, but it does not care about covert channels. This fact is documented as Security Target, so end-users can know that elimination of covert channel is over spec for the product. They can choose the product with their own responsibility. At first, we should understand is individual security features have its own targets, coverages and so on. I assumes the target of SE-PostgreSQL is enterprise class users who are interested in web application security like a cloud-computing platform, similar to Oracle's one. The security-geek reason why not is that it represents a huge information leak. The database-geek reason why not is that this will permanently destroy our ability to do a lot of important optimizations, eg join removal on the basis of foreign key constraints. (There are probably other good reasons, but that one will do for starters.) Perhaps this is fixable by constraining what a security policy is allowed to do, or in some other way that I don't know about, but I've seen no serious discussion about that. IIRC, we had discussions refering to ISO/IEC15408 a few times at the past. I believe it was serious discussions. Polyinstantiation technology might help the situation, but we decided not to port it on PostgreSQL due to widespread unacceptable changes. I make clear it again. It is an over spec for SE-PostgreSQL to eliminate all the covert channels, however, we documented the limitation for end-users, and they can choose SE-PostgreSQL with their own responsibility. 2. I don't understand where to draw the dividing line between database system accesses (which can't be security constrained, at least not without breaking things entirely --- eg it will do you little good to imagine that you can hide rows in pg_security from the security-enforcement code ;-)) and user accesses that should be security-constrained. Please consider the symmetrical architecture between OS and DBMS. A process accesses resources managed by OS via system calls, and SELinux acquire the system calls and applies its decision making. In other hand, a client accesses database object managed by DBMS via SQL queries, and SE-PostgreSQL applies its decision come from security policy of SELinux. Please note that SELinux cannot apply its security policy on accesses come from in-kernel code in principle, although it enables to control kernel-loadable-modules. In same manner, SE-PostgreSQL cannot apply its access controls on internal database system accesses. It come from characteristics of a reference monitor which applies its security policy for all the required accesses, but internal steps are exceptions. For example, if SELinux allows a malicious one to load a kernel loadable module which hijacks filesystems, any other users have a possibility to invoke the malicious code indirectly which can bypass SELinux's checks. I am certain that the line is muddied beyond usability in the current system; there are a lot of user-exposed functions that are making use of the same infrastructure that core system accesses do. As an example, some of
Re: [HACKERS] 8.4 release planning
Bruce Momjian wrote: OK, time for me to chime in. I think the outstanding commit-fest items can be broken down into four sections: o Log streaming o Hot standby o SE-PostgreSQL o Others - snip - SE-PostgreSQL has been in steady development for a year so this is the time to decide about it. My feeling is if we don't accept it now, we are never going to have SE-Linux or row-level security. The next week should show us the right direction when we start discussion on Wednesday, noon GMT. Hmm...? This conditional punishment of death seems to me not a reasonable one. If we can found a matter as a result of discussion, which is impossible to fix within reasonable term, I'll agree it being postponed to v8.5. However, why is the punishment of death necessary here? Thanks, -- OSS Platform Development Division, NEC KaiGai Kohei kai...@ak.jp.nec.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Josh Berkus j...@agliodbs.com writes: The idea behind having new reviewers take on all the small patches, was, of course, to give the main committers more time with patches like SEPostgres. It worked with other stuff (like Windowing and CTE). Huh? There were certainly non-committer reviewers for the window functions and CTE patches. But the larger point here is that SEPostgres has been around for nearly two years, and has been submitted into every 8.4 commit fest not just this last one, and has drawn no noticeable amount of enthusiasm from the pghacker community. It's not just a matter of we haven't gotten to it yet; AFAICS no one has wanted to get to it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
