Re: [HACKERS] Streaming replication, some small issues
On Tue, Dec 8, 2009 at 5:30 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: - If a WAL file is not found in the master for some reason, standby goes into an infinite loop retrying it: ERROR: could not read xlog records: FATAL: could not open file pg_xlog/0001 (log file 0, segment 0): No such file or directory I also fixed this problem. git://git.postgresql.org/git/users/fujii/postgres.git branch: replication Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Largeobject Access Controls (r2460)
KaiGai Kohei kai...@ak.jp.nec.com wrote: We have to reference pg_largeobject_metadata to check whether a certain large objct exists, or not. It is a case when we create a new large object, but write nothing. OK, that makes sense. In addition of the patch, we also need to fix pg_restore with --clean option. I added DropBlobIfExists() in pg_backup_db.c. A revised patch attached. Please check further mistakes. BTW, we can optimize lo_truncate because we allow metadata-only large objects. inv_truncate() doesn't have to update the first data tuple to be zero length. It only has to delete all corresponding tuples like as: DELETE FROM pg_largeobject WHERE loid = {obj_desc-id} Regards, --- Takahiro Itagaki NTT Open Source Software Center pgsql-blob-priv-fix_v2.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Largeobject Access Controls (r2460)
Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp wrote: In addition of the patch, we also need to fix pg_restore with --clean option. I added DropBlobIfExists() in pg_backup_db.c. A revised patch attached. Please check further mistakes. ...and here is an additional fix for contrib modules. Regards, --- Takahiro Itagaki NTT Open Source Software Center fix-lo-contrib.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL/Lite Review
Stephen Frost wrote: * Greg Smith (g...@2ndquadrant.com) wrote: I personally feel that Steven Frost's recent comments here about how the PostgreSQL code makes this harder than it should be really cuts to the core of a next step here. The problem facing us isn't is SEPostgreSQL the right solution for providing external security checks?; it's how can the PostgreSQL code be improved so that integrating external security is easier? Looking Thanks for that support, Greg. This was what I was principally trying to do with KaiGai and the commitfest patch I reviewed of his last round. Unfortunately, the committer comments I received on that patch didn't help us outline a path forward, just declared that the approach in the current patch wasn't workable. My, now much more optimistic thanks to our meeting, view is that the concept of abstracting the access controls is solid and a necessary first step; but we need to find a better way to implement it. I agree. SELinux should be one of the user selectable options. An abstraction layer for enhanced access controls are already adopted in a few open source projects (Linux kernel, X-window), and gets successed. Also thanks to our discussion, I've got a much better handle on how SELinux and the general secuirty community views PostgreSQL (and the Linux kernel for that matter)- it's an application which consists of a set of object managers. That then leads into an approach to address at least some of Tom's comments: Let's start by taking the patch I reviewed and splitting up security/access_control.c along object lines. Of course, the individual security/object_ac.c files would only include the .h's that are necessary. This would be a set of much smaller and much more managable files which only know about what they should know about- the object type they're responsible for. Basically, agreed. This file had more than 4KL in the last commit fest. IMO, however, we should consider these issues before splitting up old patch. (1) Whether the framework should host the default PG model, not only enhanced security features, or not? This patch tried to host both of the default PG model and SELinux. But, the default PG model does not have same origin with label-based mandatory access controls, such as SELinux, from a historical angle. So, this patch needed many of unobvious changes to the core routines. (2) Whether the framework should be comprehensive, or not? This patch tried to replace all the default PG checks in the core routines from the begining. It required to review many of unobvious changes in the core routine. Because I've been repeatedly pointed out to keep changeset smaller, I'm frightened for the direction. The coverage of the latest SE-PgSQL/Lite patch is only databases, schemas, tables and columns. I think it is a good start to deploy security hooks on the routines related to these objects, rather than comprehensive support from the begining. (3) In the future, whether we should allow multiple enhanced securities at the same time, or not? It was not clear in the previous commit fest. But, in fact, Linux kernel does not support multiple security features in same time, because it just makes security label management complex. (It also have another reasons, but just now unlear.) I don't think we have any good reason to activate multiple enhanced securities at same time. I think most of folks can agree with (2) and (3). But I expect opinion is divided at (1). For example, if we host the default PG checks to create a new database, all the existing checks shall be moved as follows: http://code.google.com/p/sepgsql/source/browse/trunk/pgsec/src/backend/security/access_control.c?r=2273#1430 In addition, it has to return the security context to be assigned on the new database, when it hosts SELinux. Even if this rework is limited to four kind of object classes, I'm worry about it overs an acceptable complexity. If the framework hosts only enhanced securities, this hook shall be implemented as: /* * ac_database_create * It checks permission to create a new database, and returns a security * label to be assigned on, or NULL if unnecessary. * * datName : name of the new database * srcDatOid : OID of the template database * datLabel : user given security label, or NULL */ Value * ac_database_create(const char *datName, Oid srcDatOid, Value *datLabel) { #ifdef HAVE_SELINUX if (sepgsql_is_enabled()) return sepgsql_database_create(datName, srcDatOid, datLabel); #endif return NULL; } In the caller (createdb()), this invocation shall be placed as follows: : + datLabel = ac_database_create(dbname, src_dboid, userDatLabel); : + if (!datLabel) + new_record_nulls[Anum_pg_database_datselabel - 1] = true; + else + new_record[Anum_pg_database_datselabel - 1] + =
Re: [HACKERS] Adding support for SE-Linux security
On Fri, Dec 11, 2009 at 05:45, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Dec 10, 2009 at 5:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: My guess is that a credible SEPostgres offering will require a long-term amount of work at least equal to, and very possibly a good deal more than, what it took to make a native Windows port. The SEPostgres community is surely a lot smaller than the Windows community, but I'm not sure whether the effort estimate is accurate or not. If credible includes row-level security, then I think I might agree, but right now we're just trying to get off the ground. It's been perfectly clear since day one, and was reiterated as recently as today http://archives.postgresql.org/message-id/4b21757e.7090...@2ndquadrant.com that what the security community wants is row-level security. The If that is true, then shouldn't we have an implementation of row level security *first*, and then an implementation of selinux hooks that work with this row level security feature? Rather than first doing selinux hooks, then row level security, which will likely need new and/or changed hooks... I'm not convinced that row level security is actually that necessary (though it's a nice feature, with or without selinux), but if it is, it seems we are approaching the problem from the wrong direction. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Tom, * Tom Lane (t...@sss.pgh.pa.us) wrote: It's been perfectly clear since day one, and was reiterated as recently as today http://archives.postgresql.org/message-id/4b21757e.7090...@2ndquadrant.com that what the security community wants is row-level security. Yes, they do want row-level security. That being said, KaiGai, and others, have pointed out time and time over again that SEPG without row-level security is still useful. Additionally, I see absolutely no way that PG would accept a full SEPG+PGACE+row-level security, etc, patch in as one whole patch, ever. I have extreme doubt it would even be something done over one *release*. That all aside, for the moment, I feel that we should begin a 'two-prong' attack here. First, continue down the path that I've started to lay out for SEPG. Second, let's hash out a design for row-level security using the existing PG security model; ideally using the best features and design decisions of the numerous row-level security systems already implemented by the major SQL vendors today. I'll start a new thread on this specific topic to hopefully pull out anyone who's focus is more on that than on SEPG. The proposals to make SEPostgres drive regular SQL permissions never came out of anyone from that side, they were proposed by PG people looking for a manageable first step. I do not believe this to be accurate. Josh, were you able to find any public documentation on Trusted Rubix (is that the right name?)? The RDBMS security policy hashed out on the SELinux list during the discussion of Rubix and SEPG certainly included support for table-level objects, did it not? I expect that the SELinux list contributors would have pointed out if they didn't feel that was at all valuable. Perhaps what is at issue is the terminology being used here though, or the approach to enforment being considered. Part of the discussion at the BWPUG meeting involved the option for SEPG to be a more-restrictive only model in it's implementation. Essentially, this means that all permissions handling is done the same as it is today, except that once the PG model has decided an action is allowed, SEPG kicks in and does any additional checking of the action being requested it wants and may deny it. At the end of the day, I don't feel that it really changes the architecture of the code though. Perhaps users of SELinux will always want that, but the argument we've heard time and time again here is that this should be a generalized approach that other security managers could hook into and use. To do that, I feel we first have to start with the PG model, which *does* care about all the SQL permissions. Let's extract the various complaints and concerns about SELinux that have been thrown around this list and instead consider our first client of the PG modular security framework to be the existing PG SQL permissions system. If we can agree to that, then it's clear that we can't just hand-wave the requirement that it be capable of driving the regular SQL permissions. Whatever you might believe about the potential market for SEPostgres, you should divide by about a hundred as long as it's only an alternate interface to SQL permissions. See particularly here: http://wiki.postgresql.org/wiki/SEPostgreSQL_Review_at_the_BWPUG#Revisiting_row-level_security Without it, it's questionable whether committing the existing stripped-down patch really accomplishes anything --- how much clearer can they be? Again, let's please address row-level security first at the PG level. That was recommended previously by many on this list and is clearly a useful feature which can stand alone in any case. If you're not prepared to assume that we're going to do row level security, it's not apparent why we should be embarking on this course at all. And if you do assume that, I strongly believe that my effort estimate above is on the optimistic side. I do assume we're going to do row level security, but I do not feel that we need to particularly put one in front of the other. I also feel that SEPG will be valuable even without row-level security. One of the realms that we discussed at BWPUG for this is PCI compliance. I'm hopeful Josh will have an opportunity to review the PCI compliance cheat-sheet that I recall Robert Treat offering and comes to agreement that SEPG w/o row-level security would greatly improve our ability to have a PCI compliant system backed with PG. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] navigation menu for documents
Richard Huxton wrote: Reworked jquery-based document menu is attached. Untar will produce bin/ and html/ cd .../html cp /path/to/htmldocs/* . ../bin/add_js.pl *html The contents should be on almost all relevant pages, and: - tested on FF3.5, IE7, Opera 9.x - expanded if screen width 800px, closed otherwise - contain all page-headings and sub-page headings - highlight the current page in the tree - provide tooltips for entries that are too wide for the panel Issues: - slow to reload on each page (can be fixed) - based on hacked-together perl (this is temporary) - layout is perhaps a bit cramped? This last one is tricky. I can space it better if I can assume as 1280px width screen. I can space it better if I have a menu that slides out and covers the whole page. Having the menu items wrap onto a second or third line just didn't work. At least I couldn't find a way that made it clear where the separating line between items was without cluttering the whole menu horribly. Settled on not indenting too much, no word-wrap and tooltips. Comments/suggestions/rotten tomatoes? Richard, First, many thanks for doing this. I don't see any comments having been made. I guess either everyone has been busy or they all think it's just fine (or both) :-) I think you are definitely on the right track. I will use this, committed or not. But I'd like to see it finished and committed. Can you fix the 'slow to reload' problem - that is a bit of a pain. I'm not so worried about the use of perl to add the script calls - you need perl to build the docs anyway. It wasn't terribly cramped for me. I'm not sure everyone will want this. Maybe we need to provide with-js-menu and without-js-menu makefile target variants for html. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Fri, Dec 11, 2009 at 4:31 AM, Magnus Hagander mag...@hagander.net wrote: On Fri, Dec 11, 2009 at 05:45, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Dec 10, 2009 at 5:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: My guess is that a credible SEPostgres offering will require a long-term amount of work at least equal to, and very possibly a good deal more than, what it took to make a native Windows port. The SEPostgres community is surely a lot smaller than the Windows community, but I'm not sure whether the effort estimate is accurate or not. If credible includes row-level security, then I think I might agree, but right now we're just trying to get off the ground. It's been perfectly clear since day one, and was reiterated as recently as today http://archives.postgresql.org/message-id/4b21757e.7090...@2ndquadrant.com that what the security community wants is row-level security. The If that is true, then shouldn't we have an implementation of row level security *first*, and then an implementation of selinux hooks that work with this row level security feature? Rather than first doing selinux hooks, then row level security, which will likely need new and/or changed hooks... I'm not convinced that row level security is actually that necessary (though it's a nice feature, with or without selinux), but if it is, it seems we are approaching the problem from the wrong direction. I don't think there's a correct ordering to SE-PostgreSQL and row-level security. They're better together, but I don't think either has to be done first. If we were going to pick one to do first, I'd pick SE-PostgreSQL. Row-level security is going to be a @$#! of a project if we want it done right (and we do). ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL/Lite Review
KaiGai, * KaiGai Kohei (kai...@ak.jp.nec.com) wrote: (1) Whether the framework should host the default PG model, not only enhanced security features, or not? This patch tried to host both of the default PG model and SELinux. But, the default PG model does not have same origin with label-based mandatory access controls, such as SELinux, from a historical angle. So, this patch needed many of unobvious changes to the core routines. I certainly understand what you're saying here. However, I feel that these changes to the core of PG are good, and are taking PG in the right direction to have a much clearer and better implemented security infrastructure. The fact that you've found a number of odd cases (duplicate checking, odd failure cases due to checks being done at various parts of the process, etc) is a testement that this is bringing the PG code forward even without the addition of SELinux support. I realize that makes it more work for you and I don't envy you that. (2) Whether the framework should be comprehensive, or not? This patch tried to replace all the default PG checks in the core routines from the begining. It required to review many of unobvious changes in the core routine. Because I've been repeatedly pointed out to keep changeset smaller, I'm frightened for the direction. The coverage of the latest SE-PgSQL/Lite patch is only databases, schemas, tables and columns. I think it is a good start to deploy security hooks on the routines related to these objects, rather than comprehensive support from the begining. I don't believe we will get support to commit a framework which is intended to eventually support the PG model (following from my answer to #1) in a partial form. Perhaps it would be good to split the patch up on a per-object-type basis, to make it simpler/easier to review (as in, patch #1: database_ac.c + changes to core for it; patch #2: table_ac.c + changes to core for it, etc; earlier patches assumed to be already done in later patches), but the assumption will almost certainly be that they will all be committed to wonderful CVS at the same time. (3) In the future, whether we should allow multiple enhanced securities at the same time, or not? It was not clear in the previous commit fest. But, in fact, Linux kernel does not support multiple security features in same time, because it just makes security label management complex. (It also have another reasons, but just now unlear.) I don't think we have any good reason to activate multiple enhanced securities at same time. To be honest, I feel that this will just fall out once we get through doing #1. I've just heard from Casey (author of the SMACK security manager, an alternative to SELinux) that the PGACE framework (which is what we suggested he look at) would be easily modified to support SMACK. I feel the same would be true of what we're talking about in #1. If you disagree with that, please let me know. I think most of folks can agree with (2) and (3). But I expect opinion is divided at (1). For example, if we host the default PG checks to create a new database, all the existing checks shall be moved as follows: http://code.google.com/p/sepgsql/source/browse/trunk/pgsec/src/backend/security/access_control.c?r=2273#1430 In addition, it has to return the security context to be assigned on the new database, when it hosts SELinux. Even if this rework is limited to four kind of object classes, I'm worry about it overs an acceptable complexity. I feel that what was at issue is that the complexity of access_control.c was far too great- because *everything* was in that one file. Splitting access_control.c into smaller files would make them more managable, and if we implement them all in a similar manner using a similar policy for function names, arguments, etc, then once the first object type is reviewed, the others will go more easily for the reviwer. If the security framework does *not* support the SQL model, I think we'll get push back from the community that it clearly couldn't be used by any other security manager generically without alot of potential changes and additional hooks that could end up bringing us all the way to being capable of supporting the SQL model anyway. I'm certainly willing to hear core indicate otherwise though. Thanks again, KaiGai. Stephen signature.asc Description: Digital signature
Re: [HACKERS] [PATCH] dtrace probes for memory manager
--On 10. Dezember 2009 23:55:49 -0500 Robert Haas robertmh...@gmail.com wrote: If there's some real-world test where this probe costs 0.3%-0.4%, I think that is sufficient grounds for rejecting this patch. I understand the desire of people to be able to use dtrace, but our performance is too hard-won for me to want to give any measurable of it up for tracing and instrumentation hooks that will only be used by a small number of users in a small number of situations. I repeated the pgbench runs per Greg's advice (see upthread) and it seems there is actually a small slowdown which supports this argument, unfortunately. After repeating the pgbench runs with and without the new probes (note: i've used the new version of the patch, too), the numbers are going to stabilize as follows: without compiled probes: AVG(2531.68) with compiled probes: AVG(2511.97) I can repeat that tests over and over, but this doesn't seem to change the whole picture (so there seems some real argument for a 0.4 - 0.6% cost, at least on *my* machine here with pgbench). -- Thanks Bernd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Bernd Helmle escribió: I repeated the pgbench runs per Greg's advice (see upthread) and it seems there is actually a small slowdown which supports this argument, unfortunately. After repeating the pgbench runs with and without the new probes (note: i've used the new version of the patch, too), the numbers are going to stabilize as follows: without compiled probes: AVG(2531.68) with compiled probes: AVG(2511.97) Were the probes enabled? -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
2009/12/11 KaiGai Kohei kai...@ak.jp.nec.com: It tried to provide a set of comprehensive entry points to replace existing PG checks at once. However, the SE-PgSQL/Lite patch covers accesses on only database, schema, tables and columns. Is it necessary to be comprehensive from the beginning? It might be too aggressive changes at once. Frankly, I hesitate to salvage the patch once rejected, as is. If we implement a set of security hooks, It seems to me the following approach is reasonable: * It does not touch the existing PG default checks. The purpose of security hooks are to host enhanced security features. * It does not deploy hooks on which no security provider is now proposed. It is important to reduce unnecessary changeset. I think that we should try to move the PG default checks inside the hook functions. If we can't do that cleanly, it's a good sign that the hook functions are not correctly placed to enforce arbitrary security policy. Furthermore, it defeats what I think would be a good side goal here, which is to better modularize the existing code. What I would suggest is that you develop a version of that patch that is stripped down to apply to only a single object type (databases? tables and columns - these might have to be together??) and which addresses Tom's criticisms from the last time around, and post that (on a new thread) for discussion. That will be much easier to review (and I will personally commit to reviewing it) but should allow us to flush out some of the design issues. If we can get agreement on that as a concept patch, we can move on to talking about which object types should be included in a committable version of that patch. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] thread safety on clients
Jaime Casanova wrote: On Thu, Dec 10, 2009 at 11:33 PM, Jaime Casanova jcasa...@systemguards.com.ec wrote: On Thu, Dec 10, 2009 at 11:22 PM, Tom Lane t...@sss.pgh.pa.us wrote: My bet is that the real problem was a build inconsistency in the backend. Does make distclean and rebuild make it go away? actually it was a clean build just after a cvs co (not an updated tree), i build the binaries and installed it in just created directory... i will try again now with the patch Bruce just committed the problem has gone Yes, but what if you test with the broken pgbench? As Tom says, it should not be able to crash the backend no matter what it does. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Stephen Frost wrote: Tom, snip The proposals to make SEPostgres drive regular SQL permissions never came out of anyone from that side, they were proposed by PG people looking for a manageable first step. I do not believe this to be accurate. Josh, were you able to find any public documentation on Trusted Rubix (is that the right name?)? The RDBMS security policy hashed out on the SELinux list during the discussion of Rubix and SEPG certainly included support for table-level objects, did it not? I expect that the SELinux list contributors would have pointed out if they didn't feel that was at all valuable. Trusted RUBIX does use the same SELinux object classes and permissions that were originally added to the policy to support SEPostgreSQL. You can look at http://rubix.com/downloads/documentation/RX_SELinux_Guide_6_0_R4.pdf and see how they use SELinux in their RDBMS. Pay particular attention to page 15 where they are saying which object classes and permissions they are using. They even implement row level security (the db_tuple object class) Perhaps what is at issue is the terminology being used here though, or the approach to enforment being considered. Part of the discussion at the BWPUG meeting involved the option for SEPG to be a more-restrictive only model in it's implementation. Essentially, this means that all permissions handling is done the same as it is today, except that once the PG model has decided an action is allowed, SEPG kicks in and does any additional checking of the action being requested it wants and may deny it. At the end of the day, I don't feel that it really changes the architecture of the code though. Perhaps users of SELinux will always want that, but the argument we've heard time and time again here is that this should be a generalized approach that other security managers could hook into and use. To do that, I feel we first have to start with the PG model, which *does* care about all the SQL permissions. Let's extract the various complaints and concerns about SELinux that have been thrown around this list and instead consider our first client of the PG modular security framework to be the existing PG SQL permissions system. If we can agree to that, then it's clear that we can't just hand-wave the requirement that it be capable of driving the regular SQL permissions. Whatever you might believe about the potential market for SEPostgres, you should divide by about a hundred as long as it's only an alternate interface to SQL permissions. See particularly here: http://wiki.postgresql.org/wiki/SEPostgreSQL_Review_at_the_BWPUG#Revisiting_row-level_security Without it, it's questionable whether committing the existing stripped-down patch really accomplishes anything --- how much clearer can they be? Again, let's please address row-level security first at the PG level. That was recommended previously by many on this list and is clearly a useful feature which can stand alone in any case. If you're not prepared to assume that we're going to do row level security, it's not apparent why we should be embarking on this course at all. And if you do assume that, I strongly believe that my effort estimate above is on the optimistic side. I do assume we're going to do row level security, but I do not feel that we need to particularly put one in front of the other. I also feel that SEPG will be valuable even without row-level security. One of the realms that we discussed at BWPUG for this is PCI compliance. I'm hopeful Josh will have an opportunity to review the PCI compliance cheat-sheet that I recall Robert Treat offering and comes to agreement that SEPG w/o row-level security would greatly improve our ability to have a PCI compliant system backed with PG. Thanks, Stephen -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Fri, 2009-12-11 at 09:20 -0500, Robert Haas wrote: On Fri, Dec 11, 2009 at 4:31 AM, Magnus Hagander mag...@hagander.net wrote: On Fri, Dec 11, 2009 at 05:45, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Dec 10, 2009 at 5:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: My guess is that a credible SEPostgres offering will require a long-term amount of work at least equal to, and very possibly a good deal more than, what it took to make a native Windows port. The SEPostgres community is surely a lot smaller than the Windows community, but I'm not sure whether the effort estimate is accurate or not. If credible includes row-level security, then I think I might agree, but right now we're just trying to get off the ground. It's been perfectly clear since day one, and was reiterated as recently as today http://archives.postgresql.org/message-id/4b21757e.7090...@2ndquadrant.com that what the security community wants is row-level security. The If that is true, then shouldn't we have an implementation of row level security *first*, and then an implementation of selinux hooks that work with this row level security feature? Rather than first doing selinux hooks, then row level security, which will likely need new and/or changed hooks... I'm not convinced that row level security is actually that necessary (though it's a nice feature, with or without selinux), but if it is, it seems we are approaching the problem from the wrong direction. I don't think there's a correct ordering to SE-PostgreSQL and row-level security. They're better together, but I don't think either has to be done first. If we were going to pick one to do first, I'd pick SE-PostgreSQL. Row-level security is going to be a @$#! of a project if we want it done right (and we do). I'm not sure it is a strong analogy, but as an example, in the case of Linux, we started by integrating support for a base set of access controls over directories and files, and only later introduced support for multi-level/polyinstantiated directories by building upon Linux's per-process filesystem namespace construct. The base set of access controls for directories and files were certainly useful on their own even before we had the support for polyinstantiated directories. In any event, I would agree that support for applying MAC over the database objects and operations is useful even without row-level security, although ultimately we would like both. -- Stephen Smalley National Security Agency -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL/Lite Review
Robert Haas wrote: One comment I have in general about this process is that I think it would enormously reduce the level of pain associated with making these kinds of changes if we could get patches that were not full of minor issues that need to be cleaned up (like comments not properly adjusted, spelling/grammar errors, formatting inconsistent with the rest of the code base, poor English). Sure, it's not fair to ask the committers to do that. But it's not really something we can expect to get in all situations from our English as a second language contributors either--it's hard enough to get these details right even on stuff done in this country by native speakers. At least we've got a big stack of comments that usually make sense if you read them to work with on the SE-PostgreSQL patches, which is much better than the alternative. I think we need a two pronged attack on this issue. Eventually I think someone who wants this feature in there will need to sponsor someone (and not even necessarily a coder) to do a sizable round of plain old wording cleanup on the comment text of the patch. Robert is right to point out the committers are too expensive, in terms of what work for the community they could be doing besides that, to waste on that job. English editing help good enough to help a lot for this purpose isn't hard to find though. As far as formatting, it drives me a little crazy that it's not more common practice for contributors to do their own targeted pgindent runs to at least nail some of the low-hanging fruit in that area, due the difficulty of working with the tool. That's something else I think it would be nice to invest a little time in improving. For example, there's no reason we should still have to kick back patches regularly just because of tab/space issues, and I know I run into one of those nearly every CommitFest. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] thread safety on clients
Alvaro Herrera alvhe...@commandprompt.com writes: Yes, but what if you test with the broken pgbench? As Tom says, it should not be able to crash the backend no matter what it does. The crash is real --- I've replicated it here. Still trying to figure out what is the real cause. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL/Lite Review
Stephen Frost wrote: KaiGai, * KaiGai Kohei (kai...@ak.jp.nec.com) wrote: (1) Whether the framework should host the default PG model, not only enhanced security features, or not? This patch tried to host both of the default PG model and SELinux. But, the default PG model does not have same origin with label-based mandatory access controls, such as SELinux, from a historical angle. So, this patch needed many of unobvious changes to the core routines. I certainly understand what you're saying here. However, I feel that these changes to the core of PG are good, and are taking PG in the right direction to have a much clearer and better implemented security infrastructure. The fact that you've found a number of odd cases (duplicate checking, odd failure cases due to checks being done at various parts of the process, etc) is a testement that this is bringing the PG code forward even without the addition of SELinux support. I realize that makes it more work for you and I don't envy you that. The reason why I prefer MAC only framework is not a technical reason. As long as we can review it and acceptable, I have no reason to avoid it. (2) Whether the framework should be comprehensive, or not? This patch tried to replace all the default PG checks in the core routines from the begining. It required to review many of unobvious changes in the core routine. Because I've been repeatedly pointed out to keep changeset smaller, I'm frightened for the direction. The coverage of the latest SE-PgSQL/Lite patch is only databases, schemas, tables and columns. I think it is a good start to deploy security hooks on the routines related to these objects, rather than comprehensive support from the begining. I don't believe we will get support to commit a framework which is intended to eventually support the PG model (following from my answer to #1) in a partial form. Perhaps it would be good to split the patch up on a per-object-type basis, to make it simpler/easier to review (as in, patch #1: database_ac.c + changes to core for it; patch #2: table_ac.c + changes to core for it, etc; earlier patches assumed to be already done in later patches), but the assumption will almost certainly be that they will all be committed to wonderful CVS at the same time. As Rober Haas already suggested in another message, my patch in the last commit fest is too large. It tried to rework anything in a single patch. The per-object-type basis make sense for me. In my cosmetic preference, ace_ is better than ac_. The 'e' means extendable, and ace feels like something cool. :-) In X-window system, it calls its framework XACE. And I'd like to store these files in backend/security/ace/*.c, because backend/security will also store other security modules! src/ + backend/ + security/ + ace/ ... access control framework + sepgsql/ ... selinux specific code + smack/ ... (upcoming?) smack specific code + solaristx/ ... (upcoming?) solaris-tx specific code : (3) In the future, whether we should allow multiple enhanced securities at the same time, or not? It was not clear in the previous commit fest. But, in fact, Linux kernel does not support multiple security features in same time, because it just makes security label management complex. (It also have another reasons, but just now unlear.) I don't think we have any good reason to activate multiple enhanced securities at same time. To be honest, I feel that this will just fall out once we get through doing #1. I've just heard from Casey (author of the SMACK security manager, an alternative to SELinux) that the PGACE framework (which is what we suggested he look at) would be easily modified to support SMACK. I feel the same would be true of what we're talking about in #1. If you disagree with that, please let me know. Ahh, what I want to say is, whether we should allow enhanced securities. If we host the default PG model on the framework, it is an exception of the count. The reason why I prefer the default PG check + one enhanced security at most is simplification of the security label management. If two label-based enhanced securities are enabled at same time, we need two field on pg_class and others to store security context. In addition, OS allows to choose one enhanced security at most eventually. In my image, the hook should be as: Value * ac_database_create([arguments ...]) { /* * The default PG checks here. * It is never disabled */ /* enhanced security as follows */ #if defined(HAVE_SELINUX) if (sepgsql_is_enabled()) return sepgsql_database_create(...); #elif defined(HAVE_SMACK) if (smack_is_enabled()) return smack_database_create(...); #elif defined(HAVE_SOLARISTX) if (soltx_is_enabled()) return soltx_database_create(...); #endif
Re: [HACKERS] Adding support for SE-Linux security
On Fri, 2009-12-11 at 09:32 -0500, Robert Haas wrote: 2009/12/11 KaiGai Kohei kai...@ak.jp.nec.com: It tried to provide a set of comprehensive entry points to replace existing PG checks at once. However, the SE-PgSQL/Lite patch covers accesses on only database, schema, tables and columns. Is it necessary to be comprehensive from the beginning? It might be too aggressive changes at once. Frankly, I hesitate to salvage the patch once rejected, as is. If we implement a set of security hooks, It seems to me the following approach is reasonable: * It does not touch the existing PG default checks. The purpose of security hooks are to host enhanced security features. * It does not deploy hooks on which no security provider is now proposed. It is important to reduce unnecessary changeset. I think that we should try to move the PG default checks inside the hook functions. If we can't do that cleanly, it's a good sign that the hook functions are not correctly placed to enforce arbitrary security policy. Furthermore, it defeats what I think would be a good side goal here, which is to better modularize the existing code. So from the meeting on Wednesday I got the impression that Steve already did this. However it was rejected because too much information was need to be passed around. I gathered and I could be wrong but the reason for this is that instead of developing proper abstractions for the security code people made use of whatever local stuff was available to them at that location. With the X-ACE work that Eamon Walsh did he did exactly what you're saying. He figured out the object model for the X-Server and created the hook framework. In the merge of the hook framework he also moved the existing X server security model into the framework. What I would suggest is that you develop a version of that patch that is stripped down to apply to only a single object type (databases? tables and columns - these might have to be together??) and which addresses Tom's criticisms from the last time around, and post that (on a new thread) for discussion. That will be much easier to review (and I will personally commit to reviewing it) but should allow us to flush out some of the design issues. If we can get agreement on that as a concept patch, we can move on to talking about which object types should be included in a committable version of that patch. They may have been said before but what exactly are the design issues? The main concern I hear is that people are worried that this is an SELinux specific design. I heard at the meeting on Wednesday that the Trusted Extensions people looked at the framework and said it meets their needs as well. If thats the case where does the concept that the design is SELinux specific stem from? We've asked Casey Schaufler the developer of another label based MAC system for Linux to look at the hooks as well and make a statement about their usability. Dave -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] thread safety on clients
On 12/11/09, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@commandprompt.com writes: Yes, but what if you test with the broken pgbench? As Tom says, it should not be able to crash the backend no matter what it does. The crash is real --- I've replicated it here. Still trying to figure out what is the real cause. Several threads writing to single connection? -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL/Lite Review
Greg Smith wrote: It's funny; we started out this CommitFest with me scrambling to find someone, anyone, willing to review the latest SE-PostgreSQL patch, knowing it was a big job and few were likely to volunteer. Then schedules lined up just right, and last night I managed to get a great group of people all together to do perhaps the biggest single patch review ever, to work on just that. I gathered up a list of the biggest concerns about this feature and its associated implementation, we got a number of regular PostgreSQL hackers and two of the security guys you've seen on this list all in the same room, and we talked about little but SEPostgreSQL for hours. Minutes are at http://wiki.postgresql.org/wiki/SEPostgreSQL_Review_at_the_BWPUG and I'd suggest anyone interested in this feature (or in rejecting this feature) to take a look at what we covered. I just wanted to add some talking notes here. User base for the feature: While my goals for this feature line up with military/government users this is in no way the extent of the potential user base. The fact is most people won't know they want this feature until it is available. Why is that? Well, how many of you have written webapps and implemented policy logic in your application rather than the database level? Why do people currently feel the need to do this? Is it even possible to implement some complex policies (eg., PCI compliance) at the database level? If PostgreSQL version whatever suddenly had the ability to implement the policy logic in the database, would you move it there? I know I would.. Audit: In past conversations it sounded like some of the Postgres community was skeptical even about the design of the security model. For an even earlier look (September 2006) of KaiGai and the SELinux community talking about the object model and even high level design of the solution see http://marc.info/?l=selinuxm=115762285013528w=2 I've read through some of the prior patches, but haven't done an extensive audit, not only because of the size but because it became apparent relatively quickly that it was a waste of time and the community here wasn't going to accept it anyway. If this situation changes I think you'll find a few of us willing to donate time to the cause. Policy: The policy is easy once you have an object model that covers your use cases. You can see in the above discussion how we came to the object model we have now and why I've been comfortable with it since then. An interesting aside, we must have hit the object model pretty well since another RDBMS (Trusted Rubix) uses the same one as SEPostgreSQL. See http://rubix.com/downloads/documentation/RX_SELinux_Guide_6_0_R4.pdf Works outside of SELinux: As Stephen already pointed out, Casey Schaufler (CC'd) who is the author of SMACK http://schaufler-ca.com/ believes that the abstractions provided by PGACE will allow integration of his security system as well. SMACK is already in the Linux kernel as an alternative LSM to SELinux. Further, not that I've seen the code or know how they did it, Trusted Rubix has support for multiple access control systems http://rubix.com/cms/features including Trusted Solaris, Solaris Trusted Extensions, SELinux and their own RBAC system. -- With all that said, I've very interested in seeing this work move along. In its current shape it has limited utility in my world (although I know of at least 2 solutions I've seen that run 20 Postgres servers on a single system just to have database separation). The main thing that prevents it from being used in that situation today is superuser access (eg., we can't have superusers that can go around and muck in data he's not cleared for). But I recognize that this is a first step to a potentially great system and I definitely want to see it moving forward. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Magnus, * Magnus Hagander (mag...@hagander.net) wrote: On Fri, Dec 11, 2009 at 05:45, Tom Lane t...@sss.pgh.pa.us wrote: It's been perfectly clear since day one, and was reiterated as recently as today http://archives.postgresql.org/message-id/4b21757e.7090...@2ndquadrant.com that what the security community wants is row-level security. The If that is true, then shouldn't we have an implementation of row level security *first*, and then an implementation of selinux hooks that work with this row level security feature? Rather than first doing selinux hooks, then row level security, which will likely need new and/or changed hooks... The proposal we're currently grappling with is to pull all the various checks which are sprinkled through our code into a single area. Clearly, if that work is done before we implement row-level security, then the patch for row-level security will just add it's checks in the security/ area and it'd be then easily picked up by SELinux, etc. I'm not convinced that row level security is actually that necessary (though it's a nice feature, with or without selinux), but if it is, it seems we are approaching the problem from the wrong direction. It has to be implemented independent of the security/SELinux/etc changes in any case, based on what was said previously.. So I don't particularly understand why it matters a great deal which one happens first. They're independently useful features, though both are not nearly as good on their own as when they are combined. Sorry, I just don't see this as a cart-before-the-horse case. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Adding support for SE-Linux security
On Fri, 2009-12-11 at 08:56 -0500, Stephen Frost wrote: [snip...] I do assume we're going to do row level security, but I do not feel that we need to particularly put one in front of the other. I also feel that SEPG will be valuable even without row-level security. One of the realms that we discussed at BWPUG for this is PCI compliance. I'm hopeful Josh will have an opportunity to review the PCI compliance cheat-sheet that I recall Robert Treat offering and comes to agreement that SEPG w/o row-level security would greatly improve our ability to have a PCI compliant system backed with PG. So I downloaded and read through the PCI DSS document (74 pages is pretty light compared to NFSv4.1 hehe...) and There are several areas there where I think strong access controls in the database will not only fulfill the requirement but provide much stronger guarantees than can be provided from the application server alone. The requirements in section 7 can definitely benefit from SEPG. If you implement these requirements in the application server and in PG access controls alone there is still an attack vector where a malicious user manages to steal the credentials for a particular role. With PG-ACE you can write a security module (although SEPG already allows for this) to restrict access to the data using the existing role-based access controls in PG and then apply additional restrictions such as, only this program may act as this role or access this database. This provides better guarantees than exist in current PCI compliant implementations using PG today. Dave -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Largeobject Access Controls (r2460)
KaiGai Kohei wrote: Takahiro Itagaki wrote: KaiGai Kohei kai...@ak.jp.nec.com wrote: Tom Lane wrote: Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp writes: structnamepg_largeobject/structname should not be readable by the public, since the catalog contains data in large objects of all users. This is going to be a problem, because it will break applications that expect to be able to read pg_largeobject. Like, say, pg_dump. Is it a right behavior, even if we have permission checks on large objects? Can we use column-level access control here? REVOKE ALL ON pg_largeobject FROM PUBLIC; = GRANT SELECT (loid) ON pg_largeobject TO PUBLIC; Indeed, it seems to me reasonable. We use SELECT loid FROM pg_largeobject LIMIT 1 in pg_dump. We could replace pg_largeobject_metadata instead if we try to fix only pg_dump, but it's no wonder that any other user applications use such queries. I think to allow reading loid is a balanced solution. Right, I also remind this query has to be fixed up by other reason right now. If all the large objects are empty, this query can return nothing, even if large object entries are in pg_largeobject_metadata. metadata seems very vague. Can't we come up with a more descriptive name? Also, how will this affect pg_migrator? pg_migrator copies pg_largeobject and its index from the old to the new server. Is the format inside pg_largeobject changed by this patch? What happens when there is no entry in pg_largeobject_metadata for a specific row? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
--On 11. Dezember 2009 11:28:54 -0300 Alvaro Herrera alvhe...@commandprompt.com wrote: without compiled probes: AVG(2531.68) with compiled probes: AVG(2511.97) Were the probes enabled? Hmm, they were just compiled in, i didn't anything to instrument them with dtrace. I've just started a pgbench/dtrace run with instrumented probes aset_alloc, aset_free and aset_realloc which just counts the calls to them during pgbench, the first run gives me tps = 1035.045523 (excluding connections establishing) Ideas? -- Thanks Bernd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
David, * David P. Quigley (dpqu...@tycho.nsa.gov) wrote: So I downloaded and read through the PCI DSS document (74 pages is pretty light compared to NFSv4.1 hehe...) and There are several areas there where I think strong access controls in the database will not only fulfill the requirement but provide much stronger guarantees than can be provided from the application server alone. Thanks for taking a look! That sounds like excellent news. My apologies for attributing the action item to the wrong individual. :) The requirements in section 7 can definitely benefit from SEPG. I don't mean to be a pain, and we're all busy, but perhaps you could include a short description of what 'requirements in section 7' are.. It would help keep the mailing list archive coherent, and be simpler for folks who aren't familiar with PCI to play along. A link to the specific PCI DSS document you looked at would be an alternative, tho not as good as a 'dumbed-down' description. ;) That would at least avoid confusion over which document, since I presume there's more than one out there. Thanks again for looking over this! Treat, you've dealt alot with PCI in your commercial work; could you comment on this for the benefit of the list? I don't doubt David in the least, but it never hurts to have someone as lucky as yourself in frequent dealings with PCI compliance to provide any additional insight. Thanks! Stephen signature.asc Description: Digital signature
Re: [HACKERS] Adding support for SE-Linux security
* David P. Quigley (dpqu...@tycho.nsa.gov) wrote: On Fri, 2009-12-11 at 09:32 -0500, Robert Haas wrote: I think that we should try to move the PG default checks inside the hook functions. If we can't do that cleanly, it's a good sign that the hook functions are not correctly placed to enforce arbitrary security policy. Furthermore, it defeats what I think would be a good side goal here, which is to better modularize the existing code. So from the meeting on Wednesday I got the impression that Steve already did this. However it was rejected because too much information was need to be passed around. KaiGai did all the work, but it was my suggestion to go down this route and I reviewed KaiGai's patch to do it. The specific 'review/rejection' email is here: http://archives.postgresql.org/message-id/10495.1255627...@sss.pgh.pa.us I gathered and I could be wrong but the reason for this is that instead of developing proper abstractions for the security code people made use of whatever local stuff was available to them at that location. That's certainly one concern I continue to have, but as I've been rereading the threads I'm less confident it's a huge problem- the issue seemed to be more about the single access_control.c knowing about the entire PG world. With the X-ACE work that Eamon Walsh did he did exactly what you're saying. He figured out the object model for the X-Server and created the hook framework. In the merge of the hook framework he also moved the existing X server security model into the framework. It's great to hear specific examples of other projects which have gone through this headache and come out the other side better for it. What I would suggest is that you develop a version of that patch that is stripped down to apply to only a single object type (databases? tables and columns - these might have to be together??) and which addresses Tom's criticisms from the last time around, and post that (on a new thread) for discussion. That will be much easier to review (and I will personally commit to reviewing it) but should allow us to flush out some of the design issues. If we can get agreement on that as a concept patch, we can move on to talking about which object types should be included in a committable version of that patch. They may have been said before but what exactly are the design issues? Unfortunately, design isn't nearly as well defined a term as one would hope. I believe KaiGai's latest suggestion (which is probably what his original PGACE implementation was, but I'm going to avoid looking, the community has enough egg on it's face already wrt this :/) is a good approach, along with splitting the huge access_control.c file into per-object pieces. That's a design change, tho perhaps not the kind of one others who have commented on the design were thinking about when they made those statements. Basically, there's the design of the code layout and how each piece knows about the other pieces (through header files, etc), and then there's the design of the function calls/ABI and actual code paths which are taken when the code is executed (which doesn't particularly care how the code is laid out in the source tree). I feel like the design issues raised have been more about the former and less about the latter. The main concern I hear is that people are worried that this is an SELinux specific design. I heard at the meeting on Wednesday that the Trusted Extensions people looked at the framework and said it meets their needs as well. If thats the case where does the concept that the design is SELinux specific stem from? We've asked Casey Schaufler the developer of another label based MAC system for Linux to look at the hooks as well and make a statement about their usability. Hope I didn't steal your thunder wrt Casey! Thanks again. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Adding support for SE-Linux security
On Fri, Dec 11, 2009 at 10:07 AM, David P. Quigley dpqu...@tycho.nsa.gov wrote: On Fri, 2009-12-11 at 09:32 -0500, Robert Haas wrote: 2009/12/11 KaiGai Kohei kai...@ak.jp.nec.com: It tried to provide a set of comprehensive entry points to replace existing PG checks at once. However, the SE-PgSQL/Lite patch covers accesses on only database, schema, tables and columns. Is it necessary to be comprehensive from the beginning? It might be too aggressive changes at once. Frankly, I hesitate to salvage the patch once rejected, as is. If we implement a set of security hooks, It seems to me the following approach is reasonable: * It does not touch the existing PG default checks. The purpose of security hooks are to host enhanced security features. * It does not deploy hooks on which no security provider is now proposed. It is important to reduce unnecessary changeset. I think that we should try to move the PG default checks inside the hook functions. If we can't do that cleanly, it's a good sign that the hook functions are not correctly placed to enforce arbitrary security policy. Furthermore, it defeats what I think would be a good side goal here, which is to better modularize the existing code. So from the meeting on Wednesday I got the impression that Steve already did this. However it was rejected because too much information was need to be passed around. I am not sure who Steve is or which patch you're talking about, but suffice it to say that I think the problem you are articulating here is exactly the one we need to get out from under. I don't know how to do that yet and... They may have been said before but what exactly are the design issues? ...that's the design issue I think we need to surmount. I think it will be easier to talk through that with a mini-patch that only affects one object type. I'll stop here because I see that Stephen Frost has just sent an insightful email on this topic as well. Hmm, maybe that's the Steve you were referring to. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL/Lite Review
Joshua Brindle wrote: Greg Smith wrote: It's funny; we started out this CommitFest with me scrambling to find someone, anyone, willing to review the latest SE-PostgreSQL patch, knowing it was a big job and few were likely to volunteer. Then schedules lined up just right, and last night I managed to get a great group of people all together to do perhaps the biggest single patch review ever, to work on just that. I gathered up a list of the biggest concerns about this feature and its associated implementation, we got a number of regular PostgreSQL hackers and two of the security guys you've seen on this list all in the same room, and we talked about little but SEPostgreSQL for hours. Minutes are at http://wiki.postgresql.org/wiki/SEPostgreSQL_Review_at_the_BWPUG and I'd suggest anyone interested in this feature (or in rejecting this feature) to take a look at what we covered. I just wanted to add some talking notes here. User base for the feature: While my goals for this feature line up with military/government users this is in no way the extent of the potential user base. The fact is most people won't know they want this feature until it is available. Why is that? Well, how many of you have written webapps and implemented policy logic in your application rather than the database level? Why do people currently feel the need to do this? Is it even possible to implement some complex policies (eg., PCI compliance) at the database level? If PostgreSQL version whatever suddenly had the ability to implement the policy logic in the database, would you move it there? I know I would.. Audit: In past conversations it sounded like some of the Postgres community was skeptical even about the design of the security model. For an even earlier look (September 2006) of KaiGai and the SELinux community talking about the object model and even high level design of the solution see http://marc.info/?l=selinuxm=115762285013528w=2 I highly suggest a quick read of the above thread, it shows how we established an object model in fairly short order. The conversation also continues here: http://marc.info/?l=selinuxm=115786457722767w=2 and also here: http://marc.info/?l=selinuxm=117160445604805w=2 http://marc.info/?l=selinuxm=117160445611588w=2 http://marc.info/?l=selinuxm=117160445608517w=2 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL/Lite Review
KaiGai, * KaiGai Kohei (kai...@kaigai.gr.jp) wrote: As Rober Haas already suggested in another message, my patch in the last commit fest is too large. It tried to rework anything in a single patch. The per-object-type basis make sense for me. Agreed. In my cosmetic preference, ace_ is better than ac_. The 'e' means extendable, and ace feels like something cool. :-) No complaints here.. I just hope this doesn't end up being *exactly* the same as your original PGACE patches.. I'd feel terrible if we weren't able to at least improve something with this extremely long and drawn our process! And I'd like to store these files in backend/security/ace/*.c, because backend/security will also store other security modules! This is perfectly reasonable in my view. No complaints here. src/ + backend/ + security/ + ace/ ... access control framework + sepgsql/ ... selinux specific code + smack/ ... (upcoming?) smack specific code + solaristx/ ... (upcoming?) solaris-tx specific code Looks good to me. Perhaps we'll have a smack/ patch showing up very shortly as well.. The reason why I prefer the default PG check + one enhanced security at most is simplification of the security label management. If two label-based enhanced securities are enabled at same time, we need two field on pg_class and others to store security context. Ah, yes, I see your point must more clearly now. This sounds reasonable to me. In addition, OS allows to choose one enhanced security at most eventually. In my image, the hook should be as: Value * ac_database_create([arguments ...]) { /* * The default PG checks here. * It is never disabled */ /* enhanced security as follows */ #if defined(HAVE_SELINUX) if (sepgsql_is_enabled()) return sepgsql_database_create(...); #elif defined(HAVE_SMACK) if (smack_is_enabled()) return smack_database_create(...); #elif defined(HAVE_SOLARISTX) if (soltx_is_enabled()) return soltx_database_create(...); #endif return NULL; } We can share a common image image? If all of these security modules make sense as only-more-restrictive, then I have no problem with this approach. Honestly, I'm fine with the initial hooks looking as above in any case, since it provides a clear way to deal with switching out the 'default PG checks' if someone desires it- you could #if around that block as well. As it's the principle/primary, and I could see only-more-restrictive being more popular, I don't mind having that code in-line in the hook. The check itself is still being brought out and into the security/ framework. I do think that, technically, there's no reason we couldn't allow for multiple only-more-restrictive models to be enabled and built in a single binary for systems which support it. As such, I would make those just #if defined() rather than #elif. Let it be decided at runtime which are actually used, otherwise it becomes a much bigger problem for packagers too. Thanks! Stephen signature.asc Description: Digital signature
Re: [HACKERS] SE-PostgreSQL/Lite Review
Greg, * Greg Smith (g...@2ndquadrant.com) wrote: I think we need a two pronged attack on this issue. Eventually I think someone who wants this feature in there will need to sponsor someone (and not even necessarily a coder) to do a sizable round of plain old wording cleanup on the comment text of the patch. For the benefit of this list (this was already discussed some at the BWPUG meeting), I agree with Greg on this and am actively looking to try and help (either directly in my spare time or indirectly as a sponsor). If anyone else is interested or can do so as well, that would be great! Please speak up! Thanks! Stephen signature.asc Description: Digital signature
Re: [HACKERS] Adding support for SE-Linux security
* Robert Haas (robertmh...@gmail.com) wrote: I'll stop here because I see that Stephen Frost has just sent an insightful email on this topic as well. Hmm, maybe that's the Steve you were referring to. I have doubts- but then I don't ever see my comments as insightful for some reason. ;) Thanks! Stephen signature.asc Description: Digital signature
Re: [HACKERS] EXPLAIN BUFFERS
Robert Haas escreveu: On Thu, Dec 10, 2009 at 9:35 PM, Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp wrote: Anyway, a revised patch according to the comments is attached. The new text format is: Buffers: shared hit=675 read=968, temp read=1443 written=1443 * Zero values are omitted. (Non-text formats could have zero values.) * Rename Blocks: to Buffers:. * Remove parentheses and add a comma between shared, local and temp. I did a bit of copy-editing of your doc changes to make the English a bit more correct and idiomatic. Slightly revised patch attached for your consideration. The output format looks really nice (thanks for bearing with me), and the functionality is great. Please, document that zero values are omitted in the text format. It seems intuitive but could be surprise because zero values are in non-text formats. -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Largeobject Access Controls (r2460)
KaiGai Kohei wrote: We use SELECT loid FROM pg_largeobject LIMIT 1 in pg_dump. We could replace pg_largeobject_metadata instead if we try to fix only pg_dump, but it's no wonder that any other user applications use such queries. I think to allow reading loid is a balanced solution. Right, I also remind this query has to be fixed up by other reason right now. If all the large objects are empty, this query can return nothing, even if large object entries are in pg_largeobject_metadata. metadata seems very vague. Can't we come up with a more descriptive name? What about property? The metadata was the suggested name from Robert Haas at the last commit fest, because we may store any other properties of a large object in this catalog future. Well, we usually try to be more specific about what something represents and only later abstract it out if needed, but if everyone else is fine with 'metadata', then just leave it unchanged. Also, how will this affect pg_migrator? pg_migrator copies pg_largeobject and its index from the old to the new server. Is the format inside pg_largeobject changed by this patch? The format of pg_largeobject was not touched. Good. What happens when there is no entry in pg_largeobject_metadata for a specific row? In this case, these rows become orphan. So, I think we need to create an empty large object with same LOID on pg_migrator. It makes an entry on pg_largeobject_metadata without writing anything to the pg_largeobject. I guess rest of migrations are not difference. Correct? Uh, yea, pg_migrator could do that pretty easily. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Largeobject Access Controls (r2460)
Bruce Momjian さんは書きました: KaiGai Kohei wrote: Takahiro Itagaki wrote: KaiGai Kohei kai...@ak.jp.nec.com wrote: Tom Lane wrote: Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp writes: structnamepg_largeobject/structname should not be readable by the public, since the catalog contains data in large objects of all users. This is going to be a problem, because it will break applications that expect to be able to read pg_largeobject. Like, say, pg_dump. Is it a right behavior, even if we have permission checks on large objects? Can we use column-level access control here? REVOKE ALL ON pg_largeobject FROM PUBLIC; = GRANT SELECT (loid) ON pg_largeobject TO PUBLIC; Indeed, it seems to me reasonable. We use SELECT loid FROM pg_largeobject LIMIT 1 in pg_dump. We could replace pg_largeobject_metadata instead if we try to fix only pg_dump, but it's no wonder that any other user applications use such queries. I think to allow reading loid is a balanced solution. Right, I also remind this query has to be fixed up by other reason right now. If all the large objects are empty, this query can return nothing, even if large object entries are in pg_largeobject_metadata. metadata seems very vague. Can't we come up with a more descriptive name? What about property? The metadata was the suggested name from Robert Haas at the last commit fest, because we may store any other properties of a large object in this catalog future. Also, how will this affect pg_migrator? pg_migrator copies pg_largeobject and its index from the old to the new server. Is the format inside pg_largeobject changed by this patch? The format of pg_largeobject was not touched. What happens when there is no entry in pg_largeobject_metadata for a specific row? In this case, these rows become orphan. So, I think we need to create an empty large object with same LOID on pg_migrator. It makes an entry on pg_largeobject_metadata without writing anything to the pg_largeobject. I guess rest of migrations are not difference. Correct? Thanks, -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Fri, Dec 11, 2009 at 10:07 AM, David P. Quigley dpqu...@tycho.nsa.gov wrote: The main concern I hear is that people are worried that this is an SELinux specific design. I heard at the meeting on Wednesday that the Trusted Extensions people looked at the framework and said it meets their needs as well. If thats the case where does the concept that the design is SELinux specific stem from? We've asked Casey Schaufler the developer of another label based MAC system for Linux to look at the hooks as well and make a statement about their usability. OK, on second thought I want to address this a little more, since some of these concerns came from me. SE-Linux-specific may be the wrong way to say it. There's an old saying that goes something like: if a function has 10 parameters, it has 11 parameters. In other words, if you're passing too much information across a supposed abstraction boundary, it's not really an abstraction boundary at all. If we design a security abstraction layer, the interfaces need to really be abstraction boundaries. Passing the table OID and then also the tablespace OID because PG DAC needs that to make its access control decision is crap. As soon as some other security model comes along you will need to add additional arguments for all of the things that the new security model cares about, and when a third security model comes along you will need to pass those things now too. If we're doing that sort of thing, we might as well leave the entire jumble of spaghetti code in its original location so that at least you don't have to flip back and forth between two different source files. And if we're going to do that, we might as well quit now because our heads will explode (Tom's first). I actually have an idea how to solve the problem in this particular case, but I'm reluctant to say what it is because I'm not sure if I'm right, and at any rate *I don't want to write this patch*. I may review it and if it's good I may commit it (or more likely a more extensive version of it after we have agreement on the basic principles) but I don't to want to write it or do substantial cleanup of it. I have my own list of things to work on and I presently have no reason to put this on that list. As a community, I think that at times we have a tendency to say well, the committer is just going to rewrite this anyway, so it doesn't really matter what we do. As far as I am concerned that is flat false. I do not want to rewrite your patch. I want to look at your patch, convince myself that it is correct, and commit it. That won't always happen, and I'm certainly willing to do more than that especially in cases where it's a feature that I really care about, but AFAIAC the initiative is with the patch author to provide something committable, NOT with the committer to fix what the patch author did wrong. I realize that's subjective at times, and I'm going to make my best effort not to be a jerk about it, but given that there are only so many hours in the day (and not all of the can be spent on PostgreSQL) I think that's the goal we need to shoot for. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Fri, 2009-12-11 at 11:28 -0500, Stephen Frost wrote: [snip...] The main concern I hear is that people are worried that this is an SELinux specific design. I heard at the meeting on Wednesday that the Trusted Extensions people looked at the framework and said it meets their needs as well. If thats the case where does the concept that the design is SELinux specific stem from? We've asked Casey Schaufler the developer of another label based MAC system for Linux to look at the hooks as well and make a statement about their usability. Hope I didn't steal your thunder wrt Casey! Thanks again. So we contacted Casey about another MAC model for PG using PG-ACE and he got back to us with these reponses. Josh Brindle (JB): So my question is, does smack have a facility for userspace object managers? Casey Schaufler (cs): Yes. The smack-util package includes a small library which supports a user space version of the kernel smackaccess() function. You pass it the subject label, object label, and desired access and it returns a yes/no answer based on what it reads from /smack/load. So this answers our questions on whether or not SMACK has the faculties to act as the security decision engine for PG. JB: If so, I want to make the argument that doing a smack integration using the pgace abstraction layer would not only work but be fairly easy. CS: Looking at some of the documentation I think that you can safely make that argument. The security_label column would just be the Smack label. The rules can be enforced by the user space smackaccess(). System rows, whatever that might be, could get the floor (_) label. Casey mentions the row level access control in here but its safe to say we've broken row based access control into a followup discussion/project. JB: All the sepgsql docs and code are up at http://code.google.com/p/sepgsql/ and I'd like to get your feedback before I start making claims... CS: I can't see how it would take more than about a day if pgace does what it looks like it should. This seems to be a favorable assesment of the pgace framework's ability to be used by something other than SELinux. So Casey's Smack module plus the Sun guys saying it is usable by their legacy TSOL or TX code would lend credence to the idea that pgace is bringing to the table. It may be possible that you're not happy with certain aspects of the implementation but the objects and permissions listed in pgace are definitely ones that are worth mediating. Dave Dave -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Fri, 2009-12-11 at 11:16 -0500, Stephen Frost wrote: David, * David P. Quigley (dpqu...@tycho.nsa.gov) wrote: So I downloaded and read through the PCI DSS document (74 pages is pretty light compared to NFSv4.1 hehe...) and There are several areas there where I think strong access controls in the database will not only fulfill the requirement but provide much stronger guarantees than can be provided from the application server alone. Thanks for taking a look! That sounds like excellent news. My apologies for attributing the action item to the wrong individual. :) Nahh you attributed it to the correct person I just got a little bored yesterday and poked my nose into it :) The requirements in section 7 can definitely benefit from SEPG. I don't mean to be a pain, and we're all busy, but perhaps you could include a short description of what 'requirements in section 7' are.. It would help keep the mailing list archive coherent, and be simpler for folks who aren't familiar with PCI to play along. A link to the specific PCI DSS document you looked at would be an alternative, tho not as good as a 'dumbed-down' description. ;) That would at least avoid confusion over which document, since I presume there's more than one out there. So the document I read is linked below [1]. Requirement 7 falls under Implement Strong Access Control Measures. The two main requirements under requirement 7 seem to be limit access to system components and card holder data to only those individuals whose job requires such access and Establish an access control system for systems components with multiple users that restricts access based on a user’s need to know, and is set to “deny all” unless specifically allowed. The major flaw with this system is that stolen credentials allow you to completely bypass the logic for this in the application layer. If the person manages to get the login and password for an account on the database it doesn't matter what their authenticated use is because the logic to prevent accountant bob from cutting his own payroll check is in the application layer. The way that MAC controls can strengthen the protections by making sure that only certain components can perform certain actions. If you want to make sure that only the accounting application can mess with that data regardless of whether you have the database credentials or not then you can do that. You can write policy for SEPG to say only programs with this label can perform these actions on the database. Only applications labeled with the accounting_tool label can modify the table labeled accounting_data. This way if the system is compromised and someone manages to get the username and password for the database account/role that you were protecting the table with since the request isn't coming from a process labeled accounting_tool the database will deny those accesses. This is why I mean by adding stronger protections. This way you've minimized the amount of code that you have to accredit for compliance in your application server. [1] https://www.pcisecuritystandards.org/security_standards/download.html?id=pci_dss_v1-2.pdf Thanks again for looking over this! Treat, you've dealt alot with PCI in your commercial work; could you comment on this for the benefit of the list? I don't doubt David in the least, but it never hurts to have someone as lucky as yourself in frequent dealings with PCI compliance to provide any additional insight. It is definitely good to have a second opinion on this since I've just only started reading the PCI compliance documents. I'm definitely not an expert in PCI compliance but from what I've read there are definite benefits for using SEPG or PG-ACE with a special security module in making much stronger guarantees about who and what can touch the card data. Dave -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Bernd Helmle píše v pá 11. 12. 2009 v 17:13 +0100: --On 11. Dezember 2009 11:28:54 -0300 Alvaro Herrera alvhe...@commandprompt.com wrote: without compiled probes: AVG(2531.68) with compiled probes: AVG(2511.97) Were the probes enabled? Hmm, they were just compiled in, i didn't anything to instrument them with dtrace. I've just started a pgbench/dtrace run with instrumented probes aset_alloc, aset_free and aset_realloc which just counts the calls to them during pgbench, the first run gives me tps = 1035.045523 (excluding connections establishing) Ideas? When probes are activated they have performance impact. It is normal. Important is that you can use it when you need it on production system. No recompilation, no extra binary, no downtime and it is safe. Performance impact depends on Dscript Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL/Lite Review
On Fri, 2009-12-11 at 11:36 -0500, Stephen Frost wrote: [Snip...] In addition, OS allows to choose one enhanced security at most eventually. In my image, the hook should be as: Value * ac_database_create([arguments ...]) { /* * The default PG checks here. * It is never disabled */ /* enhanced security as follows */ #if defined(HAVE_SELINUX) if (sepgsql_is_enabled()) return sepgsql_database_create(...); #elif defined(HAVE_SMACK) if (smack_is_enabled()) return smack_database_create(...); #elif defined(HAVE_SOLARISTX) if (soltx_is_enabled()) return soltx_database_create(...); #endif return NULL; } We can share a common image image? If all of these security modules make sense as only-more-restrictive, then I have no problem with this approach. Honestly, I'm fine with the initial hooks looking as above in any case, since it provides a clear way to deal with switching out the 'default PG checks' if someone desires it- you could #if around that block as well. As it's the principle/primary, and I could see only-more-restrictive being more popular, I don't mind having that code in-line in the hook. The check itself is still being brought out and into the security/ framework. I do think that, technically, there's no reason we couldn't allow for multiple only-more-restrictive models to be enabled and built in a single binary for systems which support it. As such, I would make those just #if defined() rather than #elif. Let it be decided at runtime which are actually used, otherwise it becomes a much bigger problem for packagers too. So when Eamon did the XACE work he allowed for stackable security modules. We do not allow for this in the Linux kernel but we can still figure out whether we want to support it for now. I'm not convinced that the ifdef model is the right way to go. There should be a security structure where pointers to the appropriate functions reside (like we have in linux) so this way you have a register_module call which takes the mac model and puts all the necessary function pointers into the structure and the hook is just a call to security-create_db. If you want to make it stackable you make security-create_db a list of function pointers that get executed in turn. I'll ask Eamon when I see him next how he handled deregistration of a particular model. Dave -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
On Fri, Dec 11, 2009 at 12:55 PM, Zdenek Kotala zdenek.kot...@sun.com wrote: Bernd Helmle píše v pá 11. 12. 2009 v 17:13 +0100: --On 11. Dezember 2009 11:28:54 -0300 Alvaro Herrera alvhe...@commandprompt.com wrote: without compiled probes: AVG(2531.68) with compiled probes: AVG(2511.97) Were the probes enabled? Hmm, they were just compiled in, i didn't anything to instrument them with dtrace. I've just started a pgbench/dtrace run with instrumented probes aset_alloc, aset_free and aset_realloc which just counts the calls to them during pgbench, the first run gives me tps = 1035.045523 (excluding connections establishing) Ideas? When probes are activated they have performance impact. It is normal. Important is that you can use it when you need it on production system. No recompilation, no extra binary, no downtime and it is safe. Performance impact depends on Dscript Yeah. The problem here is the impact when the probes aren't enabled. I thought we had an idea of using the AllocSet dispatch mechanism to make this zero-overhead in the case where the probes are not enabled. What happened to that notion? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Robert Haas píše v čt 10. 12. 2009 v 23:55 -0500: On Wed, Dec 9, 2009 at 9:04 AM, Zdenek Kotala zdenek.kot...@sun.com wrote: But in normal situation database does also other thing and palloc is only one part of code path. It is why I run second test and use sun studio profiling tools (collect/analyzer) to determine how much CPU ticks cost the probes during pg_bench run. And results are much better. AllocSet alloc function takes about 4-5% and probes assembler code takes 0.1-0.2% on 64bit. I did not test 32bit but my expectation is that it should be about 0.3-0.4%. There's not really enough detail here to determine what you tested and what the results were, and I don't think this patch has any chance at all of getting committed without that. Please clarify. If there's some real-world test where this probe costs 0.3%-0.4%, I think that is sufficient grounds for rejecting this patch. I understand the desire of people to be able to use dtrace, but our performance is too hard-won for me to want to give any measurable of it up for tracing and instrumentation hooks that will only be used by a small number of users in a small number of situations. As I mentioned I run pg_bench -c10 -t1000 and collect data from backends. collect and analyzer is similar tool to gprof. Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Robert Haas píše v pá 11. 12. 2009 v 12:55 -0500: On Fri, Dec 11, 2009 at 12:55 PM, Zdenek Kotala zdenek.kot...@sun.com wrote: Bernd Helmle píše v pá 11. 12. 2009 v 17:13 +0100: --On 11. Dezember 2009 11:28:54 -0300 Alvaro Herrera alvhe...@commandprompt.com wrote: without compiled probes: AVG(2531.68) with compiled probes: AVG(2511.97) Were the probes enabled? Hmm, they were just compiled in, i didn't anything to instrument them with dtrace. I've just started a pgbench/dtrace run with instrumented probes aset_alloc, aset_free and aset_realloc which just counts the calls to them during pgbench, the first run gives me tps = 1035.045523 (excluding connections establishing) Ideas? When probes are activated they have performance impact. It is normal. Important is that you can use it when you need it on production system. No recompilation, no extra binary, no downtime and it is safe. Performance impact depends on Dscript Yeah. The problem here is the impact when the probes aren't enabled. I thought we had an idea of using the AllocSet dispatch mechanism to make this zero-overhead in the case where the probes are not enabled. What happened to that notion? We know that performance impact is less then 1% probably less then 0.6%. The question is if it is acceptable or not. I personally think that it is acceptable. However if not, I will start work on backup solution with dtraced AllocSet and some switching mechanism. But it needs little discussion about design. And first we need decision about current performance impact. Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Fri, 2009-12-11 at 01:19 +0200, Peter Eisentraut wrote: On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: There was considerable debate earlier about whether we wanted to treat Python 3 as a separate PL so it could be available in parallel with plpython 2, because of the user-level coding incompatibilities. It looks like this patch simply ignores that problem. What is going to happen to plpython functions that depend on 2.x behavior? I have a proposal for how to handle this, and a prototype patch attached. This follows essentially what the CPython distribution itself does, which will make this tolerably easy to follow for users. We install plpython as plpython2.so or plpython3.so, depending on the version used to build it. Then, plpython.so is a symlink to plpython2.so. We then create three language definition templates: plpythonu - plpython.so plpython2u - plpython2.so plpython3u - plpython3.so Comments? Well as a Python guy... makes sense to me :) Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] thread safety on clients
I wrote: The crash is real --- I've replicated it here. Still trying to figure out what is the real cause. Okay, I've sussed it. The crash itself is due to a memory management mistake in the recently-rewritten EvalPlanQual code (pfree'ing a tuple that we still have live Datum references to). The reason the broken pgbench is able to trigger it is that it generates concurrent updates for the same row in pgbench_accounts with much higher frequency than normal. This forces the backends through the EvalPlanQual code, which is broken and crashes. QED. I'll commit a fix for that shortly, but this reminds me once again that the EvalPlanQual logic is desperately under-tested in our normal regression testing. We really need some kind of testing infrastructure that would let us exercise concurrent-update cases a bit more than not at all. Also, the reason why Bruce's mistake exposed this is interesting. Omitting the #define for ENABLE_THREAD_SAFETY does not actually break pgbench -j at all -- it has a fallback strategy that uses multiple subprocesses instead of multiple threads. However, it has only one srandom() call, which occurs in the parent process before forking. This means that the subprocesses all start with the same random number state, which means they generate the same sequence of random account IDs to update, which is why the probability of an update collision requiring EvalPlanQual resolution is so high, especially at the start of the run (and the crash does happen pretty much immediately after starting pgbench). I'm inclined to think this is bad, and we should fix pgbench to re-randomize after forking. If we don't, we'll have synchronized behavior on some platforms and not others, which doesn't seem like a good idea. On the other hand, if you did want that type of behavior, it's hard to see how you'd get it. Is it worth trying to provide that as a (probably non-default) mode in pgbench? If so, how would you do it on threaded platforms? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
On Fri, Dec 11, 2009 at 1:04 PM, Zdenek Kotala zdenek.kot...@sun.com wrote: We know that performance impact is less then 1% probably less then 0.6%. The question is if it is acceptable or not. I personally think that it is acceptable. However if not, I will start work on backup solution with dtraced AllocSet and some switching mechanism. But it needs little discussion about design. And first we need decision about current performance impact. As far as I am concerned that is way too much, particularly considering that your test case isn't designed to be particularly memory-allocation intensive, and if it is up to me I will reject this. Even a quarter-percent slowdown for a feature that will be used only by a small fraction of users only a small fraction of time time seems totally unacceptable to me. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Fri, 2009-12-11 at 11:30 -0500, Robert Haas wrote: [snip...] I'll stop here because I see that Stephen Frost has just sent an insightful email on this topic as well. Hmm, maybe that's the Steve you were referring to. ...Robert Yea I never asked Stephen if he goes by Stephen or Steve when I met him on Wednesday. I guess calling him Steve is me being a bit presumptuous :) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL/Lite Review
Stephen Frost wrote: KaiGai, snip I do think that, technically, there's no reason we couldn't allow for multiple only-more-restrictive models to be enabled and built in a single binary for systems which support it. As such, I would make those just #if defined() rather than #elif. Let it be decided at runtime which are actually used, otherwise it becomes a much bigger problem for packagers too. It isn't just a case of using #if and it magically working. You'd need a system to manage multiple labels on each object that can be addressed by different systems. So instead of having an object mapped only to system_u:object_r:mydb_t:s15 you'd also have to have it mapped to, eg., ^ for Smack. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Robert, * Robert Haas (robertmh...@gmail.com) wrote: I actually have an idea how to solve the problem in this particular case, but I'm reluctant to say what it is because I'm not sure if I'm right, and at any rate *I don't want to write this patch*. As far as crap goes, I'd have to put this at the top. If you're not willing to share ideas, then I may have to reconsider my personal feeling on if you should be a committer or not. No one is asking you to write the patch. We all know that we can be wrong (I tend to be more wrong than most), and we all hate to jerk people around, but I feel it's far worse to self-censor discussion on ideas. It's also about the worst form of rock-management that I think I could come up with in an open source community. If you don't share your idea, yet you feel that it's right, and see nothing to dissuade you from that position (after all, we can't present an argument for or against it if we don't know what it is), then I find it likely that you're going to constantly be comparing patches presented to the ideal one in your head based on your idea and we'd never get there. If you're not willing to discuss your idea, then I would ask that you not be involved in either this discussion or review of the patch. Rock-management, for those not familiar with the term, is essentially asking I need a rock, and then, when presented with a rock saying sorry, that's not the rock I wanted. This doesn't provide any insight into what kind of rock you're looking for. Rest of the I don't want to write it whine omitted. No one's asking you to. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Robert Haas robertmh...@gmail.com writes: As far as I am concerned that is way too much, particularly considering that your test case isn't designed to be particularly memory-allocation intensive, and if it is up to me I will reject this. Even a quarter-percent slowdown for a feature that will be used only by a small fraction of users only a small fraction of time time seems totally unacceptable to me. It seems to me that anyone who really needs this can instrument the alloc functions anyway --- isn't one of the features of DTrace supposed to be that you can monitor calls to a particular function without any prearranged code support? Or is that one of the things like zero overhead that turns out to be more marketing-speak than reality? Anyway I concur with Robert's opinion that the use-case is far too small to justify incurring a measurable overhead for everybody. There might be some small argument for putting these in under an extra #ifdef, but they wouldn't get into any regular production build. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
All, * Robert Haas (robertmh...@gmail.com) wrote: If we design a security abstraction layer, the interfaces need to really be abstraction boundaries. Passing the table OID and then also the tablespace OID because PG DAC needs that to make its access control decision is crap. Now, to address the small useful bit of this mail- I tend to agree with this. I'm not convinced there's an alternative, but I'd like to throw out a couple of my ideas on how it could be addressed. I'd like to solicit for feedback on these. First off, we have alot of the information available from the catalog. Basically, given an OID, we should be able to find out the other information associated with that OID (such as what kind of object it is, what tablespace it resides in, etc). OID isn't sufficient, however, as we know from the dependency system. To address this, we would need OID and SubOID. Now, any information which we can derive from those should not be included in the API. To be honest, I don't think we've actually been all that bad about this, but I'll reserve any definitive answer until I've gone back through the API we have specifically thinking about this issue. On further thought, I'm probably wrong and should have caught this during my review. Sorry. Second, the information we *don't* have from above is generally information about what the requesting action is. For example, when changing ownership of an object, we can't possibly use introspection to find out the role which is on the receiving end, since at that time we've just learned it from the parser. Now, we might build an entire new object, pass the result of this action OID to the security system and ask it can we change OID X into OID Y?, but I don't particularly like it. We really don't want to do any *changes* to things until after we've determined the permissions allow for it, and I'm not sure how to get around that without building an independent introspection system for what might be. Perhaps we're comfortable enough saying we'll just rollback the command if it turns out to have not been allowed but I just don't like it. Feels like a higher risk solution to me. I don't see a way to get around the second piece since what information is needed about the action is typically action-specific. Perhaps we could have an 'action-ID' (uh, we have an ID per command already, no? Probably doesn't fit the mold close enough though), and then a way to query information about what is this action trying to do?. Doesn't seem likely to be very clean though. Other thoughts? Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Robert Haas robertmh...@gmail.com writes: I thought we had an idea of using the AllocSet dispatch mechanism to make this zero-overhead in the case where the probes are not enabled. What happened to that notion? I must have missed that discussion, but +1 --- should be possible to get to zero-overhead-when-off that way. The trick is to figure out what/where enables the alternate implementation. The current design assumes that the callers of FooContextCreate choose the implementation, but we don't want that here. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] thread safety on clients
Tom Lane wrote: Also, the reason why Bruce's mistake exposed this is interesting. Omitting the #define for ENABLE_THREAD_SAFETY does not actually break pgbench -j at all -- it has a fallback strategy that uses multiple subprocesses instead of multiple threads. However, it has only one srandom() call, which occurs in the parent process before forking. This means that the subprocesses all start with the same random number state, which means they generate the same sequence of random account IDs to update We just got that as a bug report the other day too, with suggested fixes: http://archives.postgresql.org/pgsql-hackers/2009-12/msg00841.php I'm inclined to think this is bad, and we should fix pgbench to re-randomize after forking. If we don't, we'll have synchronized behavior on some platforms and not others, which doesn't seem like a good idea. On the other hand, if you did want that type of behavior, it's hard to see how you'd get it. Is it worth trying to provide that as a (probably non-default) mode in pgbench? If so, how would you do it on threaded platforms? It sounds like random pgbench run for a while would certainly expose the same thing you're concerned about eventually. I doubt it's worth the trouble to codify a bug just because it found another bug (it's hard enough to maintain code that only has to work right). If we want to stress this behavior, it would be easier to just test with a a bunch of clients going through a scale=1 database and use enough transactions to make update collisions likely. I'm working on a guide to stress testing new alpha builds with pgbench that will be ready in time for alpha3. I could easily add that as one of the suggested tests to run if you're concerned about getting more test coverage of that code path. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] thread safety on clients
Greg Smith g...@2ndquadrant.com writes: It sounds like random pgbench run for a while would certainly expose the same thing you're concerned about eventually. Yeah. Actually the odd thing about it is that the crash seemed to invariably be on conflicting pgbench_accounts updates, which is a fairly low-contention table in this test design (but the bug turned it into high-contention). What I would have expected is crashes on the very similar updates to pgbench_branches, which is designed to be high-contention. It might be that there is some other effect going on here that explains why that wasn't happening. Need to go back and look more closely. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Tom Lane píše v pá 11. 12. 2009 v 13:56 -0500: Robert Haas robertmh...@gmail.com writes: As far as I am concerned that is way too much, particularly considering that your test case isn't designed to be particularly memory-allocation intensive, and if it is up to me I will reject this. Even a quarter-percent slowdown for a feature that will be used only by a small fraction of users only a small fraction of time time seems totally unacceptable to me. It seems to me that anyone who really needs this can instrument the alloc functions anyway --- isn't one of the features of DTrace supposed to be that you can monitor calls to a particular function without any prearranged code support? Or is that one of the things like zero overhead that turns out to be more marketing-speak than reality? There are several types of probes. For example for PID provider probes you can monitor all entry and return point from global function. And also you can put probe on each asm instruction in the function. These probes have zero overhead, because dtrace understand ABI and know where args are. Unfortunately user defined probes has small overhead which is price for universal solution which works with all compilers and linkers. Anyway I concur with Robert's opinion that the use-case is far too small to justify incurring a measurable overhead for everybody. OK. There might be some small argument for putting these in under an extra #ifdef, but they wouldn't get into any regular production build. unfortunately #ifdef solution kills main dtrace goal - without recompilation :(. Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Fri, Dec 11, 2009 at 1:52 PM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: I actually have an idea how to solve the problem in this particular case, but I'm reluctant to say what it is because I'm not sure if I'm right, and at any rate *I don't want to write this patch*. As far as crap goes, I'd have to put this at the top. If you're not willing to share ideas, then I may have to reconsider my personal feeling on if you should be a committer or not. No one is asking you to write the patch. We all know that we can be wrong (I tend to be more wrong than most), and we all hate to jerk people around, but I feel it's far worse to self-censor discussion on ideas. OK, it's clear that I've handled this badly. Sorry. My fear (however unjustified) was that someone would go and rewrite the patch based on an opinion that I express whether they agree with it or not. I don't know the right way to do this and I'm sorry to have given you the impression that I think I do and am hiding the ball. So with that said, the idea I had was to try to pass around pre-existing data structures related to the objects on which access control decisions are being made, rather than Oids. It's pretty clear that you're never going to be able to make an access control decision just based on the Oid, but the Relation descriptor or pg_something HeapTuple might be enough - or at least whatever else you need is likely something you would have had to look up anyway, even without the interface layer. I don't know if that makes sense or actually works, but you could give it a try. It's also about the worst form of rock-management that I think I could come up with in an open source community. If you don't share your idea, yet you feel that it's right, and see nothing to dissuade you from that position (after all, we can't present an argument for or against it if we don't know what it is), then I find it likely that you're going to constantly be comparing patches presented to the ideal one in your head based on your idea and we'd never get there. It's a little unfortunate that we're arguing about this because that's exactly what I'm reacting AGAINST, and emphatically not when I intend to do. I think this is the first time in my adult life I've been criticized for being too UN-willing to share my opinions, but I guess there's a first time for everything. Again, sorry for handling this badly. I just feel like the discussions that we've had so far have been very much in the dynamic of throw some code over the wall and see if the committer likes it... looks like no, let's go back around and try again. It does have a bit of a rock management feel to it and I really want to see if we can find a way to break that cycle. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Tom Lane píše v pá 11. 12. 2009 v 14:38 -0500: Robert Haas robertmh...@gmail.com writes: I thought we had an idea of using the AllocSet dispatch mechanism to make this zero-overhead in the case where the probes are not enabled. What happened to that notion? I must have missed that discussion, but +1 --- should be possible to get to zero-overhead-when-off that way. The trick is to figure out what/where enables the alternate implementation. The current design assumes that the callers of FooContextCreate choose the implementation, but we don't want that here. I thought about it. I think we can use GUC variable (e.g. dtraced_alloc) and hook switch pointers to dtraced AsetFunctions. The problem is how to distribute to all backend. Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
David, * David P. Quigley (dpqu...@tycho.nsa.gov) wrote: So the document I read is linked below [1]. Great, thanks again. [agree with all the rest] It is definitely good to have a second opinion on this since I've just only started reading the PCI compliance documents. I'm definitely not an expert in PCI compliance but from what I've read there are definite benefits for using SEPG or PG-ACE with a special security module in making much stronger guarantees about who and what can touch the card data. Indeed. The other nice piece about getting the opinion of Treat (or others who have to deal with PCI) is that while the PCI documentation says what you're supposed to do, the PCI folks also have auditing requirments (as in, a third-party vendor has to audit your system, and there are required scans and scanning tools, etc) which don't always marry up to what they say they require. Thanks! Stephen signature.asc Description: Digital signature
Re: [HACKERS] Adding support for SE-Linux security
* David P. Quigley (dpqu...@tycho.nsa.gov) wrote: Yea I never asked Stephen if he goes by Stephen or Steve when I met him on Wednesday. I guess calling him Steve is me being a bit presumptuous :) Oh, either is fine, tho people will probably follow a bit better if you say Stephen. As a reminder- KaiGai's the one who did all the actual code, I just tried to steer him in a direction I thought made sense to make it easier to get core to accept it, and reviewed the results of his work. Thanks! Stephen signature.asc Description: Digital signature
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Zdenek Kotala zdenek.kot...@sun.com writes: I thought about it. I think we can use GUC variable (e.g. dtraced_alloc) and hook switch pointers to dtraced AsetFunctions. The problem is how to distribute to all backend. You set the GUC in postgresql.conf. No big deal. If we go this route it would be nice to think about making a facility that has some usefulness for non-DTrace platforms too. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL/Lite Review
Josh, * Joshua Brindle (met...@manicmethod.com) wrote: Stephen Frost wrote: I do think that, technically, there's no reason we couldn't allow for multiple only-more-restrictive models to be enabled and built in a single binary for systems which support it. As such, I would make those just #if defined() rather than #elif. Let it be decided at runtime which are actually used, otherwise it becomes a much bigger problem for packagers too. It isn't just a case of using #if and it magically working. You'd need a system to manage multiple labels on each object that can be addressed by different systems. So instead of having an object mapped only to system_u:object_r:mydb_t:s15 you'd also have to have it mapped to, eg., ^ for Smack. I'm not sure I see that being a problem.. We're going to have references in our object managers which make sense to us (eg: table OIDs) and then a way of mapping those to some label (or possibly a set of labels, as you describe). We might want to reconsider the catalog structure a bit if we want to support more than one at a time, but I don't see it as a huge problem to support more than one label existing for a given object. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Adding support for SE-Linux security
On Fri, Dec 11, 2009 at 2:11 PM, Stephen Frost sfr...@snowman.net wrote: Second, the information we *don't* have from above is generally information about what the requesting action is. For example, when changing ownership of an object, we can't possibly use introspection to find out the role which is on the receiving end, since at that time we've just learned it from the parser. I'm not sure that I follow what you're saying here. I think maybe it would help to discuss a concrete example, which is why I proposed a concept patch. Or perhaps you'd like just to pick out a specific case to discuss. Now, we might build an entire new object, pass the result of this action OID to the security system and ask it can we change OID X into OID Y?, but I don't particularly like it. We really don't want to do any *changes* to things until after we've determined the permissions allow for it, and I'm not sure how to get around that without building an independent introspection system for what might be. Perhaps we're comfortable enough saying we'll just rollback the command if it turns out to have not been allowed but I just don't like it. Feels like a higher risk solution to me. Yeah, that seems like a non-starter. I don't see a way to get around the second piece since what information is needed about the action is typically action-specific. Perhaps we could have an 'action-ID' (uh, we have an ID per command already, no? Probably doesn't fit the mold close enough though), and then a way to query information about what is this action trying to do?. Doesn't seem likely to be very clean though. Object creation seems to be one of the tougher cases here. When you're altering or dropping an existing object, the decision is likely to be based (in any system) on the properties of that object. When you're creating an object, the decision will of course have to be based on the properties of something else, but different access control models might not agree on the value of something else. It's not immediately clear to me how to deal with that, but again it's hard for me to discuss it in the abstract. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] thread safety on clients
I wrote: ... What I would have expected is crashes on the very similar updates to pgbench_branches, which is designed to be high-contention. It might be that there is some other effect going on here that explains why that wasn't happening. Need to go back and look more closely. ... and the answer to that is that pgbench_branches isn't subject to the bug, because its only pass-by-reference column happens to be filled with all NULLs by the initialization step, unlike the accounts filler column which happens to be filled with non-null strings. Null values mean no dangling pointers and no chance for a memory management issue. So you could have run this all day and never seen a crash on pgbench_branches updates. (If you manually set the filler column non-null before starting a run, the unpatched code crashes instantly, even with a non-bollixed pgbench.) So, nothing to see here except lack of test coverage, move along. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Tom Lane píše v pá 11. 12. 2009 v 15:11 -0500: Zdenek Kotala zdenek.kot...@sun.com writes: I thought about it. I think we can use GUC variable (e.g. dtraced_alloc) and hook switch pointers to dtraced AsetFunctions. The problem is how to distribute to all backend. You set the GUC in postgresql.conf. No big deal. If we go this route it would be nice to think about making a facility that has some usefulness for non-DTrace platforms too. Do you mean general facility for switching memory allocator? Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Fri, 2009-12-11 at 14:11 -0500, Stephen Frost wrote: All, * Robert Haas (robertmh...@gmail.com) wrote: If we design a security abstraction layer, the interfaces need to really be abstraction boundaries. Passing the table OID and then also the tablespace OID because PG DAC needs that to make its access control decision is crap. Now, to address the small useful bit of this mail- I tend to agree with this. I'm not convinced there's an alternative, but I'd like to throw out a couple of my ideas on how it could be addressed. I'd like to solicit for feedback on these. First off, we have alot of the information available from the catalog. Basically, given an OID, we should be able to find out the other information associated with that OID (such as what kind of object it is, what tablespace it resides in, etc). OID isn't sufficient, however, as we know from the dependency system. To address this, we would need OID and SubOID. Now, any information which we can derive from those should not be included in the API. To be honest, I don't think we've actually been all that bad about this, but I'll reserve any definitive answer until I've gone back through the API we have specifically thinking about this issue. On further thought, I'm probably wrong and should have caught this during my review. Sorry. Second, the information we *don't* have from above is generally information about what the requesting action is. For example, when changing ownership of an object, we can't possibly use introspection to find out the role which is on the receiving end, since at that time we've just learned it from the parser. Now, we might build an entire new object, pass the result of this action OID to the security system and ask it can we change OID X into OID Y?, but I don't particularly like it. We really don't want to do any *changes* to things until after we've determined the permissions allow for it, and I'm not sure how to get around that without building an independent introspection system for what might be. Perhaps we're comfortable enough saying we'll just rollback the command if it turns out to have not been allowed but I just don't like it. Feels like a higher risk solution to me. I don't see a way to get around the second piece since what information is needed about the action is typically action-specific. Perhaps we could have an 'action-ID' (uh, we have an ID per command already, no? Probably doesn't fit the mold close enough though), and then a way to query information about what is this action trying to do?. Doesn't seem likely to be very clean though. Other thoughts? In the Linux LSM case, we define a separate hook interface for each logical operation/action, where hook name identifies the action/operation and the set of arguments to that hook interface are the complete set of inputs that may be relevant to deciding whether to permit the operation/action. These arguments typically include pointer(s) to one or more object data structures as well as ancillary arguments (e.g. new owner value if chown). Reference: http://www.usenix.org/event/sec02/wright.html http://lxr.linux.no/#linux+v2.6.32/include/linux/security.h The XACE framework for the X server is described by: http://www.x.org/releases/X11R7.5/doc/security/XACE-Spec.html Unlike the LSM framework, it passes resource identifiers rather than direct object pointers. The FreeBSD MAC framework is described by: http://www.freebsd.org/doc/en/books/arch-handbook/mac.html It provides more abstraction than LSM does, at a cost in the overhead of the framework itself. -- Stephen Smalley National Security Agency -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] random() in multi-threaded pgbench
Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp writes: While testing the pgbench setshell command patch with -j option, I found all threads use the same sequence of random value. Were they actually threads, or were you testing the code while it had the broken configure script that didn't set ENABLE_THREAD_SAFETY? I think you might have hit the same thing I just ran into, that in a *non-threaded* build each subprocess will generate the same random sequence. At first, I think we need to call srandom() in each thread, Each sub-job I think. but the manual says we should use random_r() instead of random() on multi-threaded programs. http://www.kernel.org/doc/man-pages/online/pages/man3/random_r.3.html It only says that you need those if you want an *independent* random sequence for each thread. pgbench never had that before and I doubt we need it now. In any case, the same page also says these are a glibc-ism not a standard API, so we can't really rely on them. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
* Robert Haas (robertmh...@gmail.com) wrote: OK, it's clear that I've handled this badly. Sorry. My fear (however unjustified) was that someone would go and rewrite the patch based on an opinion that I express whether they agree with it or not. That's always going to be a risk in an open-discussion environment. Additionally, that's exactly what happened to me last go round- KaiGai rewrote the patch based on my ideas and suggestions, and the result was summarairly tossed out by Tom. Did it suck? Yes, heavily, and it frustrated me to the point that I specifically asked to not be the reviewer for SEPG during the next commitfest. At the same time, what KaiGai or others spend time on is up to them (and/or their employers). I sincerely hope that even if you suggest an approach down the road unrelated to this on some other patch you're reviewing, and then you see the results and say whoah, that's horrible, and should never be committed, that you understand none of us would want you to commit it. Sharing your ideas or putting out suggestions isn't a commitment on your part that you'll commit the results when someone else rights it. Heck, I bet you've been down that road on your own projects and come to the realization at the end of err, bad idea and not committed it. Allow me to say, my apologies, I feel like I may have over-reacted a bit for my part as well. So with that said, the idea I had was to try to pass around pre-existing data structures related to the objects on which access control decisions are being made, rather than Oids. That thought had crossed my mind as well, but I wasn't convinced that would actually be seen as a signifigantly different API to just having the arguments passed inline... Then again, using structures does allow you to add to them without having to modify the function definitions, and would allow David's suggestion of using function pointers to work, which we do in some other specific cases. I guess I'm curious if we (PG) have any particular feeling one way or the other about function pointers; I just don't recall seeing them used terribly often and would worry about that they might be passively discouraged? It does have a bit of a rock management feel to it and I really want to see if we can find a way to break that cycle. Agreed. It's been a point of frustration for me, but I've been trying to work with it so long as we at least get some constructive critisim back (Tom's review of the patch I reviewed fell into the questionable category for me on that call, which is what really frustrated me the most about it). A cyclic approach is typical in all software development, it's when information stops flowing about why something doesn't meet expectations or requirments that progress breaks down. Thanks! Stephen signature.asc Description: Digital signature
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Zdenek Kotala zdenek.kot...@sun.com writes: Tom Lane pÃÅ¡e v pá 11. 12. 2009 v 15:11 -0500: If we go this route it would be nice to think about making a facility that has some usefulness for non-DTrace platforms too. Do you mean general facility for switching memory allocator? No, I was thinking of some sort of memory allocation stats collection that doesn't depend on DTrace. It's amazing to me that we've never gone back and improved on the original quick-and-dirty MemoryContextStats mechanism. I certainly find myself using that a lot for issues like tracking down memory leaks. While palloc has a lot of advantages, the fact that you can't easily plug in a debug-friendly substitute malloc package is not one of them :-( regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] random() in multi-threaded pgbench
I wrote: Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp writes: http://www.kernel.org/doc/man-pages/online/pages/man3/random_r.3.html It only says that you need those if you want an *independent* random sequence for each thread. pgbench never had that before and I doubt we need it now. In any case, the same page also says these are a glibc-ism not a standard API, so we can't really rely on them. ... although if anyone was sufficiently excited about this, we could make use of erand48() without adding any new platform dependency, since GEQO is already relying on that. I think there's not much point though. We'd just be moving the nearest point of failure to the seeding algorithm --- careless seeding could still result in duplicate sequences for different threads. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
* Robert Haas (robertmh...@gmail.com) wrote: On Fri, Dec 11, 2009 at 2:11 PM, Stephen Frost sfr...@snowman.net wrote: Second, the information we *don't* have from above is generally information about what the requesting action is. For example, when changing ownership of an object, we can't possibly use introspection to find out the role which is on the receiving end, since at that time we've just learned it from the parser. I'm not sure that I follow what you're saying here. I think maybe it would help to discuss a concrete example, which is why I proposed a concept patch. Or perhaps you'd like just to pick out a specific case to discuss. Hrm, I thought I had given a specific example. Didn't do a good job of it, apparently. Let me try to be a bit more clear: ALTER TABLE x OWNER TO y; If given the table OID, there's a ton of information we can then pull about the table- the tablespace, the owner, the schema, the columns, the privileges, etc, etc. What we can't possibly figure out from the OID is the value of y. Yet, in the PG security model, the value of y matters! You have to know what y is to check if y has 'create' rights on the schema. If it doesn't (and the user executing the command isn't the superuser) then the request (under the PG model) is denied. Does that help clarify my example case? Object creation seems to be one of the tougher cases here. When you're altering or dropping an existing object, the decision is likely to be based (in any system) on the properties of that object. When you're creating an object, the decision will of course have to be based on the properties of something else, but different access control models might not agree on the value of something else. It's not immediately clear to me how to deal with that, but again it's hard for me to discuss it in the abstract. That sounds like a pretty good example to me too, to be honest. It goes back to the same issue though- that's information you get from the command that's trying to be run and isn't available from existing objects. Using structures to track this information, allowing the function arguments to remain identical, is certainly something which can be done, and probably done with a minimal amount of fuss. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] SE-PostgreSQL/Lite Review
On Thursday 10 December 2009 21:47:18 KaiGai Kohei wrote: Greg Smith wrote: It's funny; we started out this CommitFest with me scrambling to find someone, anyone, willing to review the latest SE-PostgreSQL patch, knowing it was a big job and few were likely to volunteer. Then schedules lined up just right, and last night I managed to get a great group of people all together to do perhaps the biggest single patch review ever, to work on just that.I gathered up a list of the biggest concerns about this feature and its associated implementation, we got a number of regular PostgreSQL hackers and two of the security guys you've seen on this list all in the same room, and we talked about little but SEPostgreSQL for hours. Minutes are at http://wiki.postgresql.org/wiki/SEPostgreSQL_Review_at_the_BWPUG and I'd suggest anyone interested in this feature (or in rejecting this feature) to take a look at what we covered. I repent that I cannot attend here. The wikipage is a good summarize. Thanks for your efforts. There's comments there, with references for you [citation needed] types, to help answer four of the most common complaints I've seen on this list about this feature: -Is there really a user base for it? -Has it been audited by security professionals? -Is there a useful SELinux policty to support it? -Will this work with other security frameworks? I feel pretty good now that these are not really our community's problems--these have had at least modest, and in some cases extensive, due diligence applied to them. And we've confirmed there's access to expertise from the security community to help out with remaining concerns there, in person even if we plan it out right. I personally suspect they've been discouraged from getting involved more by the slow pace this has been proceeding within our community and the general disarray around it, which would be understandable. IMO, the slow pace is not a primary reason. In fact, SELinux was released at 2000 Dec, but it gets integtated into the mainline kernel at 2003 Aug with various kind of improvements. It takes about 3 years from the first relase. IIRC, now we take 2.5 years from the first announce of SE-PgSQL in this list, and various kind of improvements and cleanups had been done. It is a bit long term, but not too long term. The reason of this gap is that people have individual consciousness about their security. I often represent it as security is a subjective term. Needless to say, we don't have magic-bullets for any threats. Any technology has its metirs and limitations. However, people tend to say it is nonsense, if the feature does not match with their recognizable demands or threats. Security-folks know MAC is not magic-bullets, while it is a significant piece of system security. But some of complaints might be pointless for security experts, so had been dismotivated. From the perspective of security folks, we have to introduce it doggedly. And, I'd like to understand database folks there are various kind of security models and viewpoints here. SELinux is one of them. The parts I do believe we have reason to be concerned are with the code integration into the PostgreSQL core, and no one has any easy answers to things like isn't this going to increase CERT advisories? and who's going to maintain this besides KaiGai? I personally feel that Steven Frost's recent comments here about how the PostgreSQL code makes this harder than it should be really cuts to the core of a next step here. The problem facing us isn't is SEPostgreSQL the right solution for providing external security checks?; it's how can the PostgreSQL code be improved so that integrating external security is easier? Looking at SEPostgreSQL is great because it really highlights where the existing set of places are. This general idea matches where thinking on things like row-level security was already going too--implement this for the database in general, then link SEPostgres in as just one provider of a security restriction. Right, it seems to me the security provider is a good phrase to represent this feature. It just provides additional access control decisions based on the different perspective of security model. Please note that the access control granularity is not an essential issue. We can also assume table-level mandatory access controls for instance. The latest patch provides table/column level controls without row-level, because the current PgSQL has facilities to know what tables and columns are referenced reasonably, so SE-PgSQL also can know what tables/columns are referenced without special tricks. Please remind the earlier SE-PgSQL in v8.2.x. It walked on the Query tree to pick up what columns were accessed. I hope the review from the BWPUG helps everyone out, and that the suggestions on the wiki for the Follow-up plan are helpful. As CF
Re: [HACKERS] Adding support for SE-Linux security
Stephen (great name!), * Stephen Smalley (s...@tycho.nsa.gov) wrote: Reference: http://www.usenix.org/event/sec02/wright.html http://lxr.linux.no/#linux+v2.6.32/include/linux/security.h The XACE framework for the X server is described by: http://www.x.org/releases/X11R7.5/doc/security/XACE-Spec.html The FreeBSD MAC framework is described by: http://www.freebsd.org/doc/en/books/arch-handbook/mac.html Thanks for these pointers! I'm sure KaiGai has probably already done a review of these, but I'll take a look and would ask others who are interested to please do the same. Seeing the different options that people have gone with (as opposed to just the SELinux/kernel version) will undoubtably be helpful in deciding the best approach forward. Thanks again! Stephen signature.asc Description: Digital signature
Re: [HACKERS] Adding support for SE-Linux security
On Fri, Dec 11, 2009 at 3:28 PM, Stephen Frost sfr...@snowman.net wrote: I sincerely hope that even if you suggest an approach down the road unrelated to this on some other patch you're reviewing, and then you see the results and say whoah, that's horrible, and should never be committed, that you understand none of us would want you to commit it. I have to thank you for saying this - unfortunately, I don't think everyone takes this approach. As you can probably figure out, my alleged rock management upthread was actually a poorly executed attempt to avoid being accused of bait-and-switch. If I don't tell you how to write the patch, you can't accuse me of moving the goalposts (of course I've now discovered the pitfalls of that approach as well...). Sharing your ideas or putting out suggestions isn't a commitment on your part that you'll commit the results when someone else rights it. Heck, I bet you've been down that road on your own projects and come to the realization at the end of err, bad idea and not committed it. Well, I haven't been a committer long enough to have gone through that precise process, but sure, I've tossed out ideas when they don't turn out to be good. So with that said, the idea I had was to try to pass around pre-existing data structures related to the objects on which access control decisions are being made, rather than Oids. That thought had crossed my mind as well, but I wasn't convinced that would actually be seen as a signifigantly different API to just having the arguments passed inline... If you'll forgive me for saying so, this is exactly the sort of thinking that I think is killing us. Who cares how it will be seen? Seen by whom? We shouldn't be writing this code to be seen - we should be writing it to be good. If doing this makes a clean, tight abstraction layer, then it's a good design. If it doesn't, then it sucks. I realize that opinions enter into this at some level, but let's try to proceed as though there's a technically right answer out there and bend our best efforts to finding it. Then again, using structures does allow you to add to them without having to modify the function definitions, Personal opinion time, but I think that without having to modify the function definitions is a CRITICAL design component. Furthermore, adding to them [the structures] shouldn't be something that only happens when we decide to support a new security model, or we're little better off than if we just passed a kajillion arguments. Again, I don't want to overstate my confidence in this point, but it feels to me like we need to pass PRE-EXISTING data structures that are already being used for other things and happen to be the right stuff to enable access control decisions, and to which fields that are likely to be needed are likely to get added anyway. But it's difficult to know whether this approach can be made to work without trying it, and there are bound to be problem cases that need to be thought about, and that thinking will be more likely to lead to a good result if it happens in the community, rather than by KaiGai or any other single person in isolation. and would allow David's suggestion of using function pointers to work, which we do in some other specific cases. I guess I'm curious if we (PG) have any particular feeling one way or the other about function pointers; I just don't recall seeing them used terribly often and would worry about that they might be passively discouraged? I'm going to vote fairly strongly against inserting function pointers at the outset. I think that we should look at the first phase of this project as an attempt to restructure the code so that the access control decisions are isolated from the rest of the code. *If* we can do that successfully, I think it will represent good progress all on its own. Inserting function pointers is something that can be done later if it turns out to be useful with a very small, self-contained patch. It does have a bit of a rock management feel to it and I really want to see if we can find a way to break that cycle. Agreed. It's been a point of frustration for me, but I've been trying to work with it so long as we at least get some constructive critisim back (Tom's review of the patch I reviewed fell into the questionable category for me on that call, which is what really frustrated me the most about it). A cyclic approach is typical in all software development, it's when information stops flowing about why something doesn't meet expectations or requirments that progress breaks down. Part of my frustration here is that I don't see a lot of evidence that anyone is willing to put really tangible resources into this other than KaiGai and his employer. I don't want to keep reviewing the same patches over again just because they keep getting resubmitted with various modifications. I am willing to try to be helpful with this project if that is something that you
Re: [HACKERS] Adding support for SE-Linux security
On Fri, Dec 11, 2009 at 4:26 PM, Stephen Frost sfr...@snowman.net wrote: Hrm, I thought I had given a specific example. Didn't do a good job of it, apparently. Let me try to be a bit more clear: ALTER TABLE x OWNER TO y; If given the table OID, there's a ton of information we can then pull about the table- the tablespace, the owner, the schema, the columns, the privileges, etc, etc. What we can't possibly figure out from the OID is the value of y. Yet, in the PG security model, the value of y matters! You have to know what y is to check if y has 'create' rights on the schema. If it doesn't (and the user executing the command isn't the superuser) then the request (under the PG model) is denied. Does that help clarify my example case? That case doesn't seem terribly problematic to me. It seems clear that we'll want to pass some information about both x and y. What is less clear is exactly what the argument types will be, and the right answer probably depends somewhat on the structure of the existing code, which I have not looked at. What I'm more concerned about is if the access control decision in this case were based on u for PG DAC, v for SE-PostgreSQL, and w for Robert Haas's Personal Defensive System. If that's the case, and our function signature looks like (x,y,u,v,w), the we should worry. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
* Robert Haas (robertmh...@gmail.com) wrote: If I don't tell you how to write the patch, you can't accuse me of moving the goalposts (of course I've now discovered the pitfalls of that approach as well...). Indeed, we also yell and scream when we don't know which direction the goalposts are in. ;) That thought had crossed my mind as well, but I wasn't convinced that would actually be seen as a signifigantly different API to just having the arguments passed inline... If you'll forgive me for saying so, this is exactly the sort of thinking that I think is killing us. Who cares how it will be seen? erp. mea culpa. I meant that I didn't think of it as being enough of a novelty for you to be suggesting-it-but-not-telling-us.. It just kind of came off as too-obvious, however, the reality is that I didn't understand your suggestion, see below. Then again, using structures does allow you to add to them without having to modify the function definitions, Personal opinion time, but I think that without having to modify the function definitions is a CRITICAL design component. Furthermore, adding to them [the structures] shouldn't be something that only happens when we decide to support a new security model, or we're little better off than if we just passed a kajillion arguments. Again, I don't want to overstate my confidence in this point, but it feels to me like we need to pass PRE-EXISTING data structures that are already being used for other things and happen to be the right stuff to enable access control decisions, and to which fields that are likely to be needed are likely to get added anyway. Ah, now that makes alot of sense.. Unfortunately, having been involved in at least some of this code in the past, sadly I don't think we have such pre-existing data structures for the information that's primairly at issue here. Specifically, the data structures we tend to have pre-existing are the ones that come from the catalog and would fall out from an OID+SubOID based API. Sure, we could pass those structures in instead of using SysCache to pull the data out, but that's not really how we 'typically' do things in the code base from my experience, and I don't really see it having alot of advantage. Using OID+SubOID and SysCache *will* pick up new things as they're added to the catalogs- for the information *in* the catalogs. There are very few data structures outside of the parse tree which include the information from the parse tree.. And to be honest, the ones that *are* out there don't typically have the world's best structure for use by this kind of a framework (thinking back to specifically what's passed around for column-level privs..). But it's difficult to know whether this approach can be made to work without trying it, and there are bound to be problem cases that need to be thought about, and that thinking will be more likely to lead to a good result if it happens in the community, rather than by KaiGai or any other single person in isolation. I agree with this- one issue is, unfortunately, an overabundance from KaiGai of code-writing man-power. This is an odd situation for this community, in general, so we're having a hard time coming to grasp with it. KaiGai, can you hold off on implementing any of these approaches until we can hammer down something a bit better for you to work from as a baseline? I'll start with Robert's suggestion of a single-object example case and throw out some example code for people to shoot down of different approachs. After a few iterations of that, with comments from all (KaiGai, you're welcome to comment on it too, of course), we'll turn you loose on it again to implement fully (if you're still willing to). Following along that though, as we'd like this to be the design forum, when you come across problems or issues implementing it, could you come back to this forum and explain the issue and why it doesn't fit, as soon as you hit it? What you tend to do is disappear for a while, then come back, instead, with a full patch which works, but has places where you've gone down an unexpected path because you found a case which wasn't covered, designed a solution for it which kinda fits and then implemented it immediately. I feel that's something I've encouraged you to do, unfortunately, and my apologies for that. I'm trying to get time from my employer (and my wife) to dedicate to this, to hopefully avoid that happening in the future. I'm going to vote fairly strongly against inserting function pointers at the outset. I think that we should look at the first phase of this project as an attempt to restructure the code so that the access control decisions are isolated from the rest of the code. *If* we can do that successfully, I think it will represent good progress all on its own. Inserting function pointers is something that can be done later if it turns out to be useful with a very small, self-contained patch.
Re: [HACKERS] Adding support for SE-Linux security
* Robert Haas (robertmh...@gmail.com) wrote: On Fri, Dec 11, 2009 at 4:26 PM, Stephen Frost sfr...@snowman.net wrote: Does that help clarify my example case? That case doesn't seem terribly problematic to me. It seems clear that we'll want to pass some information about both x and y. What is less clear is exactly what the argument types will be, and the right answer probably depends somewhat on the structure of the existing code, which I have not looked at. Allow me to assist- y is never in a structure once you're out of the parser: ATExecChangeOwner(Oid relationOid, Oid newOwnerId, bool recursing) I expect you'll find this is more the rule than the exception to alot of the existing PG security model, because much of it's responsibility is in what I'll call the DDL (under commands/) aspects. The DML pieces (under the executor) are a bit better about this, specifically you could pass in a RangeTblEntry, exactly how ExecCheckRTEPerms handles it. Actually, in a fit of barely-contained mirth, it strikes me that PG really has already done what you're suggesting for the 'hard' part- and the RangeTblEntry plus ExecCheckRTEPerms is exactly it. It's all the *other* checks we do for the PG security model, under commands/, that are the problem here. The parts of the code that, to be honest, I think all us database geeks have historically cared alot less about. What I'm more concerned about is if the access control decision in this case were based on u for PG DAC, v for SE-PostgreSQL, and w for Robert Haas's Personal Defensive System. If that's the case, and our function signature looks like (x,y,u,v,w), the we should worry. Right, I understand that. What I offer in reply is that we focus our attention on using the OID+SubOID approach, as I'm suggesting, to the fullest extent possible through that mechanism, and appreciate that the other arguments passed to the function are derived specifically from the parser and therefore unlikely to be changed until and unless we change the base syntax of the command and calling function, at which time we may have to add arguments to the function signature. This would continue at least until we get to the point where we decide that the caller needs to be changed because it's got a huge function sig, and move it to something like the structure of the executor and the equivilant of ExecCheckRTEPerms() would get updated along with it, at that time. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Adding support for SE-Linux security
Stephen Frost wrote: I agree with this- one issue is, unfortunately, an overabundance from KaiGai of code-writing man-power. This is an odd situation for this community, in general, so we're having a hard time coming to grasp with it. There are plenty of parallels to when Zdenek was writing a ton of in-place upgrade code faster than anyone else was fully consuming it. The do it right or don't do it at all approach of the PG community seems particularly hard to reconcile with larger patches from people we don't get enough face time with. It's easy to get deadlocked and not have a good way to navigate out when faced with a set of difficult decisions and only electronic communications between participants. Shoot, you and Robert have spent time doing technical arguments in person and we still got a little rough patch on-list this week out of the debate. I hate to even use this terminology, but these big patches seem to need a project manager advocate sometimes: someone who knows everyone well enough to clear these stalled spots, smooth over any personality conflicts, and is motivated to intervene because they need the feature. I see it as a sort of scaling problem that might be a recurring one. It would be nice if we could all get better at identifying when it does happen, and perhaps find someone to help with planning before we waste so much time chasing code that isn't going to be accepted yet again. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Tom Lane wrote: It's amazing to me that we've never gone back and improved on the original quick-and-dirty MemoryContextStats mechanism. I certainly find myself using that a lot for issues like tracking down memory leaks. That code hasn't really gone anywhere since Neil tweaked the indentation two years ago. What sorts of improvements were you looking for? I started on trying to improve this area at one point but didn't get far. My first step was going to be just wrapping the call into a UDF to make it easier to reach without having to set loose gdb. I thought that might expand the possible uses for MemoryContextStats to help find a memory leak safely on the production box, and therefore attact more attention to improving it. People really don't like running gdb on production, but a UDF dumping the same data wouldn't seem so risky. As Daniel Farina pointed out to me one day in a duh moment, that idea is quite obviously doomed by the fact that people want dumps from processes that you won't be talking to in a UDF context. You won't be able to find a leak in the background writer by dumping the context the UDF can see. There would need to be some sort of IPC/signaling mechanism in each process if you wanted it to work everywhere, and once that realization hit I realized this was a deeper problem than it appeared. If you want to make it easier to export into user space, it seems to me the memory context stats data either needs to get pushed somewhere it's easier to get to (the way the regular stats are), or there needs to be some way of polling any random PG process to find it--presumably by passing the request the pid you want to instrument. Neither of which may really be a reasonable idea at all, particularly since that in most cases, you're using a debugger to track your leak down only after reproducing a test case for it. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
I just did a round of integrating some of the big-picture feedback that has shown up here since the meeting into http://wiki.postgresql.org/wiki/SEPostgreSQL_Review_at_the_BWPUG , mainly supplementing the references in the Works outside of SELinux section with the new suggested reading here suggested by Stephen Smalley and Joshua Brindle. I'm trying to keep that a fairly readable intro to the controversial parts rather than going deeply technical. What I'm not going to try to track is all the low-level implementation details that are bouncing around right now, my brain is too full this week to cram more about OID trivia into it right now. That would be a good idea for someone to summarize eventually and then throw that onto the wiki somewhere else, so that it's easier to remember the context of what/why decisions were made. The way Simon has been keeping an ongoing log at http://wiki.postgresql.org/wiki/Hot_Standby shows a reasonable way to organize such a thing from a similarly complicated patch. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Fri, Dec 11, 2009 at 5:36 PM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: On Fri, Dec 11, 2009 at 4:26 PM, Stephen Frost sfr...@snowman.net wrote: Does that help clarify my example case? That case doesn't seem terribly problematic to me. It seems clear that we'll want to pass some information about both x and y. What is less clear is exactly what the argument types will be, and the right answer probably depends somewhat on the structure of the existing code, which I have not looked at. Allow me to assist- y is never in a structure once you're out of the parser: Well this is why you're writing the patch and not me. :-) ATExecChangeOwner(Oid relationOid, Oid newOwnerId, bool recursing) I expect you'll find this is more the rule than the exception to alot of the existing PG security model, because much of it's responsibility is in what I'll call the DDL (under commands/) aspects. The DML pieces (under the executor) are a bit better about this, specifically you could pass in a RangeTblEntry, exactly how ExecCheckRTEPerms handles it. Actually, in a fit of barely-contained mirth, it strikes me that PG really has already done what you're suggesting for the 'hard' part- and the RangeTblEntry plus ExecCheckRTEPerms is exactly it. It's all the *other* checks we do for the PG security model, under commands/, that are the problem here. The parts of the code that, to be honest, I think all us database geeks have historically cared alot less about. Hmm, interesting. What I'm more concerned about is if the access control decision in this case were based on u for PG DAC, v for SE-PostgreSQL, and w for Robert Haas's Personal Defensive System. If that's the case, and our function signature looks like (x,y,u,v,w), the we should worry. Right, I understand that. What I offer in reply is that we focus our attention on using the OID+SubOID approach, as I'm suggesting, to the fullest extent possible through that mechanism, and appreciate that the other arguments passed to the function are derived specifically from the parser and therefore unlikely to be changed until and unless we change the base syntax of the command and calling function, at which time we may have to add arguments to the function signature. This would continue at least until we get to the point where we decide that the caller needs to be changed because it's got a huge function sig, and move it to something like the structure of the executor and the equivilant of ExecCheckRTEPerms() would get updated along with it, at that time. What exactly do you mean by a SubOID? I'm not really following that part. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL/Lite Review
Stephen Frost wrote: In my cosmetic preference, ace_ is better than ac_. The 'e' means extendable, and ace feels like something cool. :-) No complaints here.. I just hope this doesn't end up being *exactly* the same as your original PGACE patches.. I'd feel terrible if we weren't able to at least improve something with this extremely long and drawn our process! Please forget old PGACE. We're talking about our future. :-) And I'd like to store these files in backend/security/ace/*.c, because backend/security will also store other security modules! This is perfectly reasonable in my view. No complaints here. src/ + backend/ + security/ + ace/ ... access control framework + sepgsql/ ... selinux specific code + smack/ ... (upcoming?) smack specific code + solaristx/ ... (upcoming?) solaris-tx specific code Looks good to me. Perhaps we'll have a smack/ patch showing up very shortly as well.. It's a welcome feature. The reason why I prefer the default PG check + one enhanced security at most is simplification of the security label management. If two label-based enhanced securities are enabled at same time, we need two field on pg_class and others to store security context. Ah, yes, I see your point must more clearly now. This sounds reasonable to me. In addition, OS allows to choose one enhanced security at most eventually. In my image, the hook should be as: Value * ac_database_create([arguments ...]) { /* * The default PG checks here. * It is never disabled */ /* enhanced security as follows */ #if defined(HAVE_SELINUX) if (sepgsql_is_enabled()) return sepgsql_database_create(...); #elif defined(HAVE_SMACK) if (smack_is_enabled()) return smack_database_create(...); #elif defined(HAVE_SOLARISTX) if (soltx_is_enabled()) return soltx_database_create(...); #endif return NULL; } We can share a common image image? If all of these security modules make sense as only-more-restrictive, then I have no problem with this approach. Honestly, I'm fine with the initial hooks looking as above in any case, since it provides a clear way to deal with switching out the 'default PG checks' if someone desires it- you could #if around that block as well. As it's the principle/primary, and I could see only-more-restrictive being more popular, I don't mind having that code in-line in the hook. The check itself is still being brought out and into the security/ framework. Ahh, indeed, #elif does not allow a single binary to provide multiple options. You are right. I'd like to rewrite it as follows: Value * ac_database_create([arguments ...]) { Value *retval = NULL; /* * The default PG checks here. * It is never disabled */ switch (ace_feature) { #ifdef HAVE_SELINUX case ACE_SELINUX_SUPPORT: if (sepgsql_is_enabled()) retval = sepgsql_database_create(...); break; #endif #ifdef HAVE_SMACK case ACE_SMACK_SUPPORT: if (smack_is_enabled()) retval = smack_database_create(...); break; #endif #ifdef HAVE_SOLARISTX case ACE_SOLARISTX_SUPPORT: if (soltx_is_enabled()) retval = soltx_database_create(...); break; #endif default: break; } return retval; } I do think that, technically, there's no reason we couldn't allow for multiple only-more-restrictive models to be enabled and built in a single binary for systems which support it. As such, I would make those just #if defined() rather than #elif. Let it be decided at runtime which are actually used, otherwise it becomes a much bigger problem for packagers too. The reason why I don't want to support multiple enhanced securities at same time is complexity of security label management, not access control model. Unlike X-server, RDBMS have to manage the security label of database object persistently on the disk. It naturally needs enhancement of data structure, such as pg_class.relsecon field in the latest SE-PgSQL/Lite. If the third enhanced security does not use security label, basically, I think it is feasible. In fact, the default PG check is a security provider without security label, and it can coexist with SELinux. Thanks, -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL/Lite Review
Stephen Frost wrote: Josh, * Joshua Brindle (met...@manicmethod.com) wrote: Stephen Frost wrote: I do think that, technically, there's no reason we couldn't allow for multiple only-more-restrictive models to be enabled and built in a single binary for systems which support it. As such, I would make those just #if defined() rather than #elif. Let it be decided at runtime which are actually used, otherwise it becomes a much bigger problem for packagers too. It isn't just a case of using #if and it magically working. You'd need a system to manage multiple labels on each object that can be addressed by different systems. So instead of having an object mapped only to system_u:object_r:mydb_t:s15 you'd also have to have it mapped to, eg., ^ for Smack. I'm not sure I see that being a problem.. We're going to have references in our object managers which make sense to us (eg: table OIDs) and then a way of mapping those to some label (or possibly a set of labels, as you describe). We might want to reconsider the catalog structure a bit if we want to support more than one at a time, but I don't see it as a huge problem to support more than one label existing for a given object. If we allow multiple security labels on a database object, we have to expand the structure of system catalog whenever a new security feature will come in. I think it against to the purpose of the framework. Even if we store them external relations to reference the object by OID, we have to provide multiple interface to import/export a security label for each enhanced securities. For example, it requires much complex patch to the pg_dump. My preference is all the enhanced securities shares a common facility to manage security label, a common statement support and a common backup/restore support. Thanks, -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Robert Haas robertmh...@gmail.com writes: What exactly do you mean by a SubOID? I'm not really following that part. I assume he's talking about the object reference representation used in pg_depend, which is actually class OID + object OID + sub-object ID. The only object type that has sub-objects at the moment is tables, wherein the sub-objects are columns and the sub-object IDs are column numbers. The sub-object ID is zero for all other cases. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Robert Haas wrote: On Fri, Dec 11, 2009 at 4:26 PM, Stephen Frost sfr...@snowman.net wrote: Hrm, I thought I had given a specific example. Didn't do a good job of it, apparently. Let me try to be a bit more clear: ALTER TABLE x OWNER TO y; If given the table OID, there's a ton of information we can then pull about the table- the tablespace, the owner, the schema, the columns, the privileges, etc, etc. What we can't possibly figure out from the OID is the value of y. Yet, in the PG security model, the value of y matters! You have to know what y is to check if y has 'create' rights on the schema. If it doesn't (and the user executing the command isn't the superuser) then the request (under the PG model) is denied. Does that help clarify my example case? That case doesn't seem terribly problematic to me. It seems clear that we'll want to pass some information about both x and y. What is less clear is exactly what the argument types will be, and the right answer probably depends somewhat on the structure of the existing code, which I have not looked at. What I'm more concerned about is if the access control decision in this case were based on u for PG DAC, v for SE-PostgreSQL, and w for Robert Haas's Personal Defensive System. If that's the case, and our function signature looks like (x,y,u,v,w), the we should worry. Theoretically, (x,y,u) - (x,y,u,v) - (x,y,u,v,w) may happen. However, if the new suggested security model is too far from the known security model, we should consider whether it performs correctly at first before commit it. As long as we describe a security model corresponding to database objects, it is hard to consider a possibility of infinite increasing in the future. Thanks, -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] status update on Hot Standby and Streaming Replication
I'm not quite sure where we stand with these two patches. My impression is that many of the outstanding TODO items in Hot Standby have been fixed, and I'm not sure what remains. Streaming Replication I think reviewing is not as far advanced, but I'm not sure. Any chance that Hot Standby can get committed in the next few days before we bundle alpha3? Otherwise, I think it will not get as much test coverage as it really deserves. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] LDAP where DN does not include UID attribute
On Sun, Dec 6, 2009 at 11:29 PM, Greg Smith g...@2ndquadrant.com wrote: Magnus Hagander wrote: On Sun, Nov 29, 2009 at 13:05, Magnus Hagander mag...@hagander.net wrote: I'll be happy to work on this to get it ready for commit, or do you want to do the updates? Here's a patch with my work so far. Major points missing is the sanitizing of the username and the docs. It looks like you did an initial review here already. Since we haven't heard from Robert in a while and you seem interested in the patch, I just updated this one to list you as the committer and marked it ready for committer. You can commit it or bounce it from there, but it's obvious none of our other reviewers are going to be able to do anything with it. I think we should mark this returned with feedback, since it sounds like there are still open items. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: Unlike Tom (I think), I do believe that there is demand (possibly only from a limited number of people, but demand all the same) for this feature. Please note that I do not think there is *zero* demand for the feature. There is obviously some. What I find highly dubious is whether there is enough demand to justify the amount of effort, both short- and long-term, that the community would have to put into it. Well, the bottom line is that this effort should grow the development and user community of Postgres --- it if doesn't, it is a failure. And I also believe that most people in our community are generally supportive of the idea, but only a minority are willing to put in time to make it happen. So I have no problem saying to the people who want the feature - none of our committers feel like working on this. Sorry. On the other hand, I also have no problem telling them - good news, Bruce Momjian thinks this is a great feature and wants to help you get it done. I *do* have a problem with saying - we don't really know whether anyone will ever want to work on this with you or not. If I thought that Bruce could go off in a corner and make this happen and it would create no demands on anybody but him and KaiGai-san, I would say fine, if that's where you want to spend your time, go for it. But even to state that implied claim is to see how false it is. Bruce is pointing to the Windows port, but he didn't make it happen by himself, or any close approximation of that. Everybody who works on this project has been affected by that, and we're *still* putting significant amounts of time into Windows compatibility, over five years later. The Windows port was primiarly done by Magnus, Claudio Natoli, and Andrew Dunstan. The good thing about that group is that their involvement in Win32 did not take them away from existing Postgres work --- in fact I think it increased Magnus's and Andrew's involvement. As I stated above, I expect the SE-Postgres work to be done mostly by new people and to expand our development team. KaiGai is certainly a new addition, and I think there is already an indication that new people are getting involved. Of course, our existing people will have to help too, but as I stated a few days ago, I expect security-specific stuff to be maintained mostly by new people, and our existing folks are going to have to help with hooks, plus adding things like mandatory access control and row-level security to base Postgres. (I do think it is inevitable that those will be added some day. I agree the security folks will be accelerating that. Hopefully we will get more good out of this than the inconvenience of this accelerated security stuff.) My guess is that a credible SEPostgres offering will require a long-term amount of work at least equal to, and very possibly a good deal more than, what it took to make a native Windows port. If SEPostgres could bring us even 10% as many new users as the Windows port did, it'd probably be a worthwhile use of our resources. But again, that's an assumption that's difficult to type without bursting into laughter. Odds are SEPostgres will add perhaps 1% new users compared to Win32, but perhaps very important, energetic, and visible users. As stated earlier, the base Postgres security additions like row-level security are going to be inconvenient, but I do think we will eventually need them anyway, so I don't see them as SE-Postgres burdens. I am not replying to many of these emails so I don't appear to be brow-beating (forcing) the community into accepting this features. I might be brow-beating the community, but I don't want to _appear_ to be brow-beating. ;-) -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
On Fri, Dec 11, 2009 at 3:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: Zdenek Kotala zdenek.kot...@sun.com writes: Tom Lane píše v pá 11. 12. 2009 v 15:11 -0500: If we go this route it would be nice to think about making a facility that has some usefulness for non-DTrace platforms too. Do you mean general facility for switching memory allocator? No, I was thinking of some sort of memory allocation stats collection that doesn't depend on DTrace. It's amazing to me that we've never gone back and improved on the original quick-and-dirty MemoryContextStats mechanism. I certainly find myself using that a lot for issues like tracking down memory leaks. While palloc has a lot of advantages, the fact that you can't easily plug in a debug-friendly substitute malloc package is not one of them :-( This might be nice to have, but I don't think it's necessarily a prerequisite for a committable patch. For that, I think we just need something that uses the dispatch mechanism to avoid the overhead in the normal case. However, as this is a substantial redesign and there are 4 days left in the CommitFest, I am going to mark this patch Returned with Feedback for now. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Fri, Dec 11, 2009 at 8:41 PM, Bruce Momjian br...@momjian.us wrote: I am not replying to many of these emails so I don't appear to be brow-beating (forcing) the community into accepting this features. I might be brow-beating the community, but I don't want to _appear_ to be brow-beating. ;-) LOL. At least you're honest... ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Ron Mayer wrote: Bruce Momjian wrote: Well, the bottom line is that this effort should grow the development and user community of Postgres --- it if doesn't, it is a failure. Really? Even if it only allows existing Postgres users and companies to expand their use into higher security applications IMHO it's a success. If a main goal were increasing users, implementing MySQL-isms and MSFTSqlServer-isms would seem the biggest bang for the buck. Well, it should expand the user-base because right now we don't have many current users who are looking for these features --- if we did, it would be an easy decision to add these features. Second, it should add new developers because we have limited developer resources, and I am not sure adding additional security is at the top of our priority list, and we don't have many security-expert developers so by definition we need more to implement this reliably. These might be unpleasant opinions, but I believe they are accurate. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Bruce Momjian wrote: Well, the bottom line is that this effort should grow the development and user community of Postgres --- it if doesn't, it is a failure. Really? Even if it only allows existing Postgres users and companies to expand their use into higher security applications IMHO it's a success. If a main goal were increasing users, implementing MySQL-isms and MSFTSqlServer-isms would seem the biggest bang for the buck. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Need a mentor, and a project.
I am thinking about starting with the following TODO item: -- Have EXPLAIN ANALYZE issue NOTICE messages when the estimated and actual row counts differ by a specified percentage. I picked this because it is somewhat related to query processing which is what I am most interested in. It also seems like a good start up project for a newbie like me. Before I start looking into what this would involve and start a conversation on designing a solution - I wanted to know what you guys think about this particular TODO, and it suitability to a newbie. Looking forward to your comments... Thanks Ashish On Mon, 7 Dec 2009, Josh Berkus wrote: On 12/7/09 4:41 PM, Ashish wrote: Albe Joshua, thanks for the advice. I am in the process of deciding what to work on and am looking at the TODO list. I definitely do not intend to work in a vacuum :-) I am really excited about this and look forward to being challenged and learning a lot. When you decide what you want to work on, let us know and we'll try to find you an appropriate mentor. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Need a mentor, and a project.
Ashish wrote: I am thinking about starting with the following TODO item: -- Have EXPLAIN ANALYZE issue NOTICE messages when the estimated and actual row counts differ by a specified percentage. I picked this because it is somewhat related to query processing which is what I am most interested in. It also seems like a good start up project for a newbie like me. Before I start looking into what this would involve and start a conversation on designing a solution - I wanted to know what you guys think about this particular TODO, and it suitability to a newbie. Looking forward to your comments... I even have a sample patch you can use as a start, attached. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + Index: doc/src/sgml/ref/explain.sgml === RCS file: /cvsroot/pgsql/doc/src/sgml/ref/explain.sgml,v retrieving revision 1.38 diff -c -c -r1.38 explain.sgml *** doc/src/sgml/ref/explain.sgml 18 Sep 2006 19:54:01 - 1.38 --- doc/src/sgml/ref/explain.sgml 22 Dec 2006 17:09:05 - *** *** 64,72 para The literalANALYZE/literal option causes the statement to be actually executed, not only planned. The total elapsed time expended within each plan node (in !milliseconds) and total number of rows it actually returned are added to !the display. This is useful for seeing whether the planner's estimates !are close to reality. /para important --- 64,72 para The literalANALYZE/literal option causes the statement to be actually executed, not only planned. The total elapsed time expended within each plan node (in !milliseconds) and total number of rows it actually returned and variance are added to !the display. A sign of the variance indicates whether the estimate was too high or too low. !This is useful for seeing how close the planner's estimates are to reality. /para important *** *** 222,229 QUERY PLAN - ! HashAggregate (cost=39.53..39.53 rows=1 width=8) (actual time=0.661..0.672 rows=7 loops=1) !-gt; Index Scan using test_pkey on test (cost=0.00..32.97 rows=1311 width=8) (actual time=0.050..0.395 rows=99 loops=1) Index Cond: ((id gt; $1) AND (id lt; $2)) Total runtime: 0.851 ms (4 rows) --- 222,229 QUERY PLAN - ! HashAggregate (cost=39.53..39.53 rows=1 width=8) (actual time=0.661..0.672 rows=7 var=-6.00 loops=1) !-gt; Index Scan using test_pkey on test (cost=0.00..32.97 rows=1311 width=8) (actual time=0.050..0.395 rows=99 var=+12.24 loops=1) Index Cond: ((id gt; $1) AND (id lt; $2)) Total runtime: 0.851 ms (4 rows) Index: src/backend/commands/explain.c === RCS file: /cvsroot/pgsql/src/backend/commands/explain.c,v retrieving revision 1.152 diff -c -c -r1.152 explain.c *** src/backend/commands/explain.c 4 Oct 2006 00:29:51 - 1.152 --- src/backend/commands/explain.c 22 Dec 2006 17:09:09 - *** *** 57,62 --- 57,63 static void show_sort_keys(Plan *sortplan, int nkeys, AttrNumber *keycols, const char *qlabel, StringInfo str, int indent, ExplainState *es); + static double ExplainVariance(double estimate, double actual); /* * ExplainQuery - *** *** 704,713 { double nloops = planstate-instrument-nloops; ! appendStringInfo(str, (actual time=%.3f..%.3f rows=%.0f loops=%.0f), 1000.0 * planstate-instrument-startup / nloops, 1000.0 * planstate-instrument-total / nloops, planstate-instrument-ntuples / nloops, planstate-instrument-nloops); } else if (es-printAnalyze) --- 705,716 { double nloops = planstate-instrument-nloops; ! appendStringInfo(str, (actual time=%.3f..%.3f rows=%.0f var=%+.2f loops=%.0f), 1000.0 * planstate-instrument-startup / nloops, 1000.0 * planstate-instrument-total / nloops, planstate-instrument-ntuples / nloops, + ExplainVariance(plan-plan_rows, + planstate-instrument-ntuples / nloops), planstate-instrument-nloops); } else if (es-printAnalyze) *** *** 1205,1207 --- 1208,1225 appendStringInfo(str, \n); } + + + static double ExplainVariance(double estimate, double
Re: [HACKERS] EXPLAIN BUFFERS
On Fri, Dec 11, 2009 at 11:36 AM, Euler Taveira de Oliveira eu...@timbira.com wrote: Robert Haas escreveu: On Thu, Dec 10, 2009 at 9:35 PM, Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp wrote: Anyway, a revised patch according to the comments is attached. The new text format is: Buffers: shared hit=675 read=968, temp read=1443 written=1443 * Zero values are omitted. (Non-text formats could have zero values.) * Rename Blocks: to Buffers:. * Remove parentheses and add a comma between shared, local and temp. I did a bit of copy-editing of your doc changes to make the English a bit more correct and idiomatic. Slightly revised patch attached for your consideration. The output format looks really nice (thanks for bearing with me), and the functionality is great. Please, document that zero values are omitted in the text format. It seems intuitive but could be surprise because zero values are in non-text formats. OK, done, see attached. I also noticed when looking through this that the documentation says that auto_explain.log_buffers is ignored unless auto_explain.log_analyze is set. That is true and seems right to me, but for some reason explain_ExecutorEnd() had been changed to set es.analyze if either log_analyze or log_buffers was set. It actually didn't have any effect unless log_analyze was set, but only because explain_ExecutorStart doesn't set queryDesc-doInstrument in that case. So I've reverted that here for clarity. ...Robert diff --git a/contrib/auto_explain/auto_explain.c b/contrib/auto_explain/auto_explain.c index f0d907d..491f479 100644 --- a/contrib/auto_explain/auto_explain.c +++ b/contrib/auto_explain/auto_explain.c @@ -22,6 +22,7 @@ PG_MODULE_MAGIC; static int auto_explain_log_min_duration = -1; /* msec or -1 */ static bool auto_explain_log_analyze = false; static bool auto_explain_log_verbose = false; +static bool auto_explain_log_buffers = false; static int auto_explain_log_format = EXPLAIN_FORMAT_TEXT; static bool auto_explain_log_nested_statements = false; @@ -93,6 +94,16 @@ _PG_init(void) NULL, NULL); + DefineCustomBoolVariable(auto_explain.log_buffers, + Log buffers usage., + NULL, + auto_explain_log_buffers, + false, + PGC_SUSET, + 0, + NULL, + NULL); + DefineCustomEnumVariable(auto_explain.log_format, EXPLAIN format to be used for plan logging., NULL, @@ -221,6 +232,7 @@ explain_ExecutorEnd(QueryDesc *queryDesc) ExplainInitState(es); es.analyze = (queryDesc-doInstrument auto_explain_log_analyze); es.verbose = auto_explain_log_verbose; + es.buffers = (es.analyze auto_explain_log_buffers); es.format = auto_explain_log_format; ExplainBeginOutput(es); diff --git a/doc/src/sgml/auto-explain.sgml b/doc/src/sgml/auto-explain.sgml index dd3f3fd..1b9d4d9 100644 --- a/doc/src/sgml/auto-explain.sgml +++ b/doc/src/sgml/auto-explain.sgml @@ -104,6 +104,25 @@ LOAD 'auto_explain'; varlistentry term + varnameauto_explain.log_buffers/varname (typeboolean/type) +/term +indexterm + primaryvarnameauto_explain.log_buffers/ configuration parameter/primary +/indexterm +listitem + para + varnameauto_explain.log_buffers/varname causes commandEXPLAIN + (ANALYZE, BUFFERS)/ output, rather than just commandEXPLAIN/ + output, to be printed when an execution plan is logged. This parameter is + off by default. Only superusers can change this setting. This + parameter has no effect unless varnameauto_explain.log_analyze/ + parameter is set. + /para +/listitem + /varlistentry + + varlistentry +term varnameauto_explain.log_format/varname (typeenum/type) /term indexterm diff --git a/doc/src/sgml/ref/explain.sgml b/doc/src/sgml/ref/explain.sgml index 0d03469..6c68afd 100644 --- a/doc/src/sgml/ref/explain.sgml +++ b/doc/src/sgml/ref/explain.sgml @@ -31,7 +31,7 @@ PostgreSQL documentation refsynopsisdiv synopsis -EXPLAIN [ ( { ANALYZE replaceable class=parameterboolean/replaceable | VERBOSE replaceable class=parameterboolean/replaceable | COSTS replaceable class=parameterboolean/replaceable | FORMAT { TEXT | XML | JSON | YAML } } [, ...] ) ] replaceable class=parameterstatement/replaceable +EXPLAIN [ ( { ANALYZE replaceable class=parameterboolean/replaceable | VERBOSE replaceable class=parameterboolean/replaceable | COSTS replaceable class=parameterboolean/replaceable | BUFFERS replaceable class=parameterboolean/replaceable | FORMAT { TEXT | XML | JSON | YAML } } [, ...] ) ] replaceable class=parameterstatement/replaceable EXPLAIN [ ANALYZE ] [ VERBOSE ] replaceable class=parameterstatement/replaceable /synopsis /refsynopsisdiv @@ -140,6 +140,24 @@ ROLLBACK; /varlistentry varlistentry +termliteralBUFFERS/literal/term +listitem + para + Include information on buffer usage. Specifically, include the number of + shared
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Greg Smith g...@2ndquadrant.com writes: Tom Lane wrote: It's amazing to me that we've never gone back and improved on the original quick-and-dirty MemoryContextStats mechanism. That code hasn't really gone anywhere since Neil tweaked the indentation two years ago. What sorts of improvements were you looking for? Not anything like what you were thinking about, apparently ;-). What I wish for the most often is a way to track palloc usage down to the particular file/line of the alloc call. This is obviously not something that we'd pay the overhead for in a standard build, but I would love to be able to build a debug version that could do it when I needed. There are malloc substitutes that can do this, but the palloc layer means that they're not especially useful for debugging the PG backend. Of course something like that doesn't have to have anything to do with what Zdenek was wishing for, but it just struck me that creating a stats-oriented shim layer in the memory context stuff is playing on pretty much the same turf. I started on trying to improve this area at one point but didn't get far. My first step was going to be just wrapping the call into a UDF to make it easier to reach without having to set loose gdb. I thought that might expand the possible uses for MemoryContextStats to help find a memory leak safely on the production box, and therefore attact more attention to improving it. Just for context: the reason MemoryContextStats is designed as it is is so that it has a reasonable chance of telling you something useful about an out-of-memory failure. When you're down to your last ten bytes, it's not going to help if your reporting mechanism wants to allocate a data structure to put its results in. So it writes to stderr and nothing else. It may well be that that constraint means we can never do anything really creative with MemoryContextStats per se. But I see the use-case for other mechanisms as being to get some info before we reach the point of having our backs to the wall. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Need a mentor, and a project.
On Fri, Dec 11, 2009 at 9:00 PM, Ashish abin...@u.washington.edu wrote: I am thinking about starting with the following TODO item: -- Have EXPLAIN ANALYZE issue NOTICE messages when the estimated and actual row counts differ by a specified percentage. I picked this because it is somewhat related to query processing which is what I am most interested in. It also seems like a good start up project for a newbie like me. Before I start looking into what this would involve and start a conversation on designing a solution - I wanted to know what you guys think about this particular TODO, and it suitability to a newbie. Looking forward to your comments... If we're going to do this, I think we should implement this as an optional behavior controlled by a new EXPLAIN option (maybe VARIANCE, following Bruce's patch?) and generate the output using ExplainPropertysome-data-type. We could possibly make the option take an optional threshold indicating how much variance is required before the variance gets displayed, and display the variance for every node if VARIANCE is specified without an argument. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Need a mentor, and a project.
On Fri, Dec 11, 2009 at 9:05 PM, Bruce Momjian br...@momjian.us wrote: Ashish wrote: I am thinking about starting with the following TODO item: -- Have EXPLAIN ANALYZE issue NOTICE messages when the estimated and actual row counts differ by a specified percentage. I picked this because it is somewhat related to query processing which is what I am most interested in. It also seems like a good start up project for a newbie like me. Before I start looking into what this would involve and start a conversation on designing a solution - I wanted to know what you guys think about this particular TODO, and it suitability to a newbie. Looking forward to your comments... I even have a sample patch you can use as a start, attached. Interesting. The logic in ExplainVariance() doesn't look right to me - the cases where one argument is zero seem like they will produce a differently-scaled result than otherwise. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] status update on Hot Standby and Streaming Replication
On Fri, 2009-12-11 at 20:38 -0500, Robert Haas wrote: I'm not quite sure where we stand with these two patches. My impression is that many of the outstanding TODO items in Hot Standby have been fixed, and I'm not sure what remains. Streaming Replication I think reviewing is not as far advanced, but I'm not sure. Any chance that Hot Standby can get committed in the next few days before we bundle alpha3? Otherwise, I think it will not get as much test coverage as it really deserves. It's possible that HS will get committed in next few days. I'm aiming to release a final patch within 48 hours and commit by Wed. The main TODO items are fixed, but I'm now going through testing and polishing. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Need a mentor, and a project.
Robert Haas wrote: On Fri, Dec 11, 2009 at 9:05 PM, Bruce Momjian br...@momjian.us wrote: Ashish wrote: I am thinking about starting with the following TODO item: -- Have EXPLAIN ANALYZE issue NOTICE messages when the estimated and actual row counts differ by a specified percentage. I picked this because it is somewhat related to query processing which is what I am most interested in. It also seems like a good start up project for a newbie like me. Before I start looking into what this would involve and start a conversation on designing a solution - I wanted to know what you guys think about this particular TODO, and it suitability to a newbie. Looking forward to your comments... I even have a sample patch you can use as a start, attached. Interesting. The logic in ExplainVariance() doesn't look right to me - the cases where one argument is zero seem like they will produce a differently-scaled result than otherwise. Yea, it is just a starting point for him. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Need a mentor, and a project.
Bruce Momjian br...@momjian.us writes: Ashish wrote: I am thinking about starting with the following TODO item: -- Have EXPLAIN ANALYZE issue NOTICE messages when the estimated and actual row counts differ by a specified percentage. I even have a sample patch you can use as a start, attached. Of course, the reason that patch isn't already in there is that it's pretty much useless. It clutters what's already cluttered output and doesn't do much of anything to help draw one's attention to the larger estimation errors, which of course is what the TODO item is really about. IMO the hard part of the TODO item is to design a useful user interface for highlighting specific EXPLAIN entries (and NOTICE messages probably ain't it either). Getting the numbers is trivial. I'm not sure there is any really nice solution within the confines of plain ASCII text output. There was an interesting approach online at http://explain-analyze.info, but that site seems to be down now :-( regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers