Re: [HACKERS] One less footgun: deprecating pg_dump -d
On Mon, Mar 09, 2009 at 09:02:01PM +0100, Magnus Hagander wrote: Andrew Dunstan wrote: Tom Lane wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: Magnus Hagander mag...@hagander.net wrote: but maybe it's better to use -i and -I, and thus change them both? That's already used: -i, --ignore-version proceed even when server version mismatches pg_dump version Proposal: drop the short forms of these two switches entirely. Anybody who actually needs the capability can write --inserts. +1. I was just thinking the same thing. +1, that sounds like a very good idea. FWIW, +1 from me for removing the -i and -d options, leaving only long versions of what they used to do. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prepping to break every past release...
On Wed, 2009-03-04 at 12:32 -0800, Joshua D. Drake wrote: Something that continues to grind my teeth about our software is that we are horribly inconsistent with our system catalogs. Now I am fully and 100% aware that changing this will break things in user land but I want to do it anyway. In order to do that I believe we need to come up with a very loud, extremely verbose method of communicating to people that 8.5 *will* break things. I agree strongly with your general point. The most consistent negative feedback I receive about Postgres is that we make minor changes from release to release that make it extremely difficult to upgrade without re-testing the applications. So we write great software, then make it difficult for people to upgrade to it. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sampling Profler for Postgres
On Mon, 2009-03-09 at 21:57 -0400, Tom Lane wrote: ITAGAKI Takahiro itagaki.takah...@oss.ntt.co.jp writes: For resource-based profilers, we have DTrace probes[1] and continue to extend them[2], but unfortunately DTrace only works on Solaris and limited platforms. FWIW, the systemtap guys are really, really close to having a working DTrace equivalent for Linux: http://gnu.wildebeest.org/diary/2009/02/24/systemtap-09-markers-everywhere/ It's not *quite* there for our purposes https://bugzilla.redhat.com/show_bug.cgi?id=488941 but I'll be surprised if I'm not dtracing on my Fedora 10 machine before the week is out. After all this time, you think it will be done in a week :-) I'm not at all convinced that we should be putting effort into a homegrown, partial substitute for DTrace. I was, but I'm not anymore. Do you think we will be able to enable this in builds for 8.4? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] One less footgun: deprecating pg_dump -d
On Mon, 2009-03-09 at 21:02 +0100, Magnus Hagander wrote: Andrew Dunstan wrote: Tom Lane wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: Magnus Hagander mag...@hagander.net wrote: but maybe it's better to use -i and -I, and thus change them both? That's already used: -i, --ignore-version proceed even when server version mismatches pg_dump version Proposal: drop the short forms of these two switches entirely. Anybody who actually needs the capability can write --inserts. +1. I was just thinking the same thing. +1, that sounds like a very good idea. +1 good plan -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prepping to break every past release...
On Thu, 2009-03-05 at 01:27 +, Andrew Gierth wrote: Now, of course, counting the upcoming 8.4 there have been three (and a bit - the original design predates 8.1, though it did anticipate some 8.1 features) new releases against which the original concept can be tested. And, guess what, nothing in those releases has even come close to invalidating the original design concept (as we knew all along). If you're still not convinced of that fact, it would be possible to take the original design and update it to 8.4 following the original plan. But I'm not prepared to spend any time on this if the only result is going to be more argument. I see the use for some more stable views. Would it be better to publish them as an external project? That way we can still use them for both old and new releases. Once the project takes hold it might then be included in core - but that's not hugely important if you can persuade people to include the project with the Windows installer. The problem with anything included in core is that we don't/can't quickly fix design flaws, so even if we did get something in now it might not do everything we want (and then we'd have to change it...). -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
On Tue, 2009-03-10 at 09:56 +0900, KaiGai Kohei wrote: Joshua D. Drake wrote: ... Is there any possibility of having it be enabled at compile time? The default would be know but those distributions that would like to make use of it could? It was the design a half year ago, but Bruce suggested me a certain feature should not be enabled/disabled by compile time options, except for library/platform dependency. In addition, he also suggested a feature should be turned on/off by configuration option, because of it enables to distribute a single binary for more wider users. SE-PostgreSQL need the libselinux to communicate the in-kernel SELinux. So, --enable-selinux is necessary on compile time, it is fair enough. If we omit it, all the sepgsql() invocations are replaced by empty macros. seems ok. Another option to disable it would be something similar to how we currently handle DTrace ? If we compile it with --enable-selinux, it has two working modes controled by a guc option: sepostgresql (bool). If it is disabled, all the sepgsql() invocations returns at the head of themself without doing anything. I believe this behavior follows the previous suggestion. Have you been able to measure any speed difference between --enable-selinux on and off ? -- Hannu Krosing http://www.2ndQuadrant.com PostgreSQL Scalability and Availability Services, Consulting and Training -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] postgresql.conf: patch to have ParseConfigFile report all parsing errors, then bail
On Sun, 2009-03-08 at 16:27 -0700, Selena Deckelmann wrote: ParseConfigFile currently exits on the first parsing error. Changed guc_file.l to report all parsing errors before exiting: * Moved parse_error: block inside while() loop * Removed cleanup_exit: and associated 'goto' * Added ereport if ParseConfigFile() returns false * changed OK to ok ;) * Added comment - TODO: Report bogus variables in addition to parsing errors before bailing out These are very good changes, good news. Is it possible to check for parameters that have been changed, yet will not be applied at reload? It's a common error to change something like shared_buffers and then expect that to have changed when you reload. It would be useful to issue messages when that has occurred. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
Hannu Krosing wrote:
On Tue, 2009-03-10 at 09:56 +0900, KaiGai Kohei wrote:
Joshua D. Drake wrote:
...
Is there any possibility of having it be enabled at compile time? The
default would be know but those distributions that would like to make
use of it could?
It was the design a half year ago, but Bruce suggested me a certain
feature should not be enabled/disabled by compile time options,
except for library/platform dependency.
In addition, he also suggested
a feature should be turned on/off by configuration option, because of
it enables to distribute a single binary for more wider users.
SE-PostgreSQL need the libselinux to communicate the in-kernel SELinux.
So, --enable-selinux is necessary on compile time, it is fair enough.
If we omit it, all the sepgsql() invocations are replaced by empty
macros.
seems ok.
Another option to disable it would be something similar to how we
currently handle DTrace ?
DTrace uses Makefile to hack it.
I don't think it is a good example for me.
[src/backend/utils/Makefile]
probes.h: probes.d
ifeq ($(enable_dtrace), yes)
$(DTRACE) -C -h -s $ -o $...@.tmp
sed -e 's/POSTGRESQL_/TRACE_POSTGRESQL_/g' $...@.tmp $@
rm $...@.tmp
else
sed -f $(srcdir)/Gen_dummy_probes.sed $ $@
endif
Another example:
* POSIX fadvise
It puts #ifdef ... #endif block inside the functions, so it means
caller invokes an empty functions when POSIX fadvise is disabled.
int
FilePrefetch(File file, off_t offset, int amount)
{
#if defined(USE_POSIX_FADVISE) defined(POSIX_FADV_WILLNEED)
int returnCode;
Assert(FileIsValid(file));
[snip]
return returnCode;
#else
Assert(FileIsValid(file));
return 0;
#endif
}
* LDAP
It put #ifdef .. #endif block both of implementations and caller
side, but the number of blocks are quite small.
Basically, I think many of #ifdef ... #endif blocks are noisy, so
the current manner (using empty macro) can keep the code clean.
But, I'll follows the manner if we have anything in this situation.
If we compile it with --enable-selinux, it has two working modes
controled by a guc option: sepostgresql (bool).
If it is disabled, all the sepgsql() invocations returns at
the head of themself without doing anything.
I believe this behavior follows the previous suggestion.
Have you been able to measure any speed difference between
--enable-selinux on and off ?
I don't have measurement on the current revision, so please wait for
a while to get it measured.
Previous measurement includes effects in row-level access controls:
http://kaigai.sblo.jp/article/20303941.html (01 Oct 2008)
Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei kai...@ak.jp.nec.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
Stephen Frost wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: KaiGai Kohei wrote: As I promised last week, SE-PostgreSQL patches are revised here: The patch adds permission checks to SET/SHOW. If that's useful functionality, it would be nice to see that as a separate patch, not requiring SE-Linux. My goodness. This patch seems to be going FAR beyond what I thought its charter was. I agree. I thought the idea was that the first round of SE-PostgreSQL additions would be to add SE hooks for permissions that PG already implements. Other permissions would then be implemented in a PG-way first, and SE hooks then added to those later. This seems to be a recurring theme with this patch. We stripped row-level permissions, now we have SET/SHOW and the function installation permissions. And the read/write file permissions. To make progress, we need to consider each new feature like that separately, as separate patches. KaiGei: Let's drop SET/SHOW permissions from the patch, I presume that's not critical for the overall goal. Dropping the function installation permissions would simplify the patch a lot. It would make the patch as whole a lot easier to swallow. Let's ask the same question as with the row-level permissions: If we drop the function installation stuff, would the rest of the patch still be useful? We can revisit that part in the future. I have the same concern as Tom about trying to curtail what superusers can do. We have not been concerned about what a superuser can and can't do before, so there could be small loopholes all over the codebase that we haven't thought about. Just as an example, you added checks to COPY to prevent reads from/writes to files. That's restricted to superusers. However, you left pg_read_file() in src/backend/utils/adt/genfile.c wide open. If we drop the goal of trying to restrict what a superuser can do, is the patch still useful? One idea is to add a single is superuser permission to sepgsql. That can be checked in a single place: superuser_arg(). If SE-Linux says that you don't have superuser permissions, then superuser() will return false even if the current user is marked as a superuser in pg_role. It would give the same level of protection as sprinkling those fine-grained checks all over the code, just in a more coarse-grain fashion. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
Heikki Linnakangas wrote: This seems to be a recurring theme with this patch. We stripped row-level permissions, now we have SET/SHOW and the function installation permissions. And the read/write file permissions. To make progress, we need to consider each new feature like that separately, as separate patches. KaiGei: Let's drop SET/SHOW permissions from the patch, I presume that's not critical for the overall goal. OK, its significance is relatively low. Dropping the function installation permissions would simplify the patch a lot. It would make the patch as whole a lot easier to swallow. Let's ask the same question as with the row-level permissions: If we drop the function installation stuff, would the rest of the patch still be useful? We can revisit that part in the future. As far as we have the idea of superuser permission on SELinux also, we can do it later. I have the same concern as Tom about trying to curtail what superusers can do. We have not been concerned about what a superuser can and can't do before, so there could be small loopholes all over the codebase that we haven't thought about. Just as an example, you added checks to COPY to prevent reads from/writes to files. That's restricted to superusers. However, you left pg_read_file() in src/backend/utils/adt/genfile.c wide open. Oops, it was my overlooks. If we drop the goal of trying to restrict what a superuser can do, is the patch still useful? I want to keep permission checks on files specified by users, because the superuser permission affects very wide scope, and all or nothing policy in other word. However, the combination of clients and files is not so simple, and I think it is necessary to apply permission checks individually. I can agree to postpone the checks for procedure installation, C-libraries installation/loading. One idea is to add a single is superuser permission to sepgsql. That can be checked in a single place: superuser_arg(). If SE-Linux says that you don't have superuser permissions, then superuser() will return false even if the current user is marked as a superuser in pg_role. It would give the same level of protection as sprinkling those fine-grained checks all over the code, just in a more coarse-grain fashion. At least, I think it is not a strange design. In-kernel SELinux also has similar permission (capability class) to restrict root privileges, in additon to the invididual checks. (NOTE: In Linux, root privilges is a set of capability.) Maybe, I can submit the revised patch within 1.5 days. Please wait for a while. Thanks, -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
KaiGai Kohei kai...@kaigai.gr.jp writes: Heikki Linnakangas wrote: If we drop the goal of trying to restrict what a superuser can do, is the patch still useful? I want to keep permission checks on files specified by users, because the superuser permission affects very wide scope, and all or nothing policy in other word. However, the combination of clients and files is not so simple, and I think it is necessary to apply permission checks individually. I would think the big advantage of something like SELinux is precisely in cases like this. So for example a client that has a capability that allows him to read a file can pass that capability to the server and be able to use COPY to read it directly on the server. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's PostGIS support! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] V4 of PITR performance improvement for 8.4
Fujii Masao wrote: Hi, On Mon, Mar 9, 2009 at 10:20 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Thanks. This patch seems to be missing the new readahead.c file. I grabbed that from the previous patch version. Oh, sorry for the mistake. I changed one of Suzuki-san's patches to be rebased to HEAD again (readahead-20090310.patch). The other (addShBufCheck-20090120.patch) is not changed. Comment: we might reach consistent recovery state *before* redoing the safe starting point, because readahead slightly delays the actual redo. Is this safe? No. If you haven't replayed all the WAL records up to the safe starting point, the database isn't consistent yet. The distinction doesn't matter in practice without Hot Standby, though. If not, the readahead queue should be flushed before reaching that state? Yes. Or you could move the reporting that you've reached the consistent recovery state into RedoRecords, when you reach the min safe starting point. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sampling Profler for Postgres
Simon Riggs si...@2ndquadrant.com writes: On Mon, 2009-03-09 at 21:57 -0400, Tom Lane wrote: I'm not at all convinced that we should be putting effort into a homegrown, partial substitute for DTrace. I was, but I'm not anymore. Do you think we will be able to enable this in builds for 8.4? The bugzilla entry I pointed to was asking me to enable it for 8.3. Which I did. It's certainly got some rough edges today, but I fully expect it to be usable when Fedora 11 ships. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: If we drop the goal of trying to restrict what a superuser can do, is the patch still useful? One idea is to add a single is superuser permission to sepgsql. The agreement back in January was that what we'd consider for 8.4 is a patch that adds SELinux-driven enforcement of permissions checks that already exist in Postgres. Allowing the above seems to me to fit within that charter, but this other stuff definitely doesn't. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prepping to break every past release...
Simon == Simon Riggs si...@2ndquadrant.com writes: Now, of course, counting the upcoming 8.4 there have been three (and a bit - the original design predates 8.1, though it did anticipate some 8.1 features) new releases against which the original concept can be tested. And, guess what, nothing in those releases has even come close to invalidating the original design concept (as we knew all along). If you're still not convinced of that fact, it would be possible to take the original design and update it to 8.4 following the original plan. But I'm not prepared to spend any time on this if the only result is going to be more argument. Simon I see the use for some more stable views. Simon Would it be better to publish them as an external project? They already are, though they are not complete and have not been maintained much for 8.1 and later; http://pgfoundry.org/projects/newsysviews/ Simon That way we can still use them for both old and new releases. It was always expected that they would be available on pgfoundry for use on releases prior to their inclusion in core. Simon Once the project takes hold it might then be included in core Speaking purely for myself, I'm not prepared to spend any time on it without an assurance that it will go into core if the project goals are reasonably met. As for Tom's opinion that this is impossible, there's an old saying: The one who says it cannot be done should not interrupt the one who is doing it. Simon The problem with anything included in core is that we Simon don't/can't quickly fix design flaws, so even if we did get Simon something in now it might not do everything we want (and then Simon we'd have to change it...). I'm not proposing that it go into core quickly; and certainly not before the design is properly reviewed, criticised, whatever. -- Andrew. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prepping to break every past release...
On Tue, Mar 10, 2009 at 08:46:28AM +, Simon Riggs wrote: On Thu, 2009-03-05 at 01:27 +, Andrew Gierth wrote: Now, of course, counting the upcoming 8.4 there have been three (and a bit - the original design predates 8.1, though it did anticipate some 8.1 features) new releases against which the original concept can be tested. And, guess what, nothing in those releases has even come close to invalidating the original design concept (as we knew all along). If you're still not convinced of that fact, it would be possible to take the original design and update it to 8.4 following the original plan. But I'm not prepared to spend any time on this if the only result is going to be more argument. I see the use for some more stable views. Would it be better to publish them as an external project? It's been an external project, newsysviews, since before 8.1 came out. I think it's time to bring it in from the cold. Call the new schema pg_sysviews, plop it in there, and call it done :) Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] postgresql.conf: patch to have ParseConfigFile report all parsing errors, then bail
Hi! Simon Riggs wrote: On Sun, 2009-03-08 at 16:27 -0700, Selena Deckelmann wrote: ParseConfigFile currently exits on the first parsing error. Changed guc_file.l to report all parsing errors before exiting: * Moved parse_error: block inside while() loop * Removed cleanup_exit: and associated 'goto' * Added ereport if ParseConfigFile() returns false * changed OK to ok ;) * Added comment - TODO: Report bogus variables in addition to parsing errors before bailing out These are very good changes, good news. Thanks :) Is it possible to check for parameters that have been changed, yet will not be applied at reload? This was already implemented! :) For example: LOG: attempted change of parameter shared_buffers ignored DETAIL: This parameter cannot be changed after server start. LOG: attempted change of parameter max_prepared_transactions ignored DETAIL: This parameter cannot be changed after server start. A thing that could be added, however, is reporting of all invalid (as opposed to valid, but requires a restart to apply) parameters before exiting. This change requires refactoring ProcessConfigFile() more significantly, as the parsing and validity checks are done separately, and are exited with gotos. :) I haven't tried to change this yet, but was planning to. -selena -- Selena Deckelmann End Point Corporation -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
On Tue, Mar 10, 2009 at 08:02:05PM +0900, KaiGai Kohei wrote: Please wait for a while. With all due respect to your hard work, waiting for this patch, even one more hour, is exactly what we shouldn't do for 8.4. Sad as it is, even if this patch were causing no controversy in its design, it would still be too late on grounds of its size and invasiveness. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prepping to break every past release...
On Tue, 2009-03-10 at 07:28 -0700, David Fetter wrote: Would it be better to publish them as an external project? It's been an external project, newsysviews, since before 8.1 came out. I think it's time to bring it in from the cold. Call the new schema pg_sysviews, plop it in there, and call it done :) Yeh Andrew said. That I never noticed in the last 3+ years makes me think there's not many people using it... -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] postgresql.conf: patch to have ParseConfigFile report all parsing errors, then bail
On Tue, 2009-03-10 at 07:30 -0700, Selena Deckelmann wrote: Is it possible to check for parameters that have been changed, yet will not be applied at reload? This was already implemented! :) For example: LOG: attempted change of parameter shared_buffers ignored DETAIL: This parameter cannot be changed after server start. LOG: attempted change of parameter max_prepared_transactions ignored DETAIL: This parameter cannot be changed after server start. OK, had that at the back of my mind for a while I guess. A thing that could be added, however, is reporting of all invalid (as opposed to valid, but requires a restart to apply) parameters before exiting. This change requires refactoring ProcessConfigFile() more significantly, as the parsing and validity checks are done separately, and are exited with gotos. :) I haven't tried to change this yet, but was planning to. Can't we just do a reload, but with doit = false? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] postgresql.conf: patch to have ParseConfigFile report all parsing errors, then bail
Simon Riggs wrote: On Tue, 2009-03-10 at 07:30 -0700, Selena Deckelmann wrote: A thing that could be added, however, is reporting of all invalid (as opposed to valid, but requires a restart to apply) parameters before exiting. This change requires refactoring ProcessConfigFile() more significantly, as the parsing and validity checks are done separately, and are exited with gotos. :) I haven't tried to change this yet, but was planning to. Can't we just do a reload, but with doit = false? That would also be cool. Like an 'apachectl configtest'. There are five places where a 'goto' is used in guc-file.l needs to be changed to make this work (in addition to adding an option to pg_ctl). It should be pretty straightforward, and I'll have a look at it tonight. -selena -- Selena Deckelmann End Point Corporation -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
On Mon, 2009-03-09 at 20:55 -0400, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: I know we are a little uncomfortable here but KaiGai-San (forgive me if I type that wrong) has proven to be a contributor in his own right, Perhaps it would help you calibrate the problem if I stated that I wouldn't trust a patch for this purpose written by myself, let alone somebody who hasn't been hacking the backend for ten years. (Where this purpose means the type of control KaiGai-san seems to hope to enforce, as opposed to just plugging some additional constraints into the existing ACL-check routines.) I think you misunderstand me. I have watched this thread very closely because it has specific strategic interest. For the record: * This patch does scare me * With great risk comes great reward So my question is, if the default is that sepostgres is disabled and can only be enabled via a compile time option, are your concerns just as weighty? What about marking the feature experimental. ./configure --help --enable-seEnables SE version of PostgreSQL for linux platforms (experimental) Yes it would be a break from what we do but it wouldn't hurt us to be just a little bit less stodgy as long as it is done in a responsible manner. Sincerely, Joshua D. Drake regards, tom lane -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prepping to break every past release...
On Tue, 2009-03-10 at 15:02 +, Simon Riggs wrote: On Tue, 2009-03-10 at 07:28 -0700, David Fetter wrote: Would it be better to publish them as an external project? It's been an external project, newsysviews, since before 8.1 came out. I think it's time to bring it in from the cold. Call the new schema pg_sysviews, plop it in there, and call it done :) Yeh Andrew said. That I never noticed in the last 3+ years makes me think there's not many people using it... Well I know of it and have never used it. Mainly because I didn't (and still don't) really know what it does. From an outsider looking in, the project is dead. The home page isn't updated (it talks about 8.1) and the CVS repo appears to not have had a commit in 2 years. How is anyone in the general community supposed to have any idea if this is a good idea or not? Sincerely, Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
Joshua D. Drake j...@commandprompt.com writes: I think you misunderstand me. I have watched this thread very closely because it has specific strategic interest. For the record: * This patch does scare me * With great risk comes great reward ... or great failure. My key concern is that we are setting ourselves up for failure by accepting a patch that hasn't attracted sufficient community interest. This patch needs way more eyeballs on it than it has gotten; which is not only bad in terms of the level of trust we should have in the patch right now, but it is a very negative signal about how much maintenance manpower it can expect in the future. Now the entire effort on KaiGai-san's part has been founded on the assumption that if you build it, they will come; and that is exactly the same argument I hear you making for continued investment in the project. But the track record so far, in terms of attracting hackers to work on the patch, says otherwise. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prepping to break every past release...
Joshua == Joshua D Drake j...@commandprompt.com writes: On Tue, 2009-03-10 at 15:02 +, Simon Riggs wrote: Yeh Andrew said. That I never noticed in the last 3+ years makes me think there's not many people using it... The fact that it never got beyond an early incomplete alpha version is a big factor in that. Joshua Well I know of it and have never used it. Mainly because I Joshua didn't (and still don't) really know what it does. From an Joshua outsider looking in, the project is dead. The home page isn't Joshua updated (it talks about 8.1) and the CVS repo appears to not Joshua have had a commit in 2 years. Other than some experiments in getting it to load on 8.2, there hasn't been any serious work done on it since May 2005, which is when it was presented (and shot down) on -hackers. The lack of useful feedback from -hackers also means that the design hasn't had much criticism, and therefore I don't regard the current definitions, the naming conventions, etc., as being cast in stone; which is another reason for people not to use it as it stands. (The plan we had when we started on it was to produce an alpha version as a proof-of-concept, present it on -hackers, get feedback, use that to sort out the naming conventions and a definitive set of definitions, and produce a beta version intended to be in the final form.) -- Andrew. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
On Tue, 2009-03-10 at 13:08 -0400, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: I think you misunderstand me. I have watched this thread very closely because it has specific strategic interest. For the record: * This patch does scare me * With great risk comes great reward ... or great failure. Sure, which all humans and projects must do at some point. It is how one learns after all. Sometimes the only thing you can do is fail. On the other hand if we succeed it will be a great reward. My key concern is that we are setting ourselves up for failure by accepting a patch that hasn't attracted sufficient community interest. This patch needs way more eyeballs on it than it has gotten; which is not only bad in terms of the level of trust we should have in the patch right now, but it is a very negative signal about how much maintenance manpower it can expect in the future. Now the entire effort on KaiGai-san's part has been founded on the assumption that if you build it, they will come; and that is exactly the same argument I hear you making for continued investment in the project. Yes but I am also offering an opportunity for others to show up. Which denying the patch does not do. If we provide SE support (even with marking it experimental), I would wager that some Linux distributions would begin to test it themselves which would allow us in turn to benefit by taking it out of experimental. Since RH, SuSE etc... are not going to Patch postgresql outside of some general compatibility issues. But all of this is moot. I see this as coming down to a simple result. * We don't enable it by default. * We mark it as experimental (or beta or whatever) Is there a serious regression in this line of thinking? It isn't unheard of in other projects. It allows the user to make a determination if they want to test/use the feature. It also continues the positive process of removing the fork which is pulling community away from us (at least to some degree) because those who are using SEpostgres are doing so out of his tree and not ours. Sincerely, Joshua D. Drake regards, tom lane -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prepping to break every past release...
On Tue, Mar 10, 2009 at 5:23 PM, Andrew Gierth and...@tao11.riddles.org.uk wrote: Other than some experiments in getting it to load on 8.2, there hasn't been any serious work done on it since May 2005, which is when it was presented (and shot down) on -hackers. If memory serves (and it may not - I'm practically brain dead from reviewing a large pgAdmin patch all day) - much of the 'shooting down' was at the suggestion that pgAdmin (and the like) should stop using the catalogues directly and should use newsysviews instead. I still maintain that'll never happen, but that doesn't mean that newsysviews wouldn't be useful for other classes of user. Perhaps pgsql-general would be a better place to poll. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
Joshua D. Drake escribió: Yes but I am also offering an opportunity for others to show up. Which denying the patch does not do. If we provide SE support (even with marking it experimental), I would wager that some Linux distributions would begin to test it themselves which would allow us in turn to benefit by taking it out of experimental. Since RH, SuSE etc... are not going to Patch postgresql outside of some general compatibility issues. It was said upthread that SEPostgres is already packaged for Fedora. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
On Tue, 2009-03-10 at 14:47 -0300, Alvaro Herrera wrote: Joshua D. Drake escribió: Yes but I am also offering an opportunity for others to show up. Which denying the patch does not do. If we provide SE support (even with marking it experimental), I would wager that some Linux distributions would begin to test it themselves which would allow us in turn to benefit by taking it out of experimental. Since RH, SuSE etc... are not going to Patch postgresql outside of some general compatibility issues. It was said upthread that SEPostgres is already packaged for Fedora. Yes for but not by, AFAIK it is not actually included with Fedora. Essentially it is packaged like the PGSQLRPMS are packaged, available but outside of the project itself. Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] problem inserting in GIN index
A guy just reported on pgsql-es-ayuda that he's getting ERROR: item pointer (543108,2) already exists It will be fine to get test case... Apparently this message only occurs on GIN, in insertItemPointer Reading that routine I cannot help but wonder -- where is gdi-btree.curitem incremented? At dataPlaceToPage and dataSplitPage called by ginInsertValue(). -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] problem inserting in GIN index
Apparently there's a crash involved ... Are other indexes on that table broken? ( Just count(*) with only index scan enabled ) -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
Joshua D. Drake j...@commandprompt.com writes: On Tue, 2009-03-10 at 14:47 -0300, Alvaro Herrera wrote: It was said upthread that SEPostgres is already packaged for Fedora. Yes for but not by, AFAIK it is not actually included with Fedora. Included with Fedora is an extremely loose concept. You can get it via yum install from the standard Fedora download servers. I don't believe it's counted as part of the PostgreSQL package group, nor included in the core distro CD set, but the CD-set approach to distribution seems to be dying anyway. There's too much stuff out there. However, if we did accept the patch, then the question would immediately become whether Devrim and I and other packagers for SELinux-capable distros ought to enable the feature in our builds. It does not work to deny responsibility for something by making it a configure option. You're just putting the hard decision onto packagers, who have no more knowledge than you do about what their users want, and (probably) considerably less understanding of the benefits/risks of some new configure option they happen to notice. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] problem inserting in GIN index
2009/3/10 Teodor Sigaev teo...@sigaev.ru: Apparently there's a crash involved ... Are other indexes on that table broken? ( Just count(*) with only index scan enabled ) -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers Yes, there are some btree indexes broken. Alvaro asked him if the option fsync is off. -- Emanuel Calvo Franco Sumate al ARPUG ! (www.postgres-arg.org - www.arpug.com.ar) ArPUG / AOSUG Member Postgresql Support Admin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: I know we are a little uncomfortable here but KaiGai-San (forgive me if I type that wrong) has proven to be a contributor in his own right, Not to put too fine a point on it, but: no, he hasn't. Show me one significant patch he's contributed before/beside this one. The only I thought Joshua was talking about his contribtions to F/OSS in general. He's credited on the NSA site for SELinux kernel scalability and locking issues: http://www.nsa.gov/research/selinux/contrib.shtml Kaigai Kohei of NEC replaced the original Access Vector Cache (AVC) locking scheme with a RCU-based approach, which solved the major SELinux kernel scalability problem, and fixed other locking issues in the SELinux kernel code. He later optimized the SELinux ebitmap implementation to improve performance on AVC misses. He also developed SE PostgreSQL, and is one of the developers for the SE busybox project. At first glance it seems it'd be valuable to have him as an active member of this community. Frankly, what we have here is a large patch, with insanely difficult correctness requirements, written by a Postgres newbie. I'm kinda hoping the discussion could turn to what parts (no matter how small) seem both useful safe enough for 8.4 - even if the main use of the small parts ar just as hooks to make it easier for SEPostgres to live as a parallel side project. As far as I can tell, the community feels interested in the feature set; but relatively unable to contribute since none of the people have that much of a security background. It seems the best way to fix that would be to get more people with a security background more involved. Not push them away. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
On Tue, 2009-03-10 at 10:49 -0700, Joshua D. Drake wrote: It was said upthread that SEPostgres is already packaged for Fedora. Yes for but not by, AFAIK it is not actually included with Fedora. It is, with the names sepostgresql*. Essentially it is packaged like the PGSQLRPMS are packaged, available but outside of the project itself. It is included in Fedora standard repositories: https://bugzilla.redhat.com/show_bug.cgi?id=249522 (It also includes my objections.) Regards, -- Devrim GÜNDÜZ, RHCE devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
On Tue, 2009-03-10 at 14:14 -0400, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: On Tue, 2009-03-10 at 14:47 -0300, Alvaro Herrera wrote: It was said upthread that SEPostgres is already packaged for Fedora. You're just putting the hard decision onto packagers, who have no more knowledge than you do about what their users want, and (probably) considerably less understanding of the benefits/risks of some new configure option they happen to notice. Which is exactly why we have two types of RPMS, --integer-datetimes and not. We would do the same thing with SE-Postgres. Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
Joshua D. Drake j...@commandprompt.com writes: On Tue, 2009-03-10 at 14:14 -0400, Tom Lane wrote: You're just putting the hard decision onto packagers, who have no more knowledge than you do about what their users want, and (probably) considerably less understanding of the benefits/risks of some new configure option they happen to notice. Which is exactly why we have two types of RPMS, --integer-datetimes and not. Maybe Devrim is doing that, but nobody else is. Debian went for --integer-datetimes years ago, Red Hat stuck with floats. Nobody is going to go to the trouble of maintaining two sets of RPMs, even assuming they notice there's a choice. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
On Tue, 2009-03-10 at 14:59 -0400, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: On Tue, 2009-03-10 at 14:14 -0400, Tom Lane wrote: You're just putting the hard decision onto packagers, who have no more knowledge than you do about what their users want, and (probably) considerably less understanding of the benefits/risks of some new configure option they happen to notice. At this point I don't know that any of this is going anywhere. I have presented what I think is a reasonable compromise to accept the feature. A compile-time option which was as designed all along with a flag called experimental. Which will be vastly easier to get people to test over time versus having to run a fork. I am for including this patch. I believe it is worth the risk. Sincerely, Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
On Tue, 2009-03-10 at 14:59 -0400, Tom Lane wrote: Which is exactly why we have two types of RPMS, --integer-datetimes and not. Maybe Devrim is doing that, but nobody else is. It is only available *if* yum repo conf file is specially configured and if the distro is Fedora 10 and RHEL 5. Those packages are not distributed in regular channels. I already used this switch in 8.4 packages to follow upstream. -- Devrim GÜNDÜZ, RHCE devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
Ron Mayer rm...@cheapcomplexdevices.com writes: As far as I can tell, the community feels interested in the feature set; but relatively unable to contribute since none of the people have that much of a security background. It seems the best way to fix that would be to get more people with a security background more involved. It's experience with the Postgres code base that I'm worried about. I don't question KaiGai-san's security background; I do doubt that he knows where all the skeletons are buried in the PG backend. A couple of very recent examples of that: his patch to fix a problem with inheritance of column privileges was approximately the right thing, but inefficiently duplicated the functionality of nearby code: http://archives.postgresql.org/pgsql-hackers/2009-03/msg00196.php and it didn't take Heikki long at all to note an oversight in the part of the latest sepostgres patch that attempted to confine superusers' file read/write abilities: http://archives.postgresql.org/pgsql-hackers/2009-03/msg00446.php More generally, there's been no discussion or community buy-in on design questions such as whether the patch should even try to confine superusers on such a fine-grained basis. (I agree with Heikki's thought that this may be a lost cause given our historical design assumption that superusers can do anything.) So I remain strongly of the opinion that what this patch lacks is review from longtime PG hackers. It's not the security community that is missing from the equation. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
On Tue, 2009-03-10 at 11:44 -0700, Joshua D. Drake wrote: We would do the same thing with SE-Postgres. No, no. I already experienced this with --integer-datetimes sets, and I don't ever want to maintain another set. It is horrible. -- Devrim GÜNDÜZ, RHCE devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Sampling Profler for Postgres
Hi! Tom Lane writes: I'm not at all convinced that we should be putting effort into a homegrown, partial substitute for DTrace. In my opinion providing DTrace as the only means of profiling would except a number of users from the tuning benefits. DTrace seems to rely on specific kernel options on Linux, which you might not be able to influence if you run your business on leased virtual servers hosted somewhere. DTrace is also not available for all platforms, most notably Windows. DTrace might be a great tool for the developers and should probably be used. For the rest of the world I see a benefit in having something like the proposed solution that could be enabled by the database administrator on every server or maybe even be the default. I think it would reduce the guesswork on why something might me slow and the work on 'probable' causes and establish more of a 'tuning by numbers' attitude. Looking at the existing probes in HEAD it this seems to be your target to provide high-level resource usage patterns to the user and I agree that this is the right abstraction layer. With this proposal I see a way of providing the resource usage in a (database) user-friendly way: namely as tupels that the user can access in a familiar manner and without using shell commands on a server that he might not even have access to. I also see an easy way of keeping historic data by copying the current state with a timestamp to a different table and then being able to look at performance problems of last night when nobody was there to notice it and fire up a profiler to watch it. Just my 0.02€. -- Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] problem inserting in GIN index
Emanuel Calvo Franco escribió: 2009/3/10 Teodor Sigaev teo...@sigaev.ru: Apparently there's a crash involved ... Are other indexes on that table broken? ( Just count(*) with only index scan enabled ) Yes, there are some btree indexes broken. Alvaro asked him if the option fsync is off. We don't know whether the btree indexes are on the same table, though :-) Since he already erased all the evidence, I don't think there's anything to do about this for now. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libxml incompatibility
Andrew Dunstan and...@dunslane.net writes: David Lee Lambert wrote: Is it supposed to be OK to call xmlCheckVersion() more than once? You are certainly not supposed to call xmlInitParser more than once - see http://xmlsoft.org/html/libxml-parser.html#xmlInitParser No, what that says is that it can't be called concurrently by more than one thread. If there were such a restriction then our own code wouldn't work at all, because we call it every time through xml_parse() or xpath(). Even if this were fixed, however, I'm still not convinced that we'll be able to call libxml2 from perl after we've installed our memory handler (xml_palloc). Yeah, I'm wondering about that too. It certainly wouldn't have the behavior that perl is expecting. We could possibly use xmlMemGet() to fetch the prior settings and then restore them after we are done, but making sure that happens after an error would be a bit tricky. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] parallel restore fixes
Tom Lane wrote:
How about this: by default, fmtId uses the same logic as now (one static
PQExpBuffer). If told to by a call of init_parallel_dump_utils(), which
need only be called by pg_restore during its startup, then it switches to
using per-thread storage. init_parallel_dump_utils can be the place
that calls TlsAlloc. This is almost the same as what you suggested a
couple messages back, but perhaps a bit clearer as to what's going on;
and it definitely cuts the number of places we need to touch.
OK, here 'tis.
Moving on to the deadlock with crossed FKs issue.
cheers
andrew
? src/bin/pg_dump/.deps
? src/bin/pg_dump/kwlookup.c
? src/bin/pg_dump/win32ver.rc
Index: src/bin/pg_dump/dumputils.c
===
RCS file: /home/cvsmirror/pg/pgsql/src/bin/pg_dump/dumputils.c,v
retrieving revision 1.44
diff -c -r1.44 dumputils.c
*** src/bin/pg_dump/dumputils.c 22 Jan 2009 20:16:07 - 1.44
--- src/bin/pg_dump/dumputils.c 10 Mar 2009 19:38:07 -
***
*** 31,55
static void AddAcl(PQExpBuffer aclbuf, const char *keyword,
const char *subname);
/*
! * Quotes input string if it's not a legitimate SQL identifier as-is.
*
! * Note that the returned string must be used before calling fmtId again,
! * since we re-use the same return buffer each time. Non-reentrant but
! * avoids memory leakage.
*/
const char *
fmtId(const char *rawid)
{
! static PQExpBuffer id_return = NULL;
const char *cp;
bool need_quotes = false;
if (id_return)/* first time through? */
resetPQExpBuffer(id_return);
else
id_return = createPQExpBuffer();
/*
* These checks need to match the identifier production in scan.l. Don't
--- 31,102
static void AddAcl(PQExpBuffer aclbuf, const char *keyword,
const char *subname);
+ #ifdef WIN32
+ static bool parallel_init_done = false;
+ static DWORD tls_index;
+ #endif
+
+ void
+ init_parallel_dump_utils(void)
+ {
+ #ifdef WIN32
+ if (! parallel_init_done)
+ {
+ tls_index = TlsAlloc();
+ parallel_init_done = true;
+ }
+ #endif
+ }
/*
! * Quotes input string if it's not a legitimate SQL identifier as-is.
*
! * Note that the returned string must be used before calling fmtId again,
! * since we re-use the same return buffer each time. Non-reentrant but
! * reduces memory leakage. (On Windows the memory leakage will be one buffer
! * per thread, which is at least better than one per call).
*/
const char *
fmtId(const char *rawid)
{
! /*
! * The Tls code goes awry if we use a static var, so we provide for both
! * static and auto, and omit any use of the static var when using Tls.
! */
! static PQExpBuffer s_id_return = NULL;
! PQExpBuffer id_return;
!
const char *cp;
bool need_quotes = false;
+ #ifdef WIN32
+ if (parallel_init_done)
+ id_return = (PQExpBuffer) TlsGetValue(tls_index); /* 0 when not set */
+ else
+ id_return = s_id_return;
+ #else
+ id_return = s_id_return;
+ #endif
+
if (id_return)/* first time through? */
+ {
+ /* same buffer, just wipe contents */
resetPQExpBuffer(id_return);
+ }
else
+ {
+ /* new buffer */
id_return = createPQExpBuffer();
+ #ifdef WIN32
+ if (parallel_init_done)
+ TlsSetValue(tls_index,id_return);
+ else
+ s_id_return = id_return;
+ #else
+ s_id_return = id_return;
+ #endif
+
+ }
/*
* These checks need to match the identifier production in scan.l. Don't
Index: src/bin/pg_dump/dumputils.h
===
RCS file: /home/cvsmirror/pg/pgsql/src/bin/pg_dump/dumputils.h,v
retrieving revision 1.23
diff -c -r1.23 dumputils.h
*** src/bin/pg_dump/dumputils.h 22 Jan 2009 20:16:07 - 1.23
--- src/bin/pg_dump/dumputils.h 10 Mar 2009 19:38:07 -
***
*** 19,24
--- 19,25
#include libpq-fe.h
#include pqexpbuffer.h
+ extern void init_parallel_dump_utils(void);
extern const char *fmtId(const char *identifier);
extern void appendStringLiteral(PQExpBuffer buf, const char *str,
int encoding, bool std_strings);
Index: src/bin/pg_dump/pg_backup_archiver.c
===
RCS file: /home/cvsmirror/pg/pgsql/src/bin/pg_dump/pg_backup_archiver.c,v
retrieving revision 1.165
diff -c -r1.165 pg_backup_archiver.c
*** src/bin/pg_dump/pg_backup_archiver.c 5 Mar 2009 14:51:10 - 1.165
--- src/bin/pg_dump/pg_backup_archiver.c 10 Mar 2009 19:38:07 -
***
*** 3467,3478
/*
* Close and reopen the input file so we have a private file pointer
! * that doesn't stomp on anyone else's file pointer.
*
! * Note: on Windows, since we are using threads not processes, this
! * *doesn't* close the original file pointer but just open a new one.
*/
! (AH-ReopenPtr) (AH);
/*
* We need our own database
Re: [HACKERS] parallel restore fixes
Andrew Dunstan and...@dunslane.net writes: OK, here 'tis. Looks fairly reasonable to me, but of course I haven't tested it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] About the new %sdt macro in F-11 package
Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= dev...@gunduz.org writes: I was trying to make a scratch build in Koji for orafce package, and it failed: http://koji.fedoraproject.org/koji/getfile?taskID=1235602name=build.log I consulted #fedora-devel @ Freenode, and the conclusion was this: walters devrimgunduz: the devel subpackage needs a Requires: systemtap-sdt-devel He is talking about postgresql-devel package actually. The reason this is a problem is that in 8.3, pg_trace.h is included by c.h, which means that *anything* built against Postgres headers needs access to the dtrace headers if we configured --enable-dtrace. As of CVS HEAD, I see that we got rid of that, and pg_trace.h is now included only by the .c files that actually need it. ISTM that we should make 8.3 do likewise. Having postgresql-devel drag in systemtap-sdt-devel would pretty much suck IMHO. Comments? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] About the new %sdt macro in F-11 package
Tom Lane wrote: As of CVS HEAD, I see that we got rid of that, and pg_trace.h is now included only by the .c files that actually need it. ISTM that we should make 8.3 do likewise. Here's a patch for this. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support Index: src/backend/access/transam/xact.c === RCS file: /home/alvherre/Code/cvs/pgsql/src/backend/access/transam/xact.c,v retrieving revision 1.257.2.2 diff -c -p -r1.257.2.2 xact.c *** src/backend/access/transam/xact.c 26 Apr 2008 23:35:33 - 1.257.2.2 --- src/backend/access/transam/xact.c 10 Mar 2009 23:02:58 - *** *** 33,38 --- 33,39 #include executor/spi.h #include libpq/be-fsstubs.h #include miscadmin.h + #include pg_trace.h #include pgstat.h #include storage/fd.h #include storage/lmgr.h Index: src/backend/storage/lmgr/lock.c === RCS file: /home/alvherre/Code/cvs/pgsql/src/backend/storage/lmgr/lock.c,v retrieving revision 1.181.2.1 diff -c -p -r1.181.2.1 lock.c *** src/backend/storage/lmgr/lock.c 4 Mar 2008 19:54:13 - 1.181.2.1 --- src/backend/storage/lmgr/lock.c 10 Mar 2009 23:04:22 - *** *** 36,41 --- 36,42 #include access/twophase.h #include access/twophase_rmgr.h #include miscadmin.h + #include pg_trace.h #include pgstat.h #include utils/memutils.h #include utils/ps_status.h Index: src/backend/storage/lmgr/lwlock.c === RCS file: /home/alvherre/Code/cvs/pgsql/src/backend/storage/lmgr/lwlock.c,v retrieving revision 1.50 diff -c -p -r1.50 lwlock.c *** src/backend/storage/lmgr/lwlock.c 1 Jan 2008 19:45:52 - 1.50 --- src/backend/storage/lmgr/lwlock.c 10 Mar 2009 23:04:42 - *** *** 25,30 --- 25,31 #include access/multixact.h #include access/subtrans.h #include miscadmin.h + #include pg_trace.h #include storage/ipc.h #include storage/proc.h #include storage/spin.h Index: src/include/c.h === RCS file: /home/alvherre/Code/cvs/pgsql/src/include/c.h,v retrieving revision 1.222.2.1 diff -c -p -r1.222.2.1 c.h *** src/include/c.h 23 Feb 2008 19:11:55 - 1.222.2.1 --- src/include/c.h 10 Mar 2009 23:01:54 - *** *** 57,63 #include pg_config_os.h /* must be before any system header files */ #endif #include postgres_ext.h - #include pg_trace.h #if _MSC_VER = 1400 #define errcode __msvc_errcode --- 57,62 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] About the new %sdt macro in F-11 package
Alvaro Herrera alvhe...@commandprompt.com writes: Tom Lane wrote: As of CVS HEAD, I see that we got rid of that, and pg_trace.h is now included only by the .c files that actually need it. ISTM that we should make 8.3 do likewise. Here's a patch for this. Oh, thanks, I was just about to go off and do that myself, but you saved me the trouble. Please commit. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
Hannu Krosing wrote: If we compile it with --enable-selinux, it has two working modes controled by a guc option: sepostgresql (bool). If it is disabled, all the sepgsql() invocations returns at the head of themself without doing anything. I believe this behavior follows the previous suggestion. Have you been able to measure any speed difference between --enable-selinux on and off ? At the last night, I measured TPS by pgbench in three cases: 1) A binary compiled without --enable-selinux 2) A binary compiled with --enable-selinux, runtime disabled 3) A binary compiled with --enable-selinux, runtime enabled Then, I cannot observe statically meaningful differences here. * Environment CPU: Core2Duo E6400 (2.13GHz) Mem: 2048MB kernel: 2.6.28-3.fc11.i686 * Parameters - shared_buffers = 512MB - rest of parameters are in the default * Benchmarch % pgbench -i -s 10 postgres % pgbench -c 2 -t 10 postgres --- 6 times * Results (1) compiled without --enable-selinux 1st: 478.565569 2nd: 478.223391 3rd: 442.365636 4th: 468.988499 5th: 482.173836 6th: 448.208615 - AVG: 466.420924 (STD: 17.0404) (2) compiled with --enable-selinux, runtime disabled 1st: 469.005777 2nd: 485.602091 3rd: 449.096123 4th: 460.657368 5th: 476.791923 6th: 444.027405 - AVG: 464.196781 (STD: 16.0456) (3) compiled with --enable-selinux, runtime enabled 1st: 462.702242 2nd: 473.312013 3rd: 442.214347 4th: 468.465614 5th: 473.498682 6th: 468.973759 - AVG: 464.861109 (STD: 11.7768) -- OSS Platform Development Division, NEC KaiGai Kohei kai...@ak.jp.nec.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] About the new %sdt macro in F-11 package
Tom Lane wrote: Alvaro Herrera alvhe...@commandprompt.com writes: Tom Lane wrote: As of CVS HEAD, I see that we got rid of that, and pg_trace.h is now included only by the .c files that actually need it. ISTM that we should make 8.3 do likewise. Here's a patch for this. Oh, thanks, I was just about to go off and do that myself, but you saved me the trouble. Please commit. Done. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sampling Profler for Postgres
Dickson S. Guedes lis...@guedesoft.net wrote: 2) I couldn't find a clear way to disable it. There is one in this patch or are you planning this to future? Ah, I forgot sampling should be disabled when track_activities is off. I'll fix it in the next patch. Also, I'd better measure overheads by the patch. Will be very nice if I could on/off it. When done, please send us. I'd like to test it in some stress scenarios, enabling and disabling it on some environment and comparing with my old benchmarks. Here is a new version of the patch. I added a new GUC parameter 'profiling_interval' (ms). Profiling is disabled when the value is 0. The default value is 1 second. You could get more granular results if you set the value to 100-500ms, but 1 sec should be enough for continuous regular load (like benchmarks). I cannot see any differences whether profiling is on/off. So I think sampling has little overheads for now. Please notify me report if you see troubles. Regards, --- ITAGAKI Takahiro NTT Open Source Software Center profiler_0311.tar.gz Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] V4 of PITR performance improvement for 8.4
Hi, 2009/3/10 Heikki Linnakangas heikki.linnakan...@enterprisedb.com: Fujii Masao wrote: Hi, On Mon, Mar 9, 2009 at 10:20 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Thanks. This patch seems to be missing the new readahead.c file. I grabbed that from the previous patch version. Oh, sorry for the mistake. I changed one of Suzuki-san's patches to be rebased to HEAD again (readahead-20090310.patch). The other (addShBufCheck-20090120.patch) is not changed. Comment: we might reach consistent recovery state *before* redoing the safe starting point, because readahead slightly delays the actual redo. Is this safe? No. If you haven't replayed all the WAL records up to the safe starting point, the database isn't consistent yet. The distinction doesn't matter in practice without Hot Standby, though. If not, the readahead queue should be flushed before reaching that state? Yes. Or you could move the reporting that you've reached the consistent recovery state into RedoRecords, when you reach the min safe starting point. This arrangement can be done with Hot Standby and Sync.Rep, right? So far, it sounds that we need to add a code to handle if malloc() fails (OOM). In this case, possible way could be to skip whole readahead, although the rest of the recovery may fail because of the memory shortage. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- -- Koichi Suzuki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1710)
Heikki, it is the list of updated patches:
http://sepgsql.googlecode.com/files/sepgsql-core-8.4devel-r1710.patch
http://sepgsql.googlecode.com/files/sepgsql-utils-8.4devel-r1710.patch
http://sepgsql.googlecode.com/files/sepgsql-policy-8.4devel-r1710.patch
http://sepgsql.googlecode.com/files/sepgsql-docs-8.4devel-r1710.patch
http://sepgsql.googlecode.com/files/sepgsql-tests-8.4devel-r1710.patch
- List of updates:
* Permission checks on SET/SHOW were removed.
* Add a new permission: db_database:{superuser}
sepgsqlCheckDatabaseSuperuser() is invoked from superuser_arg()
to check whether the clietn can perform as a superuser in this
database, or not.
* Permission checks on procedure installation is separated.
* Permission checks on install/load C-libraries are separated.
* Read file checks on pg_read_file() is added.
- Scale of patches:
* r1710 (the latest revision)
60 files changed, 3686 insertions(+), 10 deletions(-), 4952 modifications(!)
* r1704 (previous revision)
60 files changed, 4048 insertions(+), 11 deletions(-), 4944 modifications(!)
... about 300 lines were downsized.
- Remaining issue:
* ACL_SELECT_FOR_UPDATE has same value with ACL_UPDATE, so SE-PostgreSQL
checks db_table:{update} permission on SELECT ... FOR SHARE OF,
instead of db_table:{lock} permission.
Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei kai...@ak.jp.nec.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1704)
Tom Lane wrote: Ron Mayer rm...@cheapcomplexdevices.com writes: As far as I can tell, the community feels interested in the feature set; but relatively unable to contribute since none of the people have that much of a security background. It seems the best way to fix that would be to get more people with a security background more involved. It's experience with the Postgres code base that I'm worried about. I don't question KaiGai-san's security background; I do doubt that he knows where all the skeletons are buried in the PG backend. A couple of very recent examples of that: his patch to fix a problem with inheritance of column privileges was approximately the right thing, but inefficiently duplicated the functionality of nearby code: http://archives.postgresql.org/pgsql-hackers/2009-03/msg00196.php and it didn't take Heikki long at all to note an oversight in the part of the latest sepostgres patch that attempted to confine superusers' file read/write abilities: http://archives.postgresql.org/pgsql-hackers/2009-03/msg00446.php Indeed, I have less than three years experience of development in PostgreSQL backend. However, I don't believe it is a productive discussion to point out such kind of failures. At least, I think it is worthwhile to report bugs/submit patches much more than keeping silent with being afraid of failures. If submitted patches are not still enough elegant, we can fix and improve them via discussions. More generally, there's been no discussion or community buy-in on design questions such as whether the patch should even try to confine superusers on such a fine-grained basis. (I agree with Heikki's thought that this may be a lost cause given our historical design assumption that superusers can do anything.) So I remain strongly of the opinion that what this patch lacks is review from longtime PG hackers. It's not the security community that is missing from the equation. Two months ago, I agreed to postpone some of features especially hot in discussion, to reduce the scale of patches and burden of reviewers on the v8.4 development phase. In addition, I also reduced more than 1,000 lines as Heikki suggested. Its purpose is to focus the points to be discussed. I would like to have a productive discssion. -- OSS Platform Development Division, NEC KaiGai Kohei kai...@ak.jp.nec.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
