Re: [HACKERS] One-Shot Plans
On Tue, Jun 14, 2011 at 1:25 PM, Simon Riggs wrote:
>
> We can work out the various paths through the traffic cop to see when
> a plan will be a "one-shot" - planned and then executed immediately,
> then discarded.
>
> In those cases we can take advantage of better optimisations. Most
> interestingly, we can evaluate stable functions at plan time, to allow
> us to handle partitioning and partial indexes better.
>
> Patch attached. Works...
>
this breaks test guc.c for me... specifically the test at
src/test/regress/sql/guc.sql circa line 226:
"""
set work_mem = '3MB';
-- but SET isn't
create or replace function myfunc(int) returns text as $$
begin
set work_mem = '2MB';
return current_setting('work_mem');
end $$
language plpgsql
set work_mem = '1MB';
select myfunc(0), current_setting('work_mem');
"""
regressions.diff
"""
*** 656,662
select myfunc(0), current_setting('work_mem');
myfunc | current_setting
+-
! 2MB| 2MB
(1 row)
set work_mem = '3MB';
--- 656,662
select myfunc(0), current_setting('work_mem');
myfunc | current_setting
+-
! 2MB| 3MB
(1 row)
set work_mem = '3MB';
"""
it seems that the effect of SET is being discarded
--
Jaime Casanova www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific
2011/6/21 Brendan Jurd :
> On 21 June 2011 14:34, Pavel Stehule wrote:
>> I don't understand to using a macro
>>
>> #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
>>
>> because you disallowed a quoting?
>
> Well, a token can only be treated as a special keyword if it is unquoted.
>
> As an example, in the 'database' field, the bare token 'replication'
> is a keyword meaning the pseudo-database for streaming rep. Whereas
> the quoted token "replication" would mean a real database which is
> called 'replication'.
>
> Likewise, the bare token 'all' in the username field is a keyword --
> it matches any username. Whereas the quoted token "all" would only
> match a user named 'all'.
>
> Therefore, token_is_keyword only returns true where the token matches
> the given string as is also unquoted.
>
> Does that make sense?
yes - it has a sense. Quoting changes sense from keyword to literal.
But then I see a significant inconsistency - every know keywords
should be only tokens.
else if (strcmp(token, "pamservice") == 0)
- {
- REQUIRE_AUTH_OPTION(uaPAM, "pamservice", "pam");
- parsedline->pamservice = pstrdup(c);
- }
because >>pamservice<< - is known keyword, but 'pamservice' is some
literal without any mean. You should to use a makro token_is_keyword
more often.
??
Regards
Pavel
>
> Cheers,
> BJ
>
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FOR KEY LOCK foreign keys
On 2011-06-20 22:11, Noah Misch wrote: On Sun, Jun 19, 2011 at 06:30:41PM +0200, Jesper Krogh wrote: I hope this hasn't been forgotten. But I cant see it has been committed or moved into the commitfest process? If you're asking about that main patch for $SUBJECT rather than those isolationtester changes specifically, I can't speak to the plans for it. I wasn't planning to move the test suite work forward independent of the core patch it serves, but we could do that if there's another application. Yes, I was actually asking about the main patch for foreign key locks. Jesper -- Jesper -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific
On 21 June 2011 14:34, Pavel Stehule wrote: > I don't understand to using a macro > > #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) > > because you disallowed a quoting? Well, a token can only be treated as a special keyword if it is unquoted. As an example, in the 'database' field, the bare token 'replication' is a keyword meaning the pseudo-database for streaming rep. Whereas the quoted token "replication" would mean a real database which is called 'replication'. Likewise, the bare token 'all' in the username field is a keyword -- it matches any username. Whereas the quoted token "all" would only match a user named 'all'. Therefore, token_is_keyword only returns true where the token matches the given string as is also unquoted. Does that make sense? Cheers, BJ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific
2011/6/21 Brendan Jurd : > On 21 June 2011 13:51, Pavel Stehule wrote: >> I have one question. I can't find any rules for work with tokens, etc, >> where is quotes allowed and disallowed? >> >> I don't see any other issues. > > I'm not sure I understand your question, but quotes are allowed > anywhere and they always act to remove any special meaning the token > might otherwise have had. It's much like quoting a column name in > SQL. I don't understand to using a macro #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) because you disallowed a quoting? Regards Pavel > > I didn't change any of the hba parsing rules, so escaping and whatnot > should all work the way it did before. The only difference should be > that after parsing, keywords will only be evaluated for fields where > they matter. > > Cheers, > BJ > -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific
On 21 June 2011 13:51, Pavel Stehule wrote: > I have one question. I can't find any rules for work with tokens, etc, > where is quotes allowed and disallowed? > > I don't see any other issues. I'm not sure I understand your question, but quotes are allowed anywhere and they always act to remove any special meaning the token might otherwise have had. It's much like quoting a column name in SQL. I didn't change any of the hba parsing rules, so escaping and whatnot should all work the way it did before. The only difference should be that after parsing, keywords will only be evaluated for fields where they matter. Cheers, BJ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch - Debug builds without optimization
On 06/20/2011 01:34 PM, Tom Lane wrote: I was trying to illustrate how to have minimal turnaround time when testing a small code change. Rebuilding from scratch is slow enough that you lose focus while waiting. (Or I do, anyway.) I just keep upgrading to the fastest CPU I can possibly justify to avoid losing focus; it goes fast with 8 cores. I was trying to demonstrate that peg makes this very high level now, and I was more jousting at the idea that everyone should bother to write their own individual reinstall script. The peg code makes it easy to assimilate whatever other neat optimization ideas one might come across. I just pushed an update out that absorbed this one, so now if you do: stop peg rebuild It uses the install-bin trick you suggested. It even does a couple of sanity checks so that it will probably fall back to a regular build if it doesn't look like you have a good install and binary tree already. Maybe I'll make a "reinstall" alias that does this combination next. I don't expect to improve your workflow. But people who haven't already invested a good chunk of work in automating things already will probably take some time to catch up with where peg puts them on day one. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific
2011/6/20 Alvaro Herrera : > Excerpts from Alvaro Herrera's message of lun jun 20 12:19:37 -0400 2011: >> Excerpts from Pavel Stehule's message of lun jun 20 11:34:25 -0400 2011: >> >> > b) probably you can simplify a memory management using own two >> > persistent memory context - and you can swap it. Then functions like >> > free_hba_record, clean_hba_list, free_lines should be removed. >> >> Yeah, I reworked the patch with something like that over the weekend. >> Not all of it though. I'll send an updated version shortly. > > Here it is, please let me know what you think. I took the liberty of > cleaning up some things that were clearly historical leftovers. > I have one question. I can't find any rules for work with tokens, etc, where is quotes allowed and disallowed? I don't see any other issues. Regards Pavel Stehule > -- > Álvaro Herrera > The PostgreSQL Company - Command Prompt, Inc. > PostgreSQL Replication, Consulting, Custom Development, 24x7 support > -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pika buildfarm member failure on isolationCheck tests
While testing the fix for this one, I found another bug. Patches for
both are attached.
The first patch addresses this bug by re-adding SXACT_FLAG_ROLLED_BACK,
in a more limited form than its previous incarnation.
We need to be able to distinguish transactions that have already
called ReleasePredicateLocks and are thus eligible for cleanup from
those that have been merely marked for abort by other
backends. Transactions that are ROLLED_BACK are excluded from
SxactGlobalXmin calculations, but those that are merely DOOMED need to
be included.
Also update a couple of assertions to ensure we only try to clean up
ROLLED_BACK transactions.
The second patch fixes a bug in PreCommit_CheckForSerializationFailure.
This function checks whether there's a dangerous structure of the form
far ---> near ---> me
where neither the "far" or "near" transactions have committed. If so,
it aborts the "near" transaction by marking it as DOOMED. However, that
transaction might already be PREPARED. We need to check whether that's
the case and, if so, abort the transaction that's trying to commit
instead.
One of the prepared_xacts regression tests actually hits this bug.
I removed the anomaly from the duplicate-gids test so that it fails in
the intended way, and added a new test to check serialization failures
with a prepared transaction.
Dan
--
Dan R. K. Ports MIT CSAILhttp://drkp.net/
diff --git a/src/backend/storage/lmgr/predicate.c b/src/backend/storage/lmgr/predicate.c
index 6c55211..3678878 100644
--- a/src/backend/storage/lmgr/predicate.c
+++ b/src/backend/storage/lmgr/predicate.c
@@ -246,7 +246,6 @@
#define SxactIsCommitted(sxact) (((sxact)->flags & SXACT_FLAG_COMMITTED) != 0)
#define SxactIsPrepared(sxact) (((sxact)->flags & SXACT_FLAG_PREPARED) != 0)
-#define SxactIsRolledBack(sxact) (((sxact)->flags & SXACT_FLAG_ROLLED_BACK) != 0)
#define SxactIsDoomed(sxact) (((sxact)->flags & SXACT_FLAG_DOOMED) != 0)
#define SxactIsReadOnly(sxact) (((sxact)->flags & SXACT_FLAG_READ_ONLY) != 0)
#define SxactHasSummaryConflictIn(sxact) (((sxact)->flags & SXACT_FLAG_SUMMARY_CONFLICT_IN) != 0)
@@ -3047,7 +3046,7 @@ SetNewSxactGlobalXmin(void)
for (sxact = FirstPredXact(); sxact != NULL; sxact = NextPredXact(sxact))
{
- if (!SxactIsRolledBack(sxact)
+ if (!SxactIsDoomed(sxact)
&& !SxactIsCommitted(sxact)
&& sxact != OldCommittedSxact)
{
@@ -3114,7 +3113,6 @@ ReleasePredicateLocks(const bool isCommit)
Assert(!isCommit || SxactIsPrepared(MySerializableXact));
Assert(!isCommit || !SxactIsDoomed(MySerializableXact));
Assert(!SxactIsCommitted(MySerializableXact));
- Assert(!SxactIsRolledBack(MySerializableXact));
/* may not be serializable during COMMIT/ROLLBACK PREPARED */
if (MySerializableXact->pid != 0)
@@ -3153,22 +3151,7 @@ ReleasePredicateLocks(const bool isCommit)
MySerializableXact->flags |= SXACT_FLAG_READ_ONLY;
}
else
- {
- /*
- * The DOOMED flag indicates that we intend to roll back this
- * transaction and so it should not cause serialization
- * failures for other transactions that conflict with
- * it. Note that this flag might already be set, if another
- * backend marked this transaction for abort.
- *
- * The ROLLED_BACK flag further indicates that
- * ReleasePredicateLocks has been called, and so the
- * SerializableXact is eligible for cleanup. This means it
- * should not be considered when calculating SxactGlobalXmin.
- */
MySerializableXact->flags |= SXACT_FLAG_DOOMED;
- MySerializableXact->flags |= SXACT_FLAG_ROLLED_BACK;
- }
if (!topLevelIsDeclaredReadOnly)
{
@@ -3544,7 +3527,7 @@ ReleaseOneSerializableXact(SERIALIZABLEXACT *sxact, bool partial,
nextConflict;
Assert(sxact != NULL);
- Assert(SxactIsRolledBack(sxact) || SxactIsCommitted(sxact));
+ Assert(SxactIsDoomed(sxact) || SxactIsCommitted(sxact));
Assert(LWLockHeldByMe(SerializableFinishedListLock));
/*
diff --git a/src/include/storage/predicate_internals.h b/src/include/storage/predicate_internals.h
index 34c661d..495983f 100644
--- a/src/include/storage/predicate_internals.h
+++ b/src/include/storage/predicate_internals.h
@@ -90,22 +90,21 @@ typedef struct SERIALIZABLEXACT
int pid; /* pid of associated process */
} SERIALIZABLEXACT;
-#define SXACT_FLAG_COMMITTED 0x0001 /* already committed */
-#define SXACT_FLAG_PREPARED0x0002 /* about to commit */
-#define SXACT_FLAG_ROLLED_BACK 0x0004 /* already rolled back */
-#define SXACT_FLAG_DOOMED0x0008 /* will roll back */
+#define SXACT_FLAG_COMMITTED0x0001 /* already committed */
+#define SXACT_FLAG_PREPARED 0x0002 /* about to commit */
+#define SXACT_FLAG_DOOMED 0x0004 /* will roll back */
/*
* The following flag actually means that the flagged transaction has a
* conflict out *to a transaction which committed ahead of it*. It's hard
* to get that into a name of a reasonable length.
*/
-#define SXACT_FLAG_CONFLICT_O
Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific
On 21 June 2011 11:11, Alvaro Herrera wrote: > I realize I took out most of the fun of this patch from you, but -- are > you still planning to do some more exhaustive testing of it? I checked > some funny scenarios (including include files and groups) but it's not > all that unlikely that I missed some cases. I also didn't check any > auth method other than ident, md5 and trust, 'cause I don't have what's > required for anything else. (It's a pity that the regression tests > don't exercise anything else.) I've pretty much tested all the things I can think to test, and I'm in the same boat as you re auth methods. If there are bugs, I think they will be of the obscure kind, that only reveal themselves under peculiar circumstances. Cheers, BJ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific
Excerpts from Brendan Jurd's message of lun jun 20 20:06:39 -0400 2011: > On 21 June 2011 06:06, Alvaro Herrera wrote: > > Excerpts from Alvaro Herrera's message of lun jun 20 12:19:37 -0400 2011: > >> Excerpts from Pavel Stehule's message of lun jun 20 11:34:25 -0400 2011: > >> > >> > b) probably you can simplify a memory management using own two > >> > persistent memory context - and you can swap it. Then functions like > >> > free_hba_record, clean_hba_list, free_lines should be removed. > >> > >> Yeah, I reworked the patch with something like that over the weekend. > >> Not all of it though. I'll send an updated version shortly. > > > > Here it is, please let me know what you think. I took the liberty of > > cleaning up some things that were clearly historical leftovers. > > > > Okay, yeah, the MemoryContext approach is far more elegant than what I > had. Boy was I ever barking up the wrong tree. Eh, whoever wrote the original code was barking up the same tree, so I don't blame you for following the well-trodden path. I realize I took out most of the fun of this patch from you, but -- are you still planning to do some more exhaustive testing of it? I checked some funny scenarios (including include files and groups) but it's not all that unlikely that I missed some cases. I also didn't check any auth method other than ident, md5 and trust, 'cause I don't have what's required for anything else. (It's a pity that the regression tests don't exercise anything else.) -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific
On 21 June 2011 06:06, Alvaro Herrera wrote: > Excerpts from Alvaro Herrera's message of lun jun 20 12:19:37 -0400 2011: >> Excerpts from Pavel Stehule's message of lun jun 20 11:34:25 -0400 2011: >> >> > b) probably you can simplify a memory management using own two >> > persistent memory context - and you can swap it. Then functions like >> > free_hba_record, clean_hba_list, free_lines should be removed. >> >> Yeah, I reworked the patch with something like that over the weekend. >> Not all of it though. I'll send an updated version shortly. > > Here it is, please let me know what you think. I took the liberty of > cleaning up some things that were clearly historical leftovers. > Okay, yeah, the MemoryContext approach is far more elegant than what I had. Boy was I ever barking up the wrong tree. Cheers, BJ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe
On Mon, Jun 20, 2011 at 11:55 PM, Simon Riggs wrote: > I agree we shouldn't do anything about the name lookups for 9.1 > That is SearchCatCache using RELNAMENSP lookups, to be precise, as > well as triggers and few other similar call types. Name lookups give ERRORs that look like this... relation "pgbench_accounts" does not exist >> The ALTER TABLE patch >> has greatly expanded the scope of the issue, and that *is* a regression >> compared to prior releases. > > I agree the scope for RELOID errors increased with my 9.1 patch. Which originally generated ERRORs like cache lookup failed for relation 16390 or could not open relation with OID 16390 which should now be absent from the server log. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe
On Mon, Jun 20, 2011 at 10:56 PM, Tom Lane wrote: > Robert Haas writes: >> On Sun, Jun 19, 2011 at 5:13 PM, Simon Riggs wrote: >>> We scan pg_class in two ways: to rebuild a relcache entry based on a >>> relation's oid (easy fix). We also scan pg_class to resolve the name >>> to oid mapping. The name to oid mapping is performed *without* a lock >>> on the relation, since we don't know which relation to lock. So the >>> name lookup can fail if we are in the middle of a pg_class update. >>> This is an existing potential bug in Postgres unrelated to my patch. > >> If this is a pre-existing bug, then it's not clear to me why we need >> to do anything about it at all right now. > > Yeah. This behavior has been there since day zero, and there have been > very few complaints about it. But note that there's only a risk for > pg_class updates, not any other catalog, and there is exactly one kind > of failure with very predictable consequences. I agree we shouldn't do anything about the name lookups for 9.1 That is SearchCatCache using RELNAMENSP lookups, to be precise, as well as triggers and few other similar call types. > The ALTER TABLE patch > has greatly expanded the scope of the issue, and that *is* a regression > compared to prior releases. I agree the scope for RELOID errors increased with my 9.1 patch. I'm now happy with the locking patch (attached), which significantly reduces the scope - back to the original error scope, in my testing. I tried to solve both, but I think that's a step too far given the timing. It seems likely that there will be objections to this patch. All I would say is that issuing a stream of ALTER TABLEs against the same table is not a common situation; if it were we would have seen more of the pre-existing bug. ALTER TABLE command encompasses many subcommands and we should evaluate each subcommand differently when we decide what to do. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services relation_definition_lock.v3.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe
Robert Haas writes: > On Sun, Jun 19, 2011 at 5:13 PM, Simon Riggs wrote: >> We scan pg_class in two ways: to rebuild a relcache entry based on a >> relation's oid (easy fix). We also scan pg_class to resolve the name >> to oid mapping. The name to oid mapping is performed *without* a lock >> on the relation, since we don't know which relation to lock. So the >> name lookup can fail if we are in the middle of a pg_class update. >> This is an existing potential bug in Postgres unrelated to my patch. > If this is a pre-existing bug, then it's not clear to me why we need > to do anything about it at all right now. Yeah. This behavior has been there since day zero, and there have been very few complaints about it. But note that there's only a risk for pg_class updates, not any other catalog, and there is exactly one kind of failure with very predictable consequences. The ALTER TABLE patch has greatly expanded the scope of the issue, and that *is* a regression compared to prior releases. BTW, it seems to me that this issue is closely related to what Noah is trying to fix here: http://archives.postgresql.org/message-id/20110612191843.gf21...@tornado.leadboat.com regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Table Partitioning
Folks, I noticed that we have some nice new speed optimizations (more properly, de-pessimizations) for partitioned tables in 9.1. Anybody care to look over the table partitioning stuff on the wiki and check it for relevance? http://wiki.postgresql.org/wiki/Table_partitioning I think I may be able to get my employer to put some people on this... Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.
On Mon, Jun 20, 2011 at 22:44, Tom Lane wrote: > Magnus Hagander writes: >> On Mon, Jun 20, 2011 at 22:20, Peter Eisentraut wrote: >>> A better way might be that translators simply work on a clone of the >>> source repository, which is then merged (as in, git merge) at release >>> time. There are some issues with that to figure out, but it sounds like >>> the obviously right thing, from an interface point of view. > >> I don't think we want to track every single translation update as >> commits in the main repository - we don't do that for non-translation >> stuff... If it's a squash-merge, that's a different thing, of >> course... > >> Other than that, yes, keeping translations in git branches seems like >> a good interface. > > My recollection is that the current setup was created mainly so that > translators wouldn't need to be given commit privileges on the main > repo. Giving them a separate repo to work in might be all right, but > of course whoever does the merges would have to be careful to only > accept changes made to the .po files and not anything else. That should be easy enough to script - have the system automatically merge the branches requied with "--squash", and then make sure that no non-.po files are modified. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.
Excerpts from Tom Lane's message of lun jun 20 16:44:20 -0400 2011: > Magnus Hagander writes: > > On Mon, Jun 20, 2011 at 22:20, Peter Eisentraut wrote: > >> A better way might be that translators simply work on a clone of the > >> source repository, which is then merged (as in, git merge) at release > >> time. There are some issues with that to figure out, but it sounds like > >> the obviously right thing, from an interface point of view. > > > I don't think we want to track every single translation update as > > commits in the main repository - we don't do that for non-translation > > stuff... If it's a squash-merge, that's a different thing, of > > course... > > > Other than that, yes, keeping translations in git branches seems like > > a good interface. > > My recollection is that the current setup was created mainly so that > translators wouldn't need to be given commit privileges on the main > repo. Yep. > Giving them a separate repo to work in might be all right, but > of course whoever does the merges would have to be careful to only > accept changes made to the .po files and not anything else. Honestly it doesn't seem all that good of an idea to me. We would have to merge changes from upstream PG repo into pgtranslation and the history would look awful on the translation side. I prefer something similar to the current system, where the two repos are kept separate and the files from pgtranslation are imported wholesale before each release, without any kind of merge. That keeps both histories clean. Maybe we could have a way to prevent commits to the .po files that don't come from the pgtranslation repository; probably we could have something with Git hooks for this (so you have to set an environment variable before being allowed the push, which wouldn't occur accidentally, or something like that.) (I admit I don't look that frequently into pgtranslation history, though.) -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUG] Denormal float values break backup/restore
On 2011-06-20 19:22, Marti Raudsepp wrote: AIUI that is defined to be a little vague, but includes denormalized numbers that would undergo any rounding at all. It says that on overflow the conversion should return the appropriate HUGE_VAL variant, and set ERANGE. On underflow it returns a reasonably appropriate value (and either may or must set ERANGE, which is the part that isn't clear to me). Which standard is that? Does IEEE 754 itself define strtod() or is there another relevant standard? Urr. No, this is C and/or Unix standards, not IEEE 754. I did some more research into this. The postgres docs do specify the range error, but seemingly based on a different interpretation of underflow than what I found in some of the instances of strtod() documentation: Numbers too close to zero that are not representable as distinct from zero will cause an underflow error. This talks about denormals that get _all_ their significant digits rounded away, but some of the documents I saw specify an underflow for denormals that get _any_ of their significant digits rounded away (and thus have an abnormally high relative rounding error). The latter would happen for any number that is small enough to be denormal, and is also not representable (note: that's not the same thing as "not representable as distinct from zero"!). It's easy to get non-representable numbers when dumping binary floats in a decimal format. For instance 0.1 is not representable, nor are 0.2, 0.01, and so on. The inherent rounding of non-representable values produces weirdness like 0.1 + 0.2 - 0.3 != 0. I made a quick round of the strtod() specifications I could find, and they seem to disagree wildly: SourceERANGE when Return what - PostgreSQL docs All digits lost zero Linux programmer's manual All digits lost zero My GNU/Linux strtod() Any digits lost rounded number SunOS 5 Any digits lost rounded number GNU documentation All digits lost zero IEEE 1003.1 (Open Group 2004) Any digits lost denormal JTC1/SC22/WG14 N794 Any digits lost denormal Sun Studio (C99)Implementation-defined ? ISO/IEC 9899:TC2Implementation-defined denormal C99 Draft N869 (1999) Implementation-defined denormal We can't guarantee very much, then. It looks like C99 disagrees with the postgres interpretation, but also leaves a lot up to the compiler. I've got a few ideas for solving this, but none of them are very good: (a) Ignore underflow errors. This could hurt anyone who relies on knowing their floating-point implementation and the underflow error to keep their rounding errors in check. It also leaves a kind of gap in the predictability of the database's floating-point behaviour. Worst hit, or possibly the only real problem, would be algorithms that divide other numbers, small enough not to produce infinities, by rounded denormals. (b) Dump REAL and DOUBLE PRECISION in hex. With this change, the representation problem goes away and ERANGE would reliably mean "this was written in a precision that I can't reproduce." We could sensibly provide an option to ignore that error for cross-platform dump/restores. This trick does raise a bunch of compatibility concerns: it's a new format of data to restore, it may not work on pre-C99 compilers, and so on. Also, output for human consumption would have to differ from pg_dump output. (c) Have pg_dump produce calculations, not literals, for denormals. Did I mention how these were not great ideas? If your database dump contains 1e-308, pg_dump could recognize that this value can be calculated in the database but possibly not entered directly, and dump e.g. "1e-307::float / 10" instead. (d) Make pg_dump set some "ignore underflows" option. This may make dumps unusable for older postgres versions. Moreover, it doesn't help ORMs and applications that are currently unable to store the "problem numbers." (e) Do what the documentation promises. Actually I have no idea how we could guarantee this. (f) Ignore ERANGE unless strtod() returns ±0 or ±HUGE_VAL. This is probably a reasonable stab at common sense. It does have the nasty property that it doesn't give a full guarantee either way: restores could still break on pre-C99 systems that return 0 on underflow, but C99 doesn't guarantee a particularly accurate denormal. In practice though, implementations seem to do their best to give you the most appropriate rounded number. Jeroen -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Another issue with invalid XML values
On Mon, Jun 20, 2011 at 09:15:51PM +0200, Florian Pflug wrote: > On Jun20, 2011, at 19:57 , Noah Misch wrote: > > I tested this patch using two systems, a CentOS 5 box with > > libxml2-2.6.26-2.1.2.8.el5_5.1 and an Ubuntu 8.04 box with libxml2 > > 2.6.31.dfsg-2ubuntu1.6. Both failed to build with this error: > > > > xml.c: In function `pg_xml_init': > > xml.c:934: error: `xmlStructuredErrorContext' undeclared (first use in this > > function) > > xml.c:934: error: (Each undeclared identifier is reported only once > > xml.c:934: error: for each function it appears in.) > > make[4]: *** [xml.o] Error 1 > > > > It seems `xmlStructuredErrorContext' was added rather recently. It's not > > obvious which version added it, but 2.7.8 has it and 2.7.2 does not. > > Looking > > at 2.6.26's sources, that version seems to use `xmlGenericErrorContext' for > > both structured and generic handler functions. Based on that, I tried with > > a > > change from `xmlStructuredErrorContext' to `xmlGenericErrorContext' in our > > xml.c. I ran the test suite and hit some failures in the xml test; see > > attached regression.diffs. I received warnings where the tests expected > > nothing or expected errors. > > Great :-(. I wonder if maybe I should simply remove the part of the patch > which try to restore the error handler and context in pg_xml_done(). I've > added that because I feared that some 3rd-party libraries setup their libxml > error handle just once, not before every library class. The code previously > didn't care about this either, but since we're using structured instead of > generic error reporting now, we'd now collide with 3-rd party libs which > also use structured error handling, whereas previously that would have worked. > OTOH, once there's more than one such library... > > Whats your stand on this? Assuming it's sufficient to save/restore xmlStructuredErrorContext when it's available and xmlGenericErrorContext otherwise, I would just add an Autoconf check to identify which one to use. > > I couldn't reconcile this description with the code. I do see that > > xpath_internal() uses XML_PURPOSE_OTHER, but so does xmlelement(). Why is > > xmlelement() also ready for the strictest error reporting? > > xmlelement() doesn't use libxml's parser, only the xml writer. I didn't > want to suppress any errors the writer might raise (I'm not sure it raises > error at all, though). Makes sense. > > Also note that we may be able to be more strict when xmloption = document. > > Yeah, probably. I think that better done in a separate patch, though - I've > tried not to change existing behaviour with this patch except where absolutely > necessary (i.e. for XPATH) Likewise. > > Ideally, an invalid namespace URI should always be an error. While > > namespace > > prefixes could become valid as you assemble a larger document, a namespace > > URI's validity is context-free. > > I agree. That, however, would require xmlelement() to check all xmlns* > attributes, and I didn't find a way to do that with libxml() (Well, > other than to parse the element after creating it, but that's a huge > kludge). So I left that issue for a later time... Likewise. > > Remembering to put a pg_xml_done() in every relevant elog(ERROR, ...) path > > seems fragile. How about using RegisterXactCallback/RegisterSubXactCallback > > in pgxml_parser_init() to handle those cases? You can also use it to Assert > > at non-abort transaction shutdown that pg_xml_done() was called as needed. > > Hm, isn't that a bit fragile too? It seems that PG_TRY/PG_CATCH blocks don't > necessarily create sub-transactions, do they? PG_TRY never creates a subtransaction. However, elog(ERROR, ...) aborts either the subtransaction, or if none, the top-level transaction. Therefore, registering the same callback with both RegisterXactCallback and RegisterSubXactCallback ensures that it gets called for any elog/ereport non-local exit. Then you only explicitly call pg_xml_done() in the non-error path. However, I see now that in the core xml.c, every error-condition pg_xml_done() appears in a PG_CATCH block. That's plenty reliable ... > Also, most elog() paths already contained xmlFree() calls, because libxml > seemingly cannot be made to use context-based memory management. So one > already needs to be pretty careful when touching these... ... and this is a good reason to keep doing it that way. So, scratch that idea. > > Consider renaming XML_REPORT_ERROR it to something like XML_CHECK_ERROR, > > emphasizing that it wraps a conditional. > > Hm, I think it was named XML_CHECK_ERROR at one point, and I changed it > because I felt it didn't convey the fact that it's actually ereport() > and not just return false on error. But I agree that XML_REPORT_ERROR isn't > very self-explanatory, either. What about XML_REPORT_PENDING_ERROR() > or XML_CHECK_AND_REPORT_ERROR()? XML_CHECK_AND_REPORT_ERROR() seems fine. Or XML_CHECK_AND_EREPORT()? > > On Thu, J
Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.
Magnus Hagander writes: > On Mon, Jun 20, 2011 at 22:20, Peter Eisentraut wrote: >> A better way might be that translators simply work on a clone of the >> source repository, which is then merged (as in, git merge) at release >> time. There are some issues with that to figure out, but it sounds like >> the obviously right thing, from an interface point of view. > I don't think we want to track every single translation update as > commits in the main repository - we don't do that for non-translation > stuff... If it's a squash-merge, that's a different thing, of > course... > Other than that, yes, keeping translations in git branches seems like > a good interface. My recollection is that the current setup was created mainly so that translators wouldn't need to be given commit privileges on the main repo. Giving them a separate repo to work in might be all right, but of course whoever does the merges would have to be careful to only accept changes made to the .po files and not anything else. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Don't use "cp -i" in the example WAL archive_command.
Fujii Masao writes: > Anyway, you seem to have forgotten to fix the example of archive_command > in "24.3.5.1. Standalone Hot Backups". > archive_command = 'test ! -f /var/lib/pgsql/backup_in_progress || > cp -i %p /var/lib/pgsql/archive/%f < /dev/null' You're right, I didn't see that one. Fixed, thanks. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.
On Mon, Jun 20, 2011 at 22:20, Peter Eisentraut wrote: > On mån, 2011-06-20 at 13:13 -0400, Alvaro Herrera wrote: >> Excerpts from Michael Meskes's message of lun jun 20 09:54:36 -0400 2011: >> > On Mon, Jun 20, 2011 at 09:15:52AM -0400, Robert Haas wrote: >> > > Yep. Peter overrides them just before each release. >> > >> > Aren't there better ways to implement this, like git submodules? This >> > redundancy seem awkward to me. >> >> There might be, but currently the translations are still using the CVS >> machinery on pgfoundry. This stuff predates our move to Git. It's >> possible that Peter already changed the msgstr in pgtranslation ... >> >> Peter is working on moving that CVS stuff to Git, but AFAIR it will >> happen once 9.1 has released. > > A better way might be that translators simply work on a clone of the > source repository, which is then merged (as in, git merge) at release > time. There are some issues with that to figure out, but it sounds like > the obviously right thing, from an interface point of view. I don't think we want to track every single translation update as commits in the main repository - we don't do that for non-translation stuff... If it's a squash-merge, that's a different thing, of course... Other than that, yes, keeping translations in git branches seems like a good interface. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.
On mån, 2011-06-20 at 13:13 -0400, Alvaro Herrera wrote: > Excerpts from Michael Meskes's message of lun jun 20 09:54:36 -0400 2011: > > On Mon, Jun 20, 2011 at 09:15:52AM -0400, Robert Haas wrote: > > > Yep. Peter overrides them just before each release. > > > > Aren't there better ways to implement this, like git submodules? This > > redundancy seem awkward to me. > > There might be, but currently the translations are still using the CVS > machinery on pgfoundry. This stuff predates our move to Git. It's > possible that Peter already changed the msgstr in pgtranslation ... > > Peter is working on moving that CVS stuff to Git, but AFAIR it will > happen once 9.1 has released. A better way might be that translators simply work on a clone of the source repository, which is then merged (as in, git merge) at release time. There are some issues with that to figure out, but it sounds like the obviously right thing, from an interface point of view. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FOR KEY LOCK foreign keys
On Sun, Jun 19, 2011 at 06:30:41PM +0200, Jesper Krogh wrote: > I hope this hasn't been forgotten. But I cant see it has been committed > or moved > into the commitfest process? If you're asking about that main patch for $SUBJECT rather than those isolationtester changes specifically, I can't speak to the plans for it. I wasn't planning to move the test suite work forward independent of the core patch it serves, but we could do that if there's another application. Thanks, nm > On 2011-03-11 16:51, Noah Misch wrote: >> On Fri, Feb 11, 2011 at 02:13:22AM -0500, Noah Misch wrote: >>> Automated tests would go a long way toward building confidence that this >>> patch >>> does the right thing. Thanks to the SSI patch, we now have an in-tree test >>> framework for testing interleaved transactions. The only thing it needs to >>> be >>> suitable for this work is a way to handle blocked commands. If you like, I >>> can >>> try to whip something up for that. >> [off-list ACK followed] >> >> Here's a patch implementing that. It applies to master, with or without your >> KEY LOCK patch also applied, though the expected outputs reflect the >> improvements from your patch. I add three isolation test specs: >> >>fk-contention: blocking-only test case from your blog post >>fk-deadlock: the deadlocking test case I used during patch review >>fk-deadlock2: Joel Jacobson's deadlocking test case -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] REMINDER: Participation Requested: Survey about Open-Source Software Development
Hi, Apologies for any inconvenience and thank you to those who have already completed the survey. We will keep the survey open for another couple of weeks. But, we do hope you will consider responding to the email request below (sent 2 weeks ago). Thanks, Dr. Jeffrey Carver Assistant Professor University of Alabama (v) 205-348-9829 (f) 205-348-0219 http://www.cs.ua.edu/~carver -Original Message- From: Jeffrey Carver [mailto:opensourcesur...@cs.ua.edu] Sent: Monday, June 13, 2011 11:58 AM To: 'pgsql-hackers@postgresql.org' Subject: Participation Requested: Survey about Open-Source Software Development Hi, Drs. Jeffrey Carver, Rosanna Guadagno, Debra McCallum, and Mr. Amiangshu Bosu, University of Alabama, and Dr. Lorin Hochstein, University of Southern California, are conducting a survey of open-source software developers. This survey seeks to understand how developers on distributed, virtual teams, like open-source projects, interact with each other to accomplish their tasks. You must be at least 19 years of age to complete the survey. The survey should take approximately 15 minutes to complete. If you are actively participating as a developer, please consider completing our survey. Here is the link to the survey: http://goo.gl/HQnux We apologize for inconvenience and if you receive multiple copies of this email. This survey has been approved by The University of Alabama IRB board. Thanks, Dr. Jeffrey Carver Assistant Professor University of Alabama (v) 205-348-9829 (f) 205-348-0219 http://www.cs.ua.edu/~carver -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types and extensions
Florian Pflug wrote: On Jun20, 2011, at 20:58 , Tom Lane wrote: Darren Duncan writes: I still think that the most elegant solution is for stuff like collation to just be built-in to the base types that the range is ranging over, meaning we have a separate text base type for each text collation, and the text operators are polymorphic over all those base types. Having collations and stuff as something off to the side not built-in to text/etc types is the root of the problem. I tend to agree that this aspect of the SQL standard isn't terribly well designed, but it's the standard and we're stuck with it. We're not going to support two parallel methods of dealing with collations. Plus, you can always define a DOMAIN for every collation you intent to use, and stay clear of COLLATE clauses except as part of these domain definitions. Most interestingly, this is also the workaround Jeff Davis suggested for those who absolutely need two range types over the same base type (i.e. define one of the ranges over a domain). best regards, Florian Pflug That DOMAIN-based solution ostensibly sounds like a good one then, under the circumstances. What I *don't* want to see is for things like ranges to have their own collations and the like. From the perspective of all range-specific things, the types over which they're defined like text should just have their own native ordering, which defines the range's sense of "before" and "after". If DOMAIN effectively does that for text types, then that is the way to go. -- Darren Duncan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch for 9.2: enhanced errors
Hello
I am sending a updated patch
>
> Coding Review
> -
>
>
> In tupdesc.c
>
> line 202 the existing code is performing a deep copy of ConstrCheck. Do you
> need to copy nkeys and conkey here as well?
>
> Then at line 250 ccname is freed but not conkey
>
fixed
>
> postgres_ext.h line 55
> + #define PG_DIAG_SCHEMA_NAME 's'
> + #define PG_DIAG_TABLE_NAME 't'
> + #define PG_DIAG_COLUMN_NAMES 'c'
> + #define PG_DIAG_CONSTRAINT_NAME 'n'
>
> The assignment of letters to parameters seems arbitrary to me, I don't have
> a different non-arbitrary mapping in mind but if anyone else does they
> should speak up. I think it will be difficult to change this after 9.2 goes
> out.
>
>
> elog.c:
> ***
> *** 2197,2202
> --- 2299,2319
> if (application_name)
> appendCSVLiteral(&buf, application_name);
>
> + /* constraint_name */
> + appendCSVLiteral(&buf, edata->constraint_name);
> + appendStringInfoChar(&buf, ',');
> +
> + /* schema name */
> + appendCSVLiteral(&buf, edata->schema_name);
> + appendStringInfoChar(&buf, ',');
>
> You need to update config.sgml at the same time you update this format.
> You need to append a "," after application name but before constraintName.
> As it stands the CSV log has something like:
> .nbtinsert.c:433","psql""a_pkey","public","a","a"
fixed
>
>
> nbtinsert.c
>
> pg_get_indrelation is named differently than everything else in this file
> (ie _bt...). My guess is that this function belongs somewhere else but I
> don't know the code well enough to say where you should move it too.
>
I renamed this function to IndexRelationGetParentRelation and muved to
relcache.c
I don't call a quote_identifier on only data error properties like
table_name or schema_name (but I am open to arguments for it or
against it). The quote_identifier is used for column names, because
there should be a more names and comma should be used inside name -
and this is consistent with pg_get_indexdef_columns.
Regards
Pavel Stehule
*** ./doc/src/sgml/config.sgml.orig 2011-06-20 18:08:39.0 +0200
--- ./doc/src/sgml/config.sgml 2011-06-20 21:12:31.688165497 +0200
***
*** 3919,3927
user query that led to the error (if any and enabled by
log_min_error_statement),
character count of the error position therein,
! location of the error in the PostgreSQL source code
(if log_error_verbosity is set to verbose),
! and application name.
Here is a sample table definition for storing CSV-format log output:
--- 3919,3933
user query that led to the error (if any and enabled by
log_min_error_statement),
character count of the error position therein,
! location of the error in the PostgreSQL source code,
(if log_error_verbosity is set to verbose),
! application name,
! (following fields to end are filled when log_error_verbosity is
! set to verbose)
! constraint name,
! schema name,
! table name,
! and column names.
Here is a sample table definition for storing CSV-format log output:
***
*** 3950,3955
--- 3956,3965
query_pos integer,
location text,
application_name text,
+ constraint_name text,
+ schema_name text,
+ table_name text,
+ column_names text,
PRIMARY KEY (session_id, session_line_num)
);
*** ./src/backend/access/common/tupdesc.c.orig 2011-06-20 18:08:39.0 +0200
--- ./src/backend/access/common/tupdesc.c 2011-06-20 20:30:15.477883102 +0200
***
*** 200,205
--- 200,217
cpy->check[i].ccname = pstrdup(constr->check[i].ccname);
if (constr->check[i].ccbin)
cpy->check[i].ccbin = pstrdup(constr->check[i].ccbin);
+ if (constr->check[i].nkeys > 0)
+ {
+ cpy->check[i].conkey = palloc(sizeof(int16) * constr->check[i].nkeys);
+ memcpy(cpy->check[i].conkey, constr->check[i].conkey,
+ sizeof(int16) * constr->check[i].nkeys);
+ cpy->check[i].nkeys = constr->check[i].nkeys;
+ }
+ else
+ {
+ cpy->check[i].conkey = NULL;
+ constr->check[i].nkeys = 0;
+ }
}
}
***
*** 249,254
--- 261,268
pfree(check[i].ccname);
if (check[i].ccbin)
pfree(check[i].ccbin);
+ if (check[i].conkey)
+ pfree(check[i].conkey);
}
pfree(check);
}
***
*** 409,414
--- 423,431
* Similarly, don't assume that the checks are always read in the
* same order; match them up by name and contents. (The name
* *should* be unique, but...)
+ *
+ * nkeys and conkey depends on ccbin, and there are not neccessary
+ * to compare it.
*/
for (j = 0; j < n; check2++, j++)
{
*** ./src/backend/access/nbtree/nbtinsert.c.orig 2011-06-20 18:08:39.0 +0200
--- ./src/backend/access/nb
Re: [HACKERS] Range Types and extensions
On Jun20, 2011, at 20:58 , Tom Lane wrote: > Darren Duncan writes: >> I still think that the most elegant solution is for stuff like collation to >> just >> be built-in to the base types that the range is ranging over, meaning we >> have a >> separate text base type for each text collation, and the text operators are >> polymorphic over all those base types. Having collations and stuff as >> something >> off to the side not built-in to text/etc types is the root of the >> problem. > > I tend to agree that this aspect of the SQL standard isn't terribly well > designed, but it's the standard and we're stuck with it. We're not > going to support two parallel methods of dealing with collations. Plus, you can always define a DOMAIN for every collation you intent to use, and stay clear of COLLATE clauses except as part of these domain definitions. Most interestingly, this is also the workaround Jeff Davis suggested for those who absolutely need two range types over the same base type (i.e. define one of the ranges over a domain). best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Another issue with invalid XML values
Hi Thanks for the extensive review, it's very much appreciated! On Jun20, 2011, at 19:57 , Noah Misch wrote: > I tested this patch using two systems, a CentOS 5 box with > libxml2-2.6.26-2.1.2.8.el5_5.1 and an Ubuntu 8.04 box with libxml2 > 2.6.31.dfsg-2ubuntu1.6. Both failed to build with this error: > > xml.c: In function `pg_xml_init': > xml.c:934: error: `xmlStructuredErrorContext' undeclared (first use in this > function) > xml.c:934: error: (Each undeclared identifier is reported only once > xml.c:934: error: for each function it appears in.) > make[4]: *** [xml.o] Error 1 > > It seems `xmlStructuredErrorContext' was added rather recently. It's not > obvious which version added it, but 2.7.8 has it and 2.7.2 does not. Looking > at 2.6.26's sources, that version seems to use `xmlGenericErrorContext' for > both structured and generic handler functions. Based on that, I tried with a > change from `xmlStructuredErrorContext' to `xmlGenericErrorContext' in our > xml.c. I ran the test suite and hit some failures in the xml test; see > attached regression.diffs. I received warnings where the tests expected > nothing or expected errors. Great :-(. I wonder if maybe I should simply remove the part of the patch which try to restore the error handler and context in pg_xml_done(). I've added that because I feared that some 3rd-party libraries setup their libxml error handle just once, not before every library class. The code previously didn't care about this either, but since we're using structured instead of generic error reporting now, we'd now collide with 3-rd party libs which also use structured error handling, whereas previously that would have worked. OTOH, once there's more than one such library... Whats your stand on this? > Looks like my version of libxml2 (2.6.26 in this > case) classifies some of these messages differently. Hm, that's exactly what I hoped that this patch would *prevent* from happening.. Will look into this. > On Thu, Jun 09, 2011 at 06:11:46PM +0200, Florian Pflug wrote: >> On Jun2, 2011, at 01:34 , Florian Pflug wrote: >> I also realized that some libxml error (like undefined namespace prefixes) >> must be ignored during xmlparse() and friends. Otherwise, it becomes >> impossible to build XML documents from individual fragments. pg_xml_init() >> therefore now takes an argument which specifies how strict the error >> checking is supposed to be. For the moment, only XPATH() uses the strict >> mode in which we report all errors. XMLPARSE() and friends only report >> parse errors, not namespace errors. > > Seems fair offhand. We must preserve the invariant that any object with type > "xml" can be passed through xml_out() and then be accepted by xml_in(). I'm > not seeing any specific risks there, but I figure I'd mention it. > > I couldn't reconcile this description with the code. I do see that > xpath_internal() uses XML_PURPOSE_OTHER, but so does xmlelement(). Why is > xmlelement() also ready for the strictest error reporting? xmlelement() doesn't use libxml's parser, only the xml writer. I didn't want to suppress any errors the writer might raise (I'm not sure it raises error at all, though). > Also note that we may be able to be more strict when xmloption = document. Yeah, probably. I think that better done in a separate patch, though - I've tried not to change existing behaviour with this patch except where absolutely necessary (i.e. for XPATH) > Ideally, an invalid namespace URI should always be an error. While namespace > prefixes could become valid as you assemble a larger document, a namespace > URI's validity is context-free. I agree. That, however, would require xmlelement() to check all xmlns* attributes, and I didn't find a way to do that with libxml() (Well, other than to parse the element after creating it, but that's a huge kludge). So I left that issue for a later time... > The patch adds some trailing whitespace. Ups, sorry for that. Will fix. > Remembering to put a pg_xml_done() in every relevant elog(ERROR, ...) path > seems fragile. How about using RegisterXactCallback/RegisterSubXactCallback > in pgxml_parser_init() to handle those cases? You can also use it to Assert > at non-abort transaction shutdown that pg_xml_done() was called as needed. Hm, isn't that a bit fragile too? It seems that PG_TRY/PG_CATCH blocks don't necessarily create sub-transactions, do they? Also, most elog() paths already contained xmlFree() calls, because libxml seemingly cannot be made to use context-based memory management. So one already needs to be pretty careful when touching these... Anyway, should we decide to give up on trying to restore the error handler and context, we could get rid of pg_xml_done() again... (See above for the details on that) > Consider renaming XML_REPORT_ERROR it to something like XML_CHECK_ERROR, > emphasizing that it wraps a conditional. Hm, I think it was named XML_CHECK_ERROR at one point, and I changed it because I f
Re: [HACKERS] [WIP] cache estimates, cache access cost
Greg Smith wrote: > The idea that any of this will run automatically is a dream at > this point, so saying you want to automatically recover from > problems with the mechanism that doesn't even exist yet is a bit > premature. Well, I certainly didn't mean it to be a reason not to move forward with development -- I wouldn't have raised the issue had you not said this upthread: > I don't see how sequential scan vs. index costing will be any > different on a fresh system than it is now. All I was saying is: I do; here's how... Carry on. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types and extensions
Darren Duncan writes: > I still think that the most elegant solution is for stuff like collation to > just > be built-in to the base types that the range is ranging over, meaning we have > a > separate text base type for each text collation, and the text operators are > polymorphic over all those base types. Having collations and stuff as > something > off to the side not built-in to text/etc types is the root of the > problem. I tend to agree that this aspect of the SQL standard isn't terribly well designed, but it's the standard and we're stuck with it. We're not going to support two parallel methods of dealing with collations. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [WIP] cache estimates, cache access cost
Kevin Grittner wrote: But its not hard to imagine an application mix where this feature could cause a surprising ten-fold performance drop after someone does a table scan which could persist indefinitely. I'm not risking that in production without a clear mechanism to automatically recover from that sort of cache skew The idea that any of this will run automatically is a dream at this point, so saying you want to automatically recover from problems with the mechanism that doesn't even exist yet is a bit premature. Some of the implementation ideas here might eventually lead to where real-time cache information is used, and that is where the really scary feedback loops you are right to be worried about come into play. The idea for now is that you'll run this new type of ANALYZE CACHE operation manually, supervised and at a time where recent activity reflects the sort of workload you want to optimize for. And then you should review its results to make sure the conclusions it drew about your cache population aren't really strange. To help with that, I was thinking of writing a sanity check tool that showed how the cached percentages this discovers compare against the historical block hit percentages for the relation. An example of how values changed from what they were already set to after a second ANALYZE CACHE is probably useful too. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgbench--new transaction type
Itagaki Takahiro wrote: Anyway, I'm not sure we need to include the query mode into the pgbench's codes. Instead, how about providing "a sample script" as a separate sql file? pgbench can execute any script files with -f option. When you execute using "-f", it doesn't correctly detect database scale. Also, the really valuable thing here is seeing the higher selects/second number come out in the report. I just realized neither Jeff nor myself ever included an example of the output in the new mode, which helps explain some of why the patch is built the way it is: $ pgbench -c 12 -j 4 -T 30 -P pgbench plgsql function created. starting vacuum...end. transaction type: SELECT only via plpgsql scaling factor: 100 query mode: simple number of clients: 12 number of threads: 4 duration: 30 s number of transactions actually processed: 9342 tps = 311.056293 (including connections establishing) tps = 311.117886 (excluding connections establishing) selects per second = 159260.822100 (including connections establishing) selects per second = 159292.357672 (excluding connections establishing) -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types and extensions
Tom Lane wrote: Florian Pflug writes: On Jun20, 2011, at 19:16 , Merlin Moncure wrote: On Mon, Jun 20, 2011 at 11:21 AM, Jeff Davis wrote: hm, what if there *was( only one range type per base type, but in the various contexts where specific ordering and collation was important you could optionally pass them in? Meaning, the specific ordering was not bound rigidly to the type, but to the operation? I suggested that previously here http://archives.postgresql.org/pgsql-hackers/2011-06/msg00846.php In the ensuing discussion, however, it became clear that by doing so range types become little more than a pair of values. More specifically, a range then *doesn't* represent a set of values, because whether or not a value is "in" the range depends on a specific sort order. Yeah, that doesn't seem like the way to go. If a range value doesn't represent a well-defined set of base-type values, we lose a lot of the mathematical underpinnings for range operations. So ... just how awful would it be if we hard-wired range types to always use their base type's default btree sort ordering and the database's default collation? In principle that sucks, but I'm not sure how wide the use-cases actually will be for other choices. The other viable alternative seems to be to require those two properties (btree opclass and collation) to be part of a specific range type definition. The complaint about that seemed to be that we couldn't infer an ANYRANGE type given only ANYELEMENT, but could we alleviate that by identifying one range type as the default for the base type, and then using that one in cases where we have no ANYRANGE input? regards, tom lane I still think that the most elegant solution is for stuff like collation to just be built-in to the base types that the range is ranging over, meaning we have a separate text base type for each text collation, and the text operators are polymorphic over all those base types. Having collations and stuff as something off to the side not built-in to text/etc types is the root of the problem. The range-specific stuff can remain ANYELEMENT and no special-casing is required. Also, besides range constructors, a generic membership test like "value in range" is polymorphic. -- Darren Duncan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Another issue with invalid XML values
Hi Florian, I tested this patch using two systems, a CentOS 5 box with libxml2-2.6.26-2.1.2.8.el5_5.1 and an Ubuntu 8.04 box with libxml2 2.6.31.dfsg-2ubuntu1.6. Both failed to build with this error: xml.c: In function `pg_xml_init': xml.c:934: error: `xmlStructuredErrorContext' undeclared (first use in this function) xml.c:934: error: (Each undeclared identifier is reported only once xml.c:934: error: for each function it appears in.) make[4]: *** [xml.o] Error 1 It seems `xmlStructuredErrorContext' was added rather recently. It's not obvious which version added it, but 2.7.8 has it and 2.7.2 does not. Looking at 2.6.26's sources, that version seems to use `xmlGenericErrorContext' for both structured and generic handler functions. Based on that, I tried with a change from `xmlStructuredErrorContext' to `xmlGenericErrorContext' in our xml.c. I ran the test suite and hit some failures in the xml test; see attached regression.diffs. I received warnings where the tests expected nothing or expected errors. Looks like my version of libxml2 (2.6.26 in this case) classifies some of these messages differently. I suggest testing the next version with both libxml2 2.6.23, the minimum supported version, and libxml2 2.7.8, the latest version. On Thu, Jun 09, 2011 at 06:11:46PM +0200, Florian Pflug wrote: > On Jun2, 2011, at 01:34 , Florian Pflug wrote: > > On Jun2, 2011, at 00:02 , Noah Misch wrote: > >> On Wed, Jun 01, 2011 at 06:16:21PM +0200, Florian Pflug wrote: > >>> Anyway, I'll try to come up with a patch that replaces > >>> xmlSetGenericErrorFunc() with xmlSetStructuredErrorFunc(). > >> > >> Sounds sensible. Will this impose any new libxml2 version dependency? > > > > xmlSetStructuredErrorFunc() seems to be available starting with libxml > > 2.6.0, > > release on Oct 20, 2003. Since we already require the version to be >= > > 2.6.23, > > we should be OK. > > > > I won't have access to my PC the next few days, but I'll try to come up with > > a patch some time next week. > > Phew... I did manage to produce a patch, but it was way more work than I > had intended to put into this. > > As it turns out, you loose the nicely formatted context information that > libxml2 provides via the generic error func once you switch to structured > error reporting. Registering handlers for both doesn't help either, since > the generic error handler isn't called once you register a structured one. > > Fortunately, libxml does export xmlParserPrintFileContext() which > generates these context messages. It, however, doesn't return a string, > but instead passes them to the generic error handler (this time, independent > from whether a structural error handler is registered or not). Interesting API, there. > As it stood, the code assumed that all third-party library re-install > their libxml error handlers before each library call, and thus didn't > bother to restore the old error handler itself. Since I revamped the > error handling anyway, I removed that requirement. There is now a > function pg_xml_done() which restores the original error handler that > we overwrote in pg_xml_init(). Seems reasonable. > I also realized that some libxml error (like undefined namespace prefixes) > must be ignored during xmlparse() and friends. Otherwise, it becomes > impossible to build XML documents from individual fragments. pg_xml_init() > therefore now takes an argument which specifies how strict the error > checking is supposed to be. For the moment, only XPATH() uses the strict > mode in which we report all errors. XMLPARSE() and friends only report > parse errors, not namespace errors. Seems fair offhand. We must preserve the invariant that any object with type "xml" can be passed through xml_out() and then be accepted by xml_in(). I'm not seeing any specific risks there, but I figure I'd mention it. I couldn't reconcile this description with the code. I do see that xpath_internal() uses XML_PURPOSE_OTHER, but so does xmlelement(). Why is xmlelement() also ready for the strictest error reporting? Also note that we may be able to be more strict when xmloption = document. > > Finally, I had to adjust contrib/xml2 because it uses some parts of > the core XML support like pg_xml_init(). > > Heres the indended behaviour with the patch applied: > > > We always use structured error handling. For now, the error messages > pretty much resemble the old ones, but it's now easy to add additional > information. > > XMLPARSE() and casting to XML check for parse errors only, like they do > without the patch. They're also capable of reporting warnings, but I > didn't find a case where the libxml parser generates a warning. > > XPATH() reports all errors and warnings. Trying to use XPATH() on > a document with e.g. inconsistent namespace usage or invalid > namespace URIs therefore now raises an error. This is *necessary* > because libxml's XPath evaluator gets confus
Re: [HACKERS] Range Types and extensions
Florian Pflug writes: > On Jun20, 2011, at 19:16 , Merlin Moncure wrote: >> On Mon, Jun 20, 2011 at 11:21 AM, Jeff Davis wrote: >> hm, what if there *was( only one range type per base type, but in the >> various contexts where specific ordering and collation was important >> you could optionally pass them in? Meaning, the specific ordering was >> not bound rigidly to the type, but to the operation? > I suggested that previously here > http://archives.postgresql.org/pgsql-hackers/2011-06/msg00846.php > In the ensuing discussion, however, it became clear that by doing so > range types become little more than a pair of values. More specifically, > a range then *doesn't* represent a set of values, because whether or > not a value is "in" the range depends on a specific sort order. Yeah, that doesn't seem like the way to go. If a range value doesn't represent a well-defined set of base-type values, we lose a lot of the mathematical underpinnings for range operations. So ... just how awful would it be if we hard-wired range types to always use their base type's default btree sort ordering and the database's default collation? In principle that sucks, but I'm not sure how wide the use-cases actually will be for other choices. The other viable alternative seems to be to require those two properties (btree opclass and collation) to be part of a specific range type definition. The complaint about that seemed to be that we couldn't infer an ANYRANGE type given only ANYELEMENT, but could we alleviate that by identifying one range type as the default for the base type, and then using that one in cases where we have no ANYRANGE input? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch - Debug builds without optimization
Alvaro Herrera writes: > Excerpts from Greg Smith's message of lun jun 20 00:25:08 -0400 2011: >> The peg utility script I use makes a reinstall as simple as: >> >> stop >> peg build > But you're building the entire server there, which was Tom's point -- > you only need to build and reinstall the backend. Right, I was trying to illustrate how to have minimal turnaround time when testing a small code change. Rebuilding from scratch is slow enough that you lose focus while waiting. (Or I do, anyway.) Granted, stuff like ccache can help with that, but why not adopt a process that's not slow in the first place? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types and extensions
On Jun20, 2011, at 19:16 , Merlin Moncure wrote: > On Mon, Jun 20, 2011 at 11:21 AM, Jeff Davis wrote: > hm, what if there *was( only one range type per base type, but in the > various contexts where specific ordering and collation was important > you could optionally pass them in? Meaning, the specific ordering was > not bound rigidly to the type, but to the operation? I suggested that previously here http://archives.postgresql.org/pgsql-hackers/2011-06/msg00846.php In the ensuing discussion, however, it became clear that by doing so range types become little more than a pair of values. More specifically, a range then *doesn't* represent a set of values, because whether or not a value is "in" the range depends on a specific sort order. Actually, you'd probably even loose the possibility of having a normalization function for discrete base types (which makes sure that we know that "[1,2]" is the same as "[1,3)"), because it's hard to image one normalization function that works sensibly for two different orderings. So by doing that, you effectively turn a RANGE into a quadruple (lower type, lower_included bool, upper type, upper_included bool). best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Boolean operators without commutators vs. ALL/ANY
On Jun20, 2011, at 19:22 , Alvaro Herrera wrote: > Excerpts from Florian Pflug's message of lun jun 20 06:55:42 -0400 2011: >> The latter (i.e. regexp literals enclosed by /../) probably isn't >> desirably for postgres, but the former definitely is (i.e. distinguishing >> regexp's and text in the type system). Please see the thread >> "Adding a distinct pattern type to resolve the ~ commutator stalemate" >> for the details of the proposal. > > 'your text' ~ regexp 'your.*foo' > column ~ regexp 'your.*foo' > > So you could do > > regexp 'foo.*bar' ~ 'your text' > > and it's immediately clear what's up. > > The question is what to do wrt implicit casting of text to regexp. > If we don't, there's a backwards compatibility hit. No, we certainly musn't allow text to be implicitly converted to regexp, for otherwise e.g. "varchar ~ varchar" becomes ambiguous. I posted a primitive prototype for a pattern type on said thread, which seems to do everything we require without causing compatibility problems. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch - Debug builds without optimization
Excerpts from Greg Smith's message of lun jun 20 00:25:08 -0400 2011: > Greg Stark wrote: > > I've always wondered what other people do to iterate quickly. > > I'd have bet money you had an elisp program for this by now! Yeah :-) > The peg utility script I use makes a reinstall as simple as: > > stop > peg build But you're building the entire server there, which was Tom's point -- you only need to build and reinstall the backend. I have my own "runpg" utility which does a lot of these things too ... The main difference (to Tom's approach) is that I don't use pg_ctl to start/stop the server, because I always keep that running in a terminal, which makes for easier debugging because the logs are always there and I can ctrl-c it ... Well I guess it's pretty much the same thing, because Tom probably has a script to stop the server. -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] possible connection leak in dblink?
Joe Conway wrote: -- Start of PGP signed section. > On 06/17/2011 01:05 PM, Tom Lane wrote: > > Peter Eisentraut writes: > >> Is this a bug fix that should be backpatched? > > > > I pinged Joe Conway about this. He is jetlagged from a trip to the Far > > East but promised to take care of it soon. I think we can wait for his > > review. > > Sorry for the delay. I'm finally caught up on most of my obligations, > and have plowed through the 1500 or so pgsql mailing list messages that > I was behind. But if everyone is OK with it I would like to aim to > commit a fix mid next week, because I still have to get through my > daughter's high school graduation tomorrow, and an out of state funeral > for my father-in-law Sunday/Monday. > > That said, I really would like to commit this myself, as I have yet to > be brave enough to commit anything under git :-( Sounds good. We will be here to help. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Boolean operators without commutators vs. ALL/ANY
Excerpts from Florian Pflug's message of lun jun 20 06:55:42 -0400 2011: > The latter (i.e. regexp literals enclosed by /../) probably isn't > desirably for postgres, but the former definitely is (i.e. distinguishing > regexp's and text in the type system). Please see the thread > "Adding a distinct pattern type to resolve the ~ commutator stalemate" > for the details of the proposal. 'your text' ~ regexp 'your.*foo' column ~ regexp 'your.*foo' So you could do regexp 'foo.*bar' ~ 'your text' and it's immediately clear what's up. The question is what to do wrt implicit casting of text to regexp. If we don't, there's a backwards compatibility hit. -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] proposal: a validator for configuration files
Florian Pflug writes: > On Jun20, 2011, at 18:16 , Tom Lane wrote: >> This is already known to happen: there are cases where the postmaster >> and a backend can come to different conclusions about whether a setting >> is valid (eg, because it depends on database encoding). Whether that's >> a bug or not isn't completely clear, but if this patch is critically >> dependent on the situation never happening, I don't think we can accept >> it. > Does that mean that some backends might currently choose to ignore an > updated config file wholesale on SIGUP (because some settings are invalid) > while others happily apply it? Meaning that they'll afterwards disagree > even on modified settings which *would* be valid for both backends? Yes. I complained about that before: http://archives.postgresql.org/pgsql-hackers/2011-04/msg00330.php but we didn't come to any consensus about fixing it. This patch might be a good vehicle for revisiting the issue, though. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe
Robert Haas wrote: > I'm currently on the other end of the spectrum: ignore and > consider for 9.2. I guess part of my point was that if we can't safely get something into the initial 9.1 release, it doesn't mean we necessarily need to wait for 9.2. Bugs can be fixed along the way in minor releases. A 9.1.1 fix might give us more time to work through details and ensure that it is a safe and well-targeted fix. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types and extensions
On Mon, Jun 20, 2011 at 11:21 AM, Jeff Davis wrote: > On Mon, 2011-06-20 at 16:01 +0200, Florian Pflug wrote: >> Hm, I'm starting to wonder if there isn't a way around that. It seems that >> this restriction comes from the desire to allow functions with the >> polymorphic signature >> (ANYELEMENT, ANYELEMENT) -> ANYRANGE. >> >> The only such function I can currently come up with is the generic >> range constructor. Is having that worth the restriction to one >> range type per base type? > > Good point. > > Having constructors is obviously important, but perhaps they don't have > to be generic. We could generate catalog entries for each constructor > for each range type, and name them after the range type itself. So, > instead of: > range(1, 10) > you'd write: > int4range(1,10) > > That actually might be better anyway, because relying on the polymorphic > version is not perfect now anyway. For instance, if you want an > int8range using the generic range() constructor, you need a cast. > > We'd still need to get the polymorphic type system to work the way we > want in this case. I'll look into that. hm, what if there *was( only one range type per base type, but in the various contexts where specific ordering and collation was important you could optionally pass them in? Meaning, the specific ordering was not bound rigidly to the type, but to the operation? merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.
Excerpts from Michael Meskes's message of lun jun 20 09:54:36 -0400 2011: > On Mon, Jun 20, 2011 at 09:15:52AM -0400, Robert Haas wrote: > > Yep. Peter overrides them just before each release. > > Aren't there better ways to implement this, like git submodules? This > redundancy seem awkward to me. There might be, but currently the translations are still using the CVS machinery on pgfoundry. This stuff predates our move to Git. It's possible that Peter already changed the msgstr in pgtranslation ... Peter is working on moving that CVS stuff to Git, but AFAIR it will happen once 9.1 has released. -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.
On Mon, Jun 20, 2011 at 9:54 AM, Michael Meskes wrote: > On Mon, Jun 20, 2011 at 09:15:52AM -0400, Robert Haas wrote: >> Yep. Peter overrides them just before each release. > > Aren't there better ways to implement this, like git submodules? This > redundancy seem awkward to me. Dunno. I just work here. :-) -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe
On Mon, Jun 20, 2011 at 9:42 AM, Kevin Grittner wrote: > Simon Riggs wrote: > >> I'm looking for opinions ranging from fix-now-and-backpatch thru >> to ignore and discuss for 9.2. > > If it's a pre-existing bug I would think that one option would be to > put it into the next bug-fix release of each supported major release > in which it is manifest. Of course, if it is *safe* to work it into > 9.1, that'd be great. I'm currently on the other end of the spectrum: ignore and consider for 9.2. But that's mostly based on the belief that there isn't going to be a way of fixing this that isn't far too invasive to back-patch. Should that turn out to be incorrect, that's a different matter, of course... -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] proposal: a validator for configuration files
On Jun20, 2011, at 18:16 , Tom Lane wrote: > Florian Pflug writes: >> The code the actually implements the "check settings first, apply later" >> logic >> isn't easy to read. Now, assume that this code has a bug. Then, with your >> patch applied, we might end up with the postmaster applying a setting >> (because >> it didn't abort early) but the backend ignoring it (because they did abort >> early). >> This is obviously bad. Depending on the setting, the consequences may range >> from slightly confusing behaviour to outright crashes I guess... > > This is already known to happen: there are cases where the postmaster > and a backend can come to different conclusions about whether a setting > is valid (eg, because it depends on database encoding). Whether that's > a bug or not isn't completely clear, but if this patch is critically > dependent on the situation never happening, I don't think we can accept > it. Does that mean that some backends might currently choose to ignore an updated config file wholesale on SIGUP (because some settings are invalid) while others happily apply it? Meaning that they'll afterwards disagree even on modified settings which *would* be valid for both backends? Or do these kinds of setting rejections happen late enough to fall out of the all-or-nothing logic in ProcessConfigFile? Anyway, the patch *doesn't* depend on all backends's setting being in sync. The issue we were discussion was whether it's OK to add another small risk of them getting out of sync by aborting early on errors in backends but not in the postmaster. I was arguing against that, while Alexey was in favour of it, on the grounds that it reduces log traffic (but only at DEBUG2 or beyond). best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding a distinct "pattern" type to resolve the "~" commutator stalemate
On Jun20, 2011, at 18:28 , David E. Wheeler wrote:
> I don't suppose there's a special quoting to be had for patterns? Perhaps one
> of these (modulo SQL parsing issues);
>
>/pattern/
>{pattern}
>qr/pattern/
>qr'pattern'
>R/pattern/
>R'pattern'
Pretty daring suggestion, I must say ;-)
I think regexp's are nearly prominent enough in SQL to warrant this.
Also, the main reason why this is such a huge deal for most programming
languages is that it avoids having to double-escape backslashes.
At least with standard_conforming_strings=on, however, that isn't a problem
in SQL because backslashes in literals aren't treated specially. For example
writing
'test' ~ '^\w+$'
Just Works (TM) if standard_conforming_strings=on, whereas in C you'd
have to write
regexp_match("test", "^\\w+$")
to give the regexp engine a chance to even see the "\".
best regards,
Florian Pflug
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding a distinct "pattern" type to resolve the "~" commutator stalemate
On Jun 19, 2011, at 4:56 PM, Florian Pflug wrote:
> Hm, it seems we either all have different idea about how such
> a pattern type would be be defined, or have grown so accustomed to
> pg's type system that we've forgotten how powerful it really
> is ;-) (For me, the latter is surely true...).
>
> I've now created a primitive prototype that uses a composite
> type for "pattern". That changes the input syntax for patterns
> (you need to enclose them in brackets), but should model all
> the implicit and explicit casting rules and operator selection
> correctly. It also uses "~~~" in place of "~", for obvious
> reasons and again without changing the casting and overloading
> rules.
Ew.
> The prototype defines
> text ~~~ text
> text ~~~ pattern
> pattern ~~~ text
> and can be found at end of this mail.
>
> With that prototype, *all* the cases (even unknown ~~~ unknown)
> work as today, i.e. all of the statements below return true
Florian++ Very nice, thanks!
I don't suppose there's a special quoting to be had for patterns? Perhaps one
of these (modulo SQL parsing issues);
/pattern/
{pattern}
qr/pattern/
qr'pattern'
R/pattern/
R'pattern'
Mike bikeshed is scarlet,
David
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types and extensions
On Mon, 2011-06-20 at 16:01 +0200, Florian Pflug wrote: > Hm, I'm starting to wonder if there isn't a way around that. It seems that > this restriction comes from the desire to allow functions with the > polymorphic signature > (ANYELEMENT, ANYELEMENT) -> ANYRANGE. > > The only such function I can currently come up with is the generic > range constructor. Is having that worth the restriction to one > range type per base type? Good point. Having constructors is obviously important, but perhaps they don't have to be generic. We could generate catalog entries for each constructor for each range type, and name them after the range type itself. So, instead of: range(1, 10) you'd write: int4range(1,10) That actually might be better anyway, because relying on the polymorphic version is not perfect now anyway. For instance, if you want an int8range using the generic range() constructor, you need a cast. We'd still need to get the polymorphic type system to work the way we want in this case. I'll look into that. > Another option might be to extend polymorphic argument matching > to allow functions with the signature > () -> > but to require the concrete output type to be specified with a cast > at the call site. For the generic range constructor, you'd then > have to write > RANGE(lower, upper)::range_type Interesting idea. > A third approach might be to first define a PAIR type and then > define ranges on top of that. Since PAIR types wouldn't include > a comparison operators, the restriction to one PAIR type per > base type wouldn't matter. Instead of a generic RANGE constructor > you'd then use the generic PAIR constructor and cast the resulting > PAIR to whatever range you desire, i.e. write > PAIR(lower, upper)::range_type. Another interesting idea. A little awkward though, and doesn't offer much opportunity to specify inclusivity/exclusivity of the bounds. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific
Excerpts from Pavel Stehule's message of lun jun 20 11:34:25 -0400 2011: > b) probably you can simplify a memory management using own two > persistent memory context - and you can swap it. Then functions like > free_hba_record, clean_hba_list, free_lines should be removed. Yeah, I reworked the patch with something like that over the weekend. Not all of it though. I'll send an updated version shortly. -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] proposal: a validator for configuration files
Florian Pflug writes: > The code the actually implements the "check settings first, apply later" logic > isn't easy to read. Now, assume that this code has a bug. Then, with your > patch applied, we might end up with the postmaster applying a setting (because > it didn't abort early) but the backend ignoring it (because they did abort > early). > This is obviously bad. Depending on the setting, the consequences may range > from slightly confusing behaviour to outright crashes I guess... This is already known to happen: there are cases where the postmaster and a backend can come to different conclusions about whether a setting is valid (eg, because it depends on database encoding). Whether that's a bug or not isn't completely clear, but if this patch is critically dependent on the situation never happening, I don't think we can accept it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [WIP] cache estimates, cache access cost
Greg Smith wrote: > On 06/19/2011 06:15 PM, Kevin Grittner wrote: >> I think the point is that if, on a fresh system, the first access >> to a table is something which uses a tables scan -- like select >> count(*) -- that all indexed access would then tend to be >> suppressed for that table. After all, for each individual query, >> selfishly looking at its own needs in isolation, it likely >> *would* be faster to use the cached heap data. > > If those accesses can compete with other activity, such that the > data really does stay in the cache rather than being evicted, then > what's wrong with that? The problem is that if somehow the index *does* find its way into cache, the queries might all run an order of magnitude faster by using it. The *first* query to bite the bullet and read through the index wouldn't, of course, since it would have all that random disk access. But its not hard to imagine an application mix where this feature could cause a surprising ten-fold performance drop after someone does a table scan which could persist indefinitely. I'm not risking that in production without a clear mechanism to automatically recover from that sort of cache skew. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Auto Start second postgres 8.3.15-1 instance MAC OS X
Hi, I'm used to work with PostgreSQL on Windows but now I've moved to OS X
and I'm having problems to create a service to auto start a new server
(instance) of PostgreSQL.
Firstly I used the PostgreSQL installer to create the first server and the
postgres user, then I used initdb to create another server and until this
step, everything turned fine.
After that, I created a folder on StartupItems with StartupParameters and
the other script.
Here are the contents of the two files:
#walkinsense-pgsql
#!/bin/sh
. /etc/rc.common
# Postgres Plus Service script for OS/X
StartService ()
{
ConsoleMessage "Starting Walkinsense PostgreSQL"
su - postgres -c "/Library/PostgreSQL/8.3/bin/pg_ctl -D
/Users/iMac1/Library/Teste/data -l
/Library/Walkinsense/data/pg_log/startup.log -o \"-p 5440\" start"
if [ -e "/Library/Walkinsense/data/postmaster.pid" ]
then
ConsoleMessage "PostgreSQL 8.3 started successfully"
else
ConsoleMessage "PostgreSQL 8.3 did not start in a timely fashion, please see
/Library/Walkinsense/data/pg_log/startup.log for details"
fi
}
StopService()
{
ConsoleMessage "Stopping PostgreSQL 8.3"
su - postgres -c "/Library/PostgreSQL/8.3/bin/pg_ctl stop -m fast -w -D
/Users/iMac1/Library/Teste/data"
}
RestartService ()
{
StopService
sleep 2
StartService
}
RunService "$1"
#SetupParameters
{
Description = "Walkinsense-pgsql";
Provides = ("walkinsense-pgsql");
Requires = ("Resolver");
Preference= "Late";
Messages =
{
start = "Starting Walkinsense PostgreSQL";
stop = "Stopping Walkinsense PostgreSQL";
};
}
Then, I changed the directory permissions doing "sudo chown root:wheel
/Library/StartupItems/walkinsense-pgsql" at the terminal.
When I do "sudo SystemStarter start walkinsense-pgsql" at the terminal, it
gives back the message:
Starting Walkinsense PostgreSQL
pg_ctl: could not open PID file
"/Users/iMac1/Library/Teste/data/postmaster.pid": Permission denied
As you see, the server doesn't start.
I hope I explained it right.
Thanks!!
Diogo
Re: [HACKERS] POSIX question
On Monday, June 20, 2011 17:11:14 Greg Stark wrote: > On Mon, Jun 20, 2011 at 4:01 PM, Florian Pflug wrote: > > Are you sure? Isn't mmap()ing /dev/null a way to *allocate* memory? > > > > Or at least this is what I always thought glibc does when you malloc() > > It mmaps /dev/zero actually. As the nitpicking has already started: Afair its just passing -1 as fd and uses the MAP_ANONYMOUS flag argument ;) Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific
sorry > a) you don't use macro "token_matches" consistently should be a) you don't use macro "token_is_keyword" consistently it should be used for all keywords Regards Pavel Stehule -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific
Hello Brendan, I checked your patch, it is applied cleanly and I don't see any mayor problem. This patch does all what is expected. I have two minor comments a) you don't use macro "token_matches" consistently func: parse_hba_line <-->if (strcmp(token->string, "local") == 0) should be if (token_is_keyword(token, "local")) ... I don't see any sense when somebody use a quotes there. b) probably you can simplify a memory management using own two persistent memory context - and you can swap it. Then functions like free_hba_record, clean_hba_list, free_lines should be removed. Regards Pavel Stehule 2011/6/18 Brendan Jurd : > On 18 June 2011 13:43, Alvaro Herrera wrote: >> Is this really a WIP patch? I'm playing a bit with it currently, seems >> fairly sane. >> > > In this case, the WIP designation is meant to convey "warning: only > casual testing has beeen done". I tried it out with various > permutations of pg_hba.conf, and it worked as advertised in those > tests, but I have not made any attempt to formulate a more rigorous > testing regimen. > > In particular I haven't tested that the more exotic authentication > methods still work properly, and I can't recall whether I tested > recursive file inclusion and group membership. > > Is that a wrongful use of the WIP designation? > > Cheers, > BJ > -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types and extensions
Greg Stark writes: > On Mon, Jun 20, 2011 at 3:17 PM, Tom Lane wrote: >> Given the need to deal with multiple collations for collatable types, >> I'd say it's not so much "unfortunate" as "utterly unworkable". At >> least unless you give up the notion of binding the collation into the >> type definition ... which has other issues, per discussion a few days >> ago. Even ignoring collations, I really think we want to allow multiple >> range types for base types that have multiple btree sort orderings. > I was imagining it would be not part of the type but part of the > internal data in the range type. The dumped representation would look > something like ['bar','baz',''en_US'] and input forms like > ['bar','baz'] would just default to the database default collation or > the session's default collation or whatever. > The most disturbing thing about this is that it would make > unrestorable dumps if any of those collation names change or are not > installed before the data is loaded. It's kind of like having your > table names embedded in a text column in your tables. It could make > things awkward to manage later. Yeah. In particular this would cause issues for pg_upgrade, which would have to somehow ensure that collation OIDs didn't change between old and new installations, which is just about impossible given the current method for assigning them. I think we need to avoid that, really. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] POSIX question
Florian Pflug Monday 20 of June 2011 17:07:55 > On Jun20, 2011, at 17:05 , Radosław Smogura wrote: > > I'm sure at 99%. When I ware "playing" with mmap I preallocated, > > probably, about 100GB of memory. > > You need to set vm.overcommit_memory to "2" to see the difference. Did > you do that? > > You can do that either with "echo 2 > /proc/sys/vm/overcommit_memory" > or by editing /etc/sysctl.conf and issuing "sysctl -p". > > best regards, > Florian Pflug I've just created 127TB mapping in Linux - maximum allowed by VM. Trying overcommit with 0,1,2. Regards, Radek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] proposal: a validator for configuration files
On Jun20, 2011, at 17:02 , Alexey Klyukin wrote: > On Jun 18, 2011, at 5:40 PM, Florian Pflug wrote: >> On Jun16, 2011, at 22:34 , Alexey Klyukin wrote: >>> Attached is the v2 of the patch to show all parse errors at postgresql.conf. >>> Changes (per review and suggestions from Florian): >>> >>> - do not stop on the first error during postmaster's start. >>> - fix errors in processing files from include directives. >>> - show only a single syntax error per line, i.e. fast forward to the EOL >>> after coming across the first one. >>> - additional comments/error messages, code cleanup >> >> Looks mostly good. >> >> I found one issue which I missed earlier. As it stands, the behaviour >> of ProcessConfigFile() changes subtly when IsUnderPostmaster because of >> the early abort on errors. Now, in theory the outcome should still be >> the same, since we don't apply any settings if there's an error in one >> of them. But still, there's a risk that this code isn't 100% waterproof, >> and then we'd end up with different settings in the postmaster compared >> to the backends. The benefit seems also quite small - since the backends >> emit their messages at DEBUG2, you usually won't see the difference >> anyway. > > I don't think it has changed at all. Previously, we did goto cleanup_list (or > cleanup_exit in ParseConfigFp) right after the first error, no matter whether > that was a postmaster or its child. What I did in my patch is removing the > goto for the postmaster's case. It was my intention to exit after the initial > error for the postmaster's child, to avoid complaining about all errors both > in the postmaster and in the normal backend (imagine seeing 100 errors from > the postmaster and the same 100 from each of the backends if your log level is > DEBUG2). I think the postmaster's child case won't cause any problems, since > we do exactly what we used to do before. Hm, I think you miss-understood what I was trying to say, probably because I explained it badly. Let me try again. I fully agree that there *shouldn't* be any difference in behaviour, because it *shouldn't* matter whether we abort early or not - we won't have applied any of the settings anway. But. The code the actually implements the "check settings first, apply later" logic isn't easy to read. Now, assume that this code has a bug. Then, with your patch applied, we might end up with the postmaster applying a setting (because it didn't abort early) but the backend ignoring it (because they did abort early). This is obviously bad. Depending on the setting, the consequences may range from slightly confusing behaviour to outright crashes I guess... Now, the risk of that happening might be very small. But on the other hand, the benefit is pretty small also - you get a little less output for log level DEBUG2, that's it. A level that people probably don't use for the production databases anyway. This convinced me that the risk/benefit ratio isn't high enough to warrant the early abort. Another benefit of removing the check is that it reduces code complexity. Maybe not when measured in line counts, but it's one less outside factor that changes ProcessConfigFiles()'s behaviour and thus one thing less you need to think when you modify that part again in the future. Again, it's a small benefit, but IMHO it still outweights the benefit. Having said that, this is my personal opinion and whoever will eventually commit this may very will assess the cost/benefit ratio differently. So, if after this more detailed explanations of my reasoning, you still feel that it makes sense to keep the early abort, then feel free to mark the patch "Ready for Committer" nevertheless. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] POSIX question
On Mon, Jun 20, 2011 at 04:16:58PM +0200, Florian Pflug wrote: > On Jun20, 2011, at 15:27 , Radosław Smogura wrote: > > 1. mmap some large amount of anonymous virtual memory (this will be maximum > > size of shared memory). > > ... > > Point 1. will no eat memory, as memory allocation is delayed and in 64bit > > platforms you may reserve quite huge chunk of this, and in future it may be > > possible using mmap / munmap to concat chunks / defrag it etc. > > I think this breaks with strict overcommit settings (i.e. > vm.overcommit_memory = 2 on linux). To fix that, you'd need a way to tell the > kernel (or glibc) to simply reserve a chunk of virtual address space for > further user. Not sure if there's a API for that... I run discless swapless cluster systems with zero overcommit (i.e. it's entirely disabled), which means that all operations are strict success/fail due to allocation being immediate. mmap of a large amount of anonymous memory would almost certainly fail on such a setup--you definitely can't assume that a large anonymous mmap will always succeed, since there is no delayed allocation. [we do in reality have a small overcommit allowance to permit efficient fork(2), but it's tiny and (in this context) irrelevant] Regards, Roger -- .''`. Roger Leigh : :' : Debian GNU/Linux http://people.debian.org/~rleigh/ `. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/ `-GPG Public Key: 0x25BFB848 Please GPG sign your mail. signature.asc Description: Digital signature
Re: [HACKERS] POSIX question
On Mon, Jun 20, 2011 at 4:01 PM, Florian Pflug wrote: > Are you sure? Isn't mmap()ing /dev/null a way to *allocate* memory? > > Or at least this is what I always thought glibc does when you malloc() It mmaps /dev/zero actually. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types and extensions
On Mon, Jun 20, 2011 at 3:17 PM, Tom Lane wrote: > Given the need to deal with multiple collations for collatable types, > I'd say it's not so much "unfortunate" as "utterly unworkable". At > least unless you give up the notion of binding the collation into the > type definition ... which has other issues, per discussion a few days > ago. Even ignoring collations, I really think we want to allow multiple > range types for base types that have multiple btree sort orderings. I was imagining it would be not part of the type but part of the internal data in the range type. The dumped representation would look something like ['bar','baz',''en_US'] and input forms like ['bar','baz'] would just default to the database default collation or the session's default collation or whatever. The most disturbing thing about this is that it would make unrestorable dumps if any of those collation names change or are not installed before the data is loaded. It's kind of like having your table names embedded in a text column in your tables. It could make things awkward to manage later. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] POSIX question
On Monday, June 20, 2011 17:05:48 Radosław Smogura wrote: > Florian Pflug Monday 20 of June 2011 17:01:40 > > > On Jun20, 2011, at 16:39 , Radosław Smogura wrote: > > > Florian Pflug Monday 20 of June 2011 16:16:58 > > > > > >> On Jun20, 2011, at 15:27 , Radosław Smogura wrote: > > >>> 1. mmap some large amount of anonymous virtual memory (this will be > > >>> maximum size of shared memory). ... > > >>> Point 1. will no eat memory, as memory allocation is delayed and in > > >>> 64bit platforms you may reserve quite huge chunk of this, and in > > >>> future it may be possible using mmap / munmap to concat chunks / > > >>> defrag it etc. > > >> > > >> I think this breaks with strict overcommit settings (i.e. > > >> vm.overcommit_memory = 2 on linux). To fix that, you'd need a way to > > >> tell the kernel (or glibc) to simply reserve a chunk of virtual > > >> address space for further user. Not sure if there's a API for that... > > >> > > >> best regards, > > >> Florian Pflug > > > > > > This may be achived by many other things, like mmap /dev/null. > > > > Are you sure? Isn't mmap()ing /dev/null a way to *allocate* memory? > > > > Or at least this is what I always thought glibc does when you malloc() > > are large block at once. (This allows it to actually return the memory > > to the kernel once you free() it, which isn't possible if the memory > > was allocated simply by extending the heap). > > > > You can work around this by mmap()ing an actual file, because then > > the kernel knows it can use the file as backing store and thus doesn't > > need to reserve actual physical memory. (In a way, this just adds > > additional swap space). Doesn't seem very clean though... > > > > Even if there's a way to work around a strict overcommit setting, unless > > the workaround is a syscall *explicitly* designed for that purpose, I'd > > be very careful with using it. You might just as well be exploiting a > > bug in the overcommit accounting logic and future kernel versions may > > simply choose to fix the bug... > > > > best regards, > > Florian Pflug > > I'm sure at 99%. When I ware "playing" with mmap I preallocated, probably, > about 100GB of memory. The default setting is to allow overcommit. Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] POSIX question
On Jun20, 2011, at 17:05 , Radosław Smogura wrote: > I'm sure at 99%. When I ware "playing" with mmap I preallocated, probably, > about 100GB of memory. You need to set vm.overcommit_memory to "2" to see the difference. Did you do that? You can do that either with "echo 2 > /proc/sys/vm/overcommit_memory" or by editing /etc/sysctl.conf and issuing "sysctl -p". best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] POSIX question
Florian Pflug Monday 20 of June 2011 17:01:40 > On Jun20, 2011, at 16:39 , Radosław Smogura wrote: > > Florian Pflug Monday 20 of June 2011 16:16:58 > > > >> On Jun20, 2011, at 15:27 , Radosław Smogura wrote: > >>> 1. mmap some large amount of anonymous virtual memory (this will be > >>> maximum size of shared memory). ... > >>> Point 1. will no eat memory, as memory allocation is delayed and in > >>> 64bit platforms you may reserve quite huge chunk of this, and in > >>> future it may be possible using mmap / munmap to concat chunks / > >>> defrag it etc. > >> > >> I think this breaks with strict overcommit settings (i.e. > >> vm.overcommit_memory = 2 on linux). To fix that, you'd need a way to > >> tell the kernel (or glibc) to simply reserve a chunk of virtual address > >> space for further user. Not sure if there's a API for that... > >> > >> best regards, > >> Florian Pflug > > > > This may be achived by many other things, like mmap /dev/null. > > Are you sure? Isn't mmap()ing /dev/null a way to *allocate* memory? > > Or at least this is what I always thought glibc does when you malloc() > are large block at once. (This allows it to actually return the memory > to the kernel once you free() it, which isn't possible if the memory > was allocated simply by extending the heap). > > You can work around this by mmap()ing an actual file, because then > the kernel knows it can use the file as backing store and thus doesn't > need to reserve actual physical memory. (In a way, this just adds > additional swap space). Doesn't seem very clean though... > > Even if there's a way to work around a strict overcommit setting, unless > the workaround is a syscall *explicitly* designed for that purpose, I'd > be very careful with using it. You might just as well be exploiting a > bug in the overcommit accounting logic and future kernel versions may > simply choose to fix the bug... > > best regards, > Florian Pflug I'm sure at 99%. When I ware "playing" with mmap I preallocated, probably, about 100GB of memory. Regards, Radek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] proposal: a validator for configuration files
Florian, On Jun 18, 2011, at 5:40 PM, Florian Pflug wrote: > On Jun16, 2011, at 22:34 , Alexey Klyukin wrote: >> Attached is the v2 of the patch to show all parse errors at postgresql.conf. >> Changes (per review and suggestions from Florian): >> >> - do not stop on the first error during postmaster's start. >> - fix errors in processing files from include directives. >> - show only a single syntax error per line, i.e. fast forward to the EOL >> after coming across the first one. >> - additional comments/error messages, code cleanup > > Looks mostly good. > > I found one issue which I missed earlier. As it stands, the behaviour > of ProcessConfigFile() changes subtly when IsUnderPostmaster because of > the early abort on errors. Now, in theory the outcome should still be > the same, since we don't apply any settings if there's an error in one > of them. But still, there's a risk that this code isn't 100% waterproof, > and then we'd end up with different settings in the postmaster compared > to the backends. The benefit seems also quite small - since the backends > emit their messages at DEBUG2, you usually won't see the difference > anyway. I don't think it has changed at all. Previously, we did goto cleanup_list (or cleanup_exit in ParseConfigFp) right after the first error, no matter whether that was a postmaster or its child. What I did in my patch is removing the goto for the postmaster's case. It was my intention to exit after the initial error for the postmaster's child, to avoid complaining about all errors both in the postmaster and in the normal backend (imagine seeing 100 errors from the postmaster and the same 100 from each of the backends if your log level is DEBUG2). I think the postmaster's child case won't cause any problems, since we do exactly what we used to do before. > > The elevel setting at the start of ProcessConfigFile() also seemed > needlessly complex, since we cannot have IsUnderPostmaster and > context == PGCS_POSTMASTER at the same time. Agreed. > > I figured it'd be harder to explain the fixes I have in mind than > simply do them and let the code speak. Attached you'll find an updated > version of your v2 patch (called v2a) as well as an incremental patch > against your v2 (called v2a_delta). > > The main changes are the removal of the early aborts when > IsUnderPostmaster and the simplification of the elevel setting > logic in ProcessConfigFile(). Attached is the new patch and the delta. It includes some of the changes from your patch, while leaving the early abort stuff for postmaster's children. Thank you, Alexey. -- Command Prompt, Inc. http://www.CommandPrompt.com PostgreSQL Replication, Consulting, Custom Development, 24x7 support pg_parser_continue_on_error_v2b.patch Description: Binary data pg_parser_continue_on_error_v2b_delta.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] POSIX question
On Jun20, 2011, at 16:39 , Radosław Smogura wrote: > Florian Pflug Monday 20 of June 2011 16:16:58 >> On Jun20, 2011, at 15:27 , Radosław Smogura wrote: >>> 1. mmap some large amount of anonymous virtual memory (this will be >>> maximum size of shared memory). ... >>> Point 1. will no eat memory, as memory allocation is delayed and in 64bit >>> platforms you may reserve quite huge chunk of this, and in future it may >>> be possible using mmap / munmap to concat chunks / defrag it etc. >> >> I think this breaks with strict overcommit settings (i.e. >> vm.overcommit_memory = 2 on linux). To fix that, you'd need a way to tell >> the kernel (or glibc) to simply reserve a chunk of virtual address space >> for further user. Not sure if there's a API for that... >> >> best regards, >> Florian Pflug > > This may be achived by many other things, like mmap /dev/null. Are you sure? Isn't mmap()ing /dev/null a way to *allocate* memory? Or at least this is what I always thought glibc does when you malloc() are large block at once. (This allows it to actually return the memory to the kernel once you free() it, which isn't possible if the memory was allocated simply by extending the heap). You can work around this by mmap()ing an actual file, because then the kernel knows it can use the file as backing store and thus doesn't need to reserve actual physical memory. (In a way, this just adds additional swap space). Doesn't seem very clean though... Even if there's a way to work around a strict overcommit setting, unless the workaround is a syscall *explicitly* designed for that purpose, I'd be very careful with using it. You might just as well be exploiting a bug in the overcommit accounting logic and future kernel versions may simply choose to fix the bug... best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] POSIX question
* Florian Pflug: > I think this breaks with strict overcommit settings > (i.e. vm.overcommit_memory = 2 on linux). To fix that, you'd need a > way to tell the kernel (or glibc) to simply reserve a chunk of virtual > address space for further user. Not sure if there's a API for that... mmap with PROT_NONE and subsequent update with mprotect does this on Linux. (It's not clear to me what this is trying to solve, though.) -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pika buildfarm member failure on isolationCheck tests
Dan Ports wrote: > It looks the problem comes from the change a couple days ago that > removed the SXACT_FLAG_ROLLED_BACK flag and changed the > SxactIsRolledBack checks to SxactIsDoomed. That's the correct > thing to do everywhere else, but gets us in trouble here. We > shouldn't be ignoring doomed transactions in > SetNewSxactGlobalXmin, because they're not eligible for cleanup > until the associated backend aborts the transaction and calls > ReleasePredicateLocks. > > However, it isn't as simple as just removing the SxactIsDoomed > check from SetNewSxactGlobalXmin. ReleasePredicateLocks calls > SetNewSxactGlobalXmin *before* it releases the aborted > transaction's SerializableXact (it pretty much has to, because we > must drop and reacquire SerializableXactHashLock in between). But > we don't want the aborted transaction included in the > SxactGlobalXmin computation. > > It seems like we do need that SXACT_FLAG_ROLLED_BACK after all, > even though it's only set for this brief interval. We need to be > able to distinguish a transaction that's just been marked for > death (doomed) from one that's already called > ReleasePredicateLocks. I just looked this over and I agree. It looks to me like we can set this flag in ReleasePredicateLocks (along with the doomed flag) and test it only in SetNewSxactGlobalXmin (instead of the doomed flag). Fundamentally, the problem is that while it's enough to know that a transaction *will be* canceled in order to ignore it during detection of rw-conflicts and dangerous structures, it's not enough when calculating the new serializable xmin -- we need to know that a transaction actually *has been* canceled to ignore it there. Essentially, the fix is to revert a very small part of http://git.postgresql.org/gitweb?p=postgresql.git;a=commit;h=264a6b127a918800d9f8bac80b5f4a8a8799d0f1 Either of us could easily fix this, but since you have the test environment which can reproduce the problem, it's probably best that you do it. Since Heikki took the trouble to put the flags in a nice logical order, we should probably put this right after the doomed flag. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: patch review : Add ability to constrain backend temporary file space
2011/6/20 Robert Haas : > On Mon, Jun 20, 2011 at 9:15 AM, Cédric Villemain > wrote: >> The feature does not work exactly as expected because the write limit >> is rounded per 8kB because we write before checking. I believe if one >> write a file of 1GB in one pass (instead of repetitive 8kB increment), >> and the temp_file_limit is 0, then the server will write the 1GB >> before aborting. > > Can we rearrange thing so we check first, and then write? probably but it needs more work to catch corner cases. We may be safe to just document that (and also in the code). The only way I see so far to have a larger value than 8kB here is to have a plugin doing the sort instead of the postgresql core sort algo. > > -- > Robert Haas > EnterpriseDB: http://www.enterprisedb.com > The Enterprise PostgreSQL Company > -- Cédric Villemain 2ndQuadrant http://2ndQuadrant.fr/ PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] POSIX question
Florian Pflug Monday 20 of June 2011 16:16:58 > On Jun20, 2011, at 15:27 , Radosław Smogura wrote: > > 1. mmap some large amount of anonymous virtual memory (this will be > > maximum size of shared memory). ... > > Point 1. will no eat memory, as memory allocation is delayed and in 64bit > > platforms you may reserve quite huge chunk of this, and in future it may > > be possible using mmap / munmap to concat chunks / defrag it etc. > > I think this breaks with strict overcommit settings (i.e. > vm.overcommit_memory = 2 on linux). To fix that, you'd need a way to tell > the kernel (or glibc) to simply reserve a chunk of virtual address space > for further user. Not sure if there's a API for that... > > best regards, > Florian Pflug This may be achived by many other things, like mmap /dev/null. Regards, Radek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types and extensions
Robert Haas writes: > On Mon, Jun 20, 2011 at 2:33 AM, Jeff Davis wrote: >> Yes, we cannot have two range types with the same base type. That is a >> consequence of the polymorphic type system, which needs to be able to >> determine the range type given the base type. > Boy, that's an unfortunate limitation. :-( Given the need to deal with multiple collations for collatable types, I'd say it's not so much "unfortunate" as "utterly unworkable". At least unless you give up the notion of binding the collation into the type definition ... which has other issues, per discussion a few days ago. Even ignoring collations, I really think we want to allow multiple range types for base types that have multiple btree sort orderings. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] POSIX question
On Jun20, 2011, at 15:27 , Radosław Smogura wrote: > 1. mmap some large amount of anonymous virtual memory (this will be maximum > size of shared memory). > ... > Point 1. will no eat memory, as memory allocation is delayed and in 64bit > platforms you may reserve quite huge chunk of this, and in future it may be > possible using mmap / munmap to concat chunks / defrag it etc. I think this breaks with strict overcommit settings (i.e. vm.overcommit_memory = 2 on linux). To fix that, you'd need a way to tell the kernel (or glibc) to simply reserve a chunk of virtual address space for further user. Not sure if there's a API for that... best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types and extensions
On Jun20, 2011, at 15:19 , Robert Haas wrote: > On Mon, Jun 20, 2011 at 2:33 AM, Jeff Davis wrote: >> On Sun, 2011-06-19 at 21:29 +0200, Florian Pflug wrote: >>> If I'm not mistaken about this, that would imply that we also cannot >>> have two range types with the same base type, the same opclass, >>> but different collations. Which seems rather unfortunate... In fact, >>> if that's true, maybe restricing range types to the database collation >>> would be best... >> >> Yes, we cannot have two range types with the same base type. That is a >> consequence of the polymorphic type system, which needs to be able to >> determine the range type given the base type. > > Boy, that's an unfortunate limitation. :-( Hm, I'm starting to wonder if there isn't a way around that. It seems that this restriction comes from the desire to allow functions with the polymorphic signature (ANYELEMENT, ANYELEMENT) -> ANYRANGE. The only such function I can currently come up with is the generic range constructor. Is having that worth the restriction to one range type per base type? Another option might be to extend polymorphic argument matching to allow functions with the signature () -> but to require the concrete output type to be specified with a cast at the call site. For the generic range constructor, you'd then have to write RANGE(lower, upper)::range_type (If we had that, we could also (finally) provide functions to set and get fields of composite types by name. As it stands, doing that cleanly is hard because the desired signature of the get function, namely (record, fieldname text) -> anyelement is not supported.) A third approach might be to first define a PAIR type and then define ranges on top of that. Since PAIR types wouldn't include a comparison operators, the restriction to one PAIR type per base type wouldn't matter. Instead of a generic RANGE constructor you'd then use the generic PAIR constructor and cast the resulting PAIR to whatever range you desire, i.e. write PAIR(lower, upper)::range_type. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.
On Mon, Jun 20, 2011 at 09:15:52AM -0400, Robert Haas wrote: > Yep. Peter overrides them just before each release. Aren't there better ways to implement this, like git submodules? This redundancy seem awkward to me. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org Jabber: michael.meskes at googlemail dot com VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types and extensions
On Sun, Jun 19, 2011 at 11:33:02PM -0700, Jeff Davis wrote: > On Sun, 2011-06-19 at 21:29 +0200, Florian Pflug wrote: > > If I'm not mistaken about this, that would imply that we also > > cannot have two range types with the same base type, the same > > opclass, but different collations. Which seems rather > > unfortunate... In fact, if that's true, maybe restricing range > > types to the database collation would be best... > > Yes, we cannot have two range types with the same base type. That is > a consequence of the polymorphic type system, which needs to be able > to determine the range type given the base type. > > A workaround is to use domains. That is effective, but awkward. For > instance, given: > CREATE DOMAIN textdomain AS text; > CREATE TYPE textdomainrange AS RANGE (subtype=textdomain); > then: > '[a,z)'::textdomainrange @> 'b'::textdomain > would work, but: > '[a,z)'::textdomainrange @> 'b' > would not, which would be annoying. > > I don't see a way around this. It's not a collation problem, but a > general "multiple range types with the same subtype" problem. How might you address that problem, assuming you had the needed resources to do it? Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe
Simon Riggs wrote: > I'm looking for opinions ranging from fix-now-and-backpatch thru > to ignore and discuss for 9.2. If it's a pre-existing bug I would think that one option would be to put it into the next bug-fix release of each supported major release in which it is manifest. Of course, if it is *safe* to work it into 9.1, that'd be great. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] POSIX question
Hello, I had some idea with hugepagse, and I read why PostgreSQL doesn't support POSIX (need of nattach). During read about POSIX/SysV I found this (thread about dynamic chunking shared memory). http://archives.postgresql.org/pgsql-hackers/2010-08/msg00586.php When playing with mmap I done some approach how to deal with growing files, so... Maybe this approach could resolve both of above problems (POSIX and dynamic shared memory). Here is idea: 1. mmap some large amount of anonymous virtual memory (this will be maximum size of shared memory). 2. init small SysV chunk for shmem header (to keep "fallout" requirements) 3. SysV remap is Linux specific so unmap few 1st vm pages of step 1. and attach there (2.) 3. a. Lock header when adding chunks (1st chunk is header) (we don't want concurrent chunk allocation) 4. allocate some other chunks of shared memory (POSIX is the best way) and put it in shmem header, put there information like chunk id/name, is this POSIX or SysV, some useful flags (hugepage?) needed by reattaching, attach those in 1. 4b. unlock 3a Point 1. will no eat memory, as memory allocation is delayed and in 64bit platforms you may reserve quite huge chunk of this, and in future it may be possible using mmap / munmap to concat chunks / defrag it etc. Mmap guarants that mmaping with mmap_fixed over already mmaped area will unmap old. A working "preview" changeset applied for sysv_memory.c maybe quite small. If someone will want to "extend" memory, he may add new chunk (ofcourse to keep header memory continuous number of chunks is limited). What do you think about this? Regards, Radek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe
On Mon, Jun 20, 2011 at 2:14 PM, Robert Haas wrote: > If this is a pre-existing bug, then it's not clear to me why we need > to do anything about it at all right now. I mean, it would be nice to > have a fix, but it's hard to imagine that any proposed fix will be > low-risk, and I don't remember user complaints about this. I continue > to think that the root of the problem here is that SnapshotNow is Evil > (TM). If we get rid of that, then this problem goes away, but that > strikes me as a long-term project. There are 2 bugs, one caused by my patch in 9.1, one that is pre-existing. The 9.1 bug can be fixed easily. I will edit my patch down and repost here shortly. The pre-existing bug is slightly harder/contentious because we have to lock the name of a possible relation, even before we know it exists. I've been looking to implement that as a lock on the uint32 hash of the relation's name and namespace. I'm looking for opinions ranging from fix-now-and-backpatch thru to ignore and discuss for 9.2. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: patch review : Add ability to constrain backend temporary file space
On Mon, Jun 20, 2011 at 9:15 AM, Cédric Villemain wrote: > The feature does not work exactly as expected because the write limit > is rounded per 8kB because we write before checking. I believe if one > write a file of 1GB in one pass (instead of repetitive 8kB increment), > and the temp_file_limit is 0, then the server will write the 1GB > before aborting. Can we rearrange thing so we check first, and then write? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types and extensions
On Mon, Jun 20, 2011 at 2:33 AM, Jeff Davis wrote: > On Sun, 2011-06-19 at 21:29 +0200, Florian Pflug wrote: >> If I'm not mistaken about this, that would imply that we also cannot >> have two range types with the same base type, the same opclass, >> but different collations. Which seems rather unfortunate... In fact, >> if that's true, maybe restricing range types to the database collation >> would be best... > > Yes, we cannot have two range types with the same base type. That is a > consequence of the polymorphic type system, which needs to be able to > determine the range type given the base type. Boy, that's an unfortunate limitation. :-( -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: patch review : Add ability to constrain backend temporary file space
2011/6/17 Cédric Villemain : > 2011/6/17 Mark Kirkwood : >> On 17/06/11 13:08, Mark Kirkwood wrote: >>> >>> On 17/06/11 09:49, Cédric Villemain wrote: I have issues applying it. Please can you remove trailing space? Also, you can generate a cool patch like this : get git-external-diff from postgres/src/tools to /usr/lib/git-core/ chmod +x it export GIT_EXTERNAL_DIFF=git-external-diff git format-patch --ext-diff origin >> >> I think I have the trailing spaces removed, and patch is updated for the >> variable renaming recently done in fd.c >> >> I have no idea why I can't get the git apply to work (obviously I have >> exceeded by git foo by quite a ways), but it should apply for you I hope (as >> it patches fine). >> > > If I didn't made mistake the attached patch does not have trailling > space anymore and I did a minor cosmetic in FileClose. It is not in > the expected format required by postgresql commiters but can be > applyed with git apply... > It looks like the issue is that patch generated with the git-ext-diff > can not be git applyed (they need to use patch). > > Either I did something wrong or git-ext-diff format is not so great. > > > I didn't test and all yet. From reading, the patch looks sane. I'll > review it later this day or this week-end. > > Submission review === I have updated the patch for cosmetic. (I attach 2 files: 1 patch for 'git-apply' and 1 for 'patch') I have also updated the guc.c entry (lacks of a NULL for the last Hooks handler and make the pg_settings comment more like other GUC) The patch includes doc and test. Usability review = The patch implements what it claims modulo the BLK size used by PostgreSQL: the filewrite operation happens before the test on file limit and PostgreSQL write per 8kB. The result is that a temp_file_limit=0 does not prevent temp_file writing or creation, but stop the writing at 8192 bytes. I've already argue to keep the value as a BLKSZ, but I don't care to have kB by default. The doc may need to be updated to reflect this behavior. Maybe we want to disable the temp_file writing completely with the value set to 0 ? (it looks useless to me atm but someone may have a usecase for that ?!) I don't believe it follows any SQL specs, and so far the community did not refuse the idea, so looks in the good way. I believe we want it, my only grippe is that a large value don't prevent server to use a lot of IO before the operation is aborted, I don't see real alternative at this level though. After initial testing some corner cases have been outline (in previous versions of the patch), so I believe the current code fix them well as I didn't have issue again. Feature test The feature does not work exactly as expected because the write limit is rounded per 8kB because we write before checking. I believe if one write a file of 1GB in one pass (instead of repetitive 8kB increment), and the temp_file_limit is 0, then the server will write the 1GB before aborting. Also, the patch does not handle a !(returnCode >=0) in FileWrite. I don't know if it is needed or not. As the counter is per session and we can have very long session, it may hurtbut the problem will be larger than just this counter in such case so...is it worth the trouble to handle it ? Performance review === not done. Coding review = I have done some update to the patch to make it follow the coding guidelines. I don't think it can exist a portability issue. Tom did say: I'd think MB would be a practical unit size, and would avoid (at least for the near term) the need to make the parameter a float. Architecture review IMO, the code used to log_temp_file is partially deprecated by this feature: the system call to stat() looks now useles and can be removed as well as the multiple if () around temp_file_limit or log_temp_file can be merged. Review review = I didn't test for performances, I have not tryed to overflow temporary_files_size but looks hard to raise. (this might be possible if the temp_file_limit is not used but the temporary_files_size incremented, but current code does not allow to increment one without using the feature, so it is good I think) Mark, please double check that my little updates of your patch did not remove anything or modify the behavior in a wrong way. Thanks for your good job, -- Cédric Villemain 2ndQuadrant http://2ndQuadrant.fr/ PostgreSQL : Expertise, Formation et Support diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 794aef4..19dc440 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -1025,6 +1025,43 @@ SET ENABLE_SEQSCAN TO OFF; + + Disk + + + + temp_file_limit (integer) + + temp_file_limit configuration parameter + + + +Specifies the amount of disk space used by int
Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.
2011/6/20 Michael Meskes : > On Mon, Jun 20, 2011 at 03:03:37PM +0300, Devrim GÜNDÜZ wrote: >> On Mon, 2011-06-20 at 11:58 +, Michael Meskes wrote: >> > >> > Fixed string in German translation that causes segfault. >> > >> > Applied patch by Christoph Berg to replace placeholder >> > "%s" by correct string. >> >> AFAIK this won't have any effect. This change needs to go to through >> pgtranslation project. > > Oops, didn't know that. Does that mean the files in our main git are not the > canonical versions? Yep. Peter overrides them just before each release. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe
On Sun, Jun 19, 2011 at 5:13 PM, Simon Riggs wrote: > On Fri, Jun 17, 2011 at 11:41 PM, Tom Lane wrote: >> Robert Haas writes: >>> On Thu, Jun 16, 2011 at 6:54 PM, Tom Lane wrote: 4. Backend #2 visits the new, about-to-be-committed version of pgbench_accounts' pg_class row just before backend #3 commits. It sees the row as not good and keeps scanning. By the time it reaches the previous version of the row, however, backend #3 *has* committed. So that version isn't good according to SnapshotNow either. >> >>> >> >>> Why isn't this a danger for every pg_class update? For example, it >>> would seem that if VACUUM updates relpages/reltuples, it would be >>> prone to this same hazard. >> >> VACUUM does that with an in-place, nontransactional update. But yes, >> this is a risk for every transactional catalog update. > > Well, after various efforts to fix the problem, I notice that there > are two distinct problems brought out by your test case. > > One of them is caused by my patch, one of them was already there in > the code - this latter one is actually the hardest to fix. > > It took me about an hour to fix the first bug, but its taken a while > of thinking about the second before I realised it was a pre-existing > bug. > > The core problem is, as you observed that a pg_class update can cause > rows to be lost with concurrent scans. > We scan pg_class in two ways: to rebuild a relcache entry based on a > relation's oid (easy fix). We also scan pg_class to resolve the name > to oid mapping. The name to oid mapping is performed *without* a lock > on the relation, since we don't know which relation to lock. So the > name lookup can fail if we are in the middle of a pg_class update. > This is an existing potential bug in Postgres unrelated to my patch. > Ref: SearchCatCache() > > I've been looking at ways to lock the relation name and namespace > prior to the lookup (or more precisely the hash), but its worth > discussing whether we want that at all? If this is a pre-existing bug, then it's not clear to me why we need to do anything about it at all right now. I mean, it would be nice to have a fix, but it's hard to imagine that any proposed fix will be low-risk, and I don't remember user complaints about this. I continue to think that the root of the problem here is that SnapshotNow is Evil (TM). If we get rid of that, then this problem goes away, but that strikes me as a long-term project. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.
On Mon, Jun 20, 2011 at 03:03:37PM +0300, Devrim GÜNDÜZ wrote: > On Mon, 2011-06-20 at 11:58 +, Michael Meskes wrote: > > > > Fixed string in German translation that causes segfault. > > > > Applied patch by Christoph Berg to replace placeholder > > "%s" by correct string. > > AFAIK this won't have any effect. This change needs to go to through > pgtranslation project. Oops, didn't know that. Does that mean the files in our main git are not the canonical versions? Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org Jabber: michael.meskes at googlemail dot com VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Grouping Sets
Hello I have not any newest patch related to GROUPING SETS. The last version of this patch is probably correct, but it is not well tested. Actually, this patch has not quality to production usage :(. It is just concept. You can test it. Regards Pavel Stehule 2011/6/18 Mariano Mara : > Hi hackers (and specially Pavel Stehule), > I could really use the grouping set feature for some complex queries I'm > migrating from other db vendor. If my WEB searching is precise, this > wiki page [1] and this thread[2] are the last updates on the subject. > I'm willing to test how these functions in my project but some questions > first: > 1- is there an up-to-date version of the patch that I should be aware of? > 2- Can I apply that patch to 8.4.8? > 3- any extra recommendations? > > TIA, > Mariano > > [1] http://wiki.postgresql.org/wiki/Grouping_Sets > [2] http://archives.postgresql.org/pgsql-hackers/2010-08/msg00647.php > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers > -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUG] Denormal float values break backup/restore
Hi, thanks for chiming in. On Sat, Jun 11, 2011 at 22:08, Jeroen Vermeulen wrote: > AIUI that is defined to be a little vague, but includes denormalized numbers > that would undergo any rounding at all. It says that on overflow the > conversion should return the appropriate HUGE_VAL variant, and set ERANGE. > On underflow it returns a reasonably appropriate value (and either may or > must set ERANGE, which is the part that isn't clear to me). Which standard is that? Does IEEE 754 itself define strtod() or is there another relevant standard? Regards, Marti -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.
On Mon, 2011-06-20 at 11:58 +, Michael Meskes wrote: > > Fixed string in German translation that causes segfault. > > Applied patch by Christoph Berg to replace placeholder > "%s" by correct string. AFAIK this won't have any effect. This change needs to go to through pgtranslation project. -- Devrim GÜNDÜZ Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz signature.asc Description: This is a digitally signed message part
Re: [HACKERS] patch: plpgsql - remove unnecessary ccache search when a array variable is updated
Hello 2011/6/20 Simon Riggs : > On Mon, Jun 20, 2011 at 10:49 AM, Pavel Stehule > wrote: > >> this patch significantly reduce a ccache searching. On my test - bubble sort > > It sounds good, but also somewhat worrying. > > The first cache is slow, so we add another cache to avoid searching > the first cache. > > What is making the first cache so slow? a using of general cache should be slower than direct access to memory. The slow down is based on catalog operations - hash calculations, hash searching and cache validations. I don't know if it is possible to optimize general cache. you can compare profile of original pg 3008 13.0493 SearchCatCache 1306 5.6657 ExecEvalParamExtern 1143 4.9586 GetSnapshotData 1122 4.8675 AllocSetAlloc 1058 4.5898 MemoryContextAllocZero 1002 4.3469 ExecMakeFunctionResultNoSets 986 4.2775 ExecEvalArrayRef 851 3.6918 LWLockAcquire 783 3.3968 LWLockRelease 664 2.8806 RevalidateCachedPlan 646 2.8025 AllocSetFree 568 2.4641 array_ref 551 2.3904 CopySnapshot 519 2.2515 AllocSetReset 510 2.2125 array_set 492 2.1344 PopActiveSnapshot 381 1.6529 ArrayGetOffset 369 1.6008 AcquireExecutorLocks 348 1.5097 pfree 347 1.5054 MemoryContextAlloc 313 1.3579 bms_is_member 285 1.2364 CatalogCacheComputeHashValue 267 1.1583 PushActiveSnapshot 266 1.1540 hash_uint32 253 1.0976 pgstat_init_function_usage 233 1.0108 array_seek.clone.0 and patched postgresql's profile 3151 7.2135 AllocSetAlloc 2887 6.6091 ExecEvalParamExtern 2844 6.5107 list_member_ptr 2353 5.3867 AllocSetFree 2318 5.3065 GetSnapshotData 2201 5.0387 ExecMakeFunctionResultNoSets 2153 4.9288 LWLockAcquire 2055 4.7045 ExecEvalArrayRef 1879 4.3015 LWLockRelease 1675 3.8345 MemoryContextAllocZero 1463 3.3492 AcquireExecutorLocks 1375 3.1477 pfree 1356 3.1043 RevalidateCachedPlan 1261 2.8868 AllocSetCheck 1257 2.8776 PopActiveSnapshot 1115 2.5525 array_set 1102 2.5228 AllocSetReset 966 2.2114 CopySnapshot 938 2.1473 MemoryContextAlloc 875 2.0031 array_ref 772 1.7673 ResourceOwnerForgetPlanCacheRef 632 1.4468 array_seek.clone.0 554 1.2683 PushActiveSnapshot 499 1.1423 check_list_invariants 475 1.0874 ExecEvalConst 473 1.0828 bms_is_member 444 1.0164 ArrayGetNItems so the most slow operation is SearchCatCache - but I am not a man who can optimize this routine :) Regards Pavel Stehule > > -- > Simon Riggs http://www.2ndQuadrant.com/ > PostgreSQL Development, 24x7 Support, Training & Services > -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Boolean operators without commutators vs. ALL/ANY
On Jun20, 2011, at 03:16 , Greg Stark wrote: > On Fri, Jun 17, 2011 at 3:49 PM, Florian Pflug wrote: >>> The regex is always to the right of the operator. >> >> Which is something you have to remember... It's not in any >> way deducible from "foo ~ bar" alone. > > Except that it's always been this way, going back to perl4 or tcl or > their predecessors. The regexp binding operator always has the regexp > on the right. Yeah. The strength of that argument really depends on one's prior exposure to these languages, though... How is that worse than the situation with "=~" and "~="? >>> >>> With =~ it is to the right, with ~= it is to the left. >> >> It's always where the tilde is. Yeah, you have to remember that. > > And when you get it wrong it will fail silently. No errors, just wrong > results. Yeah, but this is hardly the only case where you'll get unintended results if you mix up operator names. Now, one might argue, I guess, that mixing up "=~" and "~=" or more likely than mixing up, say, "~" and "~~". But ultimately, whether or not that is highly dependent on one's personal background, so we're unlikely to ever reach agreement on that... > While I've never accidentally written /foo/ =~ $_ in perl I have > *frequently* forgotten whether the operator is ~= or =~. Actually I > forget that pretty much every time I start writing some perl. I just > put whichever comes first and if I get an error I reverse it. Yeah, the nice thing in perl (and ruby also, which is *my* background) is that regexp's and strings are distinguished by the type system, and also by the parser. The latter (i.e. regexp literals enclosed by /../) probably isn't desirably for postgres, but the former definitely is (i.e. distinguishing regexp's and text in the type system). Please see the thread "Adding a distinct pattern type to resolve the ~ commutator stalemate" for the details of the proposal. > I can see the temptation to make it symmetric but it's going to cause > an awful lot of confusion. I do believe that by adding a distinct type we can actually *reduce* confusion. It makes "text ~ pattern" readable even for people who don't intuitively know that the pattern always goes on the right. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: plpgsql - remove unnecessary ccache search when a array variable is updated
On Mon, Jun 20, 2011 at 10:49 AM, Pavel Stehule wrote: > this patch significantly reduce a ccache searching. On my test - bubble sort It sounds good, but also somewhat worrying. The first cache is slow, so we add another cache to avoid searching the first cache. What is making the first cache so slow? -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] date_part for infinity intervals
The behaviour of date_part function is opaque for infinity intervals.
For example
date_part('epoch', 'infinity'::date) and date_part('year',
'infinity'::date) return zero but is supposed to return 'infinity',
date_part('day', 'infinity'::date) returns zero, should it return 'NaN'
instead?
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] patch: plpgsql - remove unnecessary ccache search when a array variable is updated
Hello
this patch significantly reduce a ccache searching. On my test - bubble sort
postgres=# \sf buble
CREATE OR REPLACE FUNCTION public.buble(integer[])
RETURNS integer[]
LANGUAGE plpgsql
AS $function$
declare
unsorted bool := true;
aux int;
begin
while unsorted
loop
unsorted := false;
for i in array_lower($1,1) .. array_upper($1,1) - 1
loop
if $1[i] > $1[i+1] then
aux := $1[i];
$1[i] := $1[i+1]; $1[i+1] := aux;
unsorted := true;
end if;
end loop;
end loop;
return $1;
end;
$function$ immutable
it decrease evaluation time about 15%.
Regards
Pavel Stehule
p.s. I know so bubble sort is not effective for large arrays. This
algorithm was used because a array is intensive modified.
*** ./pl_exec.c.orig 2011-06-20 08:55:37.0 +0200
--- ./pl_exec.c 2011-06-20 11:38:55.988650907 +0200
***
*** 153,158
--- 153,170
static void exec_assign_value(PLpgSQL_execstate *estate,
PLpgSQL_datum *target,
Datum value, Oid valtype, bool *isNull);
+ static void
+ exec_eval_array_datum(PLpgSQL_execstate *estate,
+ PLpgSQL_datum *datum,
+ Oid *arraytypoid,
+ int32 *arraytypmod,
+ int16 *arraytyplen,
+ Oid *elemtypeid,
+ int16 *elemtyplen,
+ bool *elemtybyval,
+ char *elemtypalign,
+ Datum *value,
+ bool *isnull);
static void exec_eval_datum(PLpgSQL_execstate *estate,
PLpgSQL_datum *datum,
Oid *typeid,
***
*** 3981,4001
&arraytypeid, &arraytypmod,
&oldarraydatum, &oldarrayisnull);
! /* If target is domain over array, reduce to base type */
! arraytypeid = getBaseTypeAndTypmod(arraytypeid, &arraytypmod);
!
! /* ... and identify the element type */
! arrayelemtypeid = get_element_type(arraytypeid);
! if (!OidIsValid(arrayelemtypeid))
! ereport(ERROR,
! (errcode(ERRCODE_DATATYPE_MISMATCH),
! errmsg("subscripted object is not an array")));
!
! get_typlenbyvalalign(arrayelemtypeid,
! &elemtyplen,
! &elemtypbyval,
! &elemtypalign);
! arraytyplen = get_typlen(arraytypeid);
/*
* Evaluate the subscripts, switch into left-to-right order.
--- 3993,4009
&arraytypeid, &arraytypmod,
&oldarraydatum, &oldarrayisnull);
! /* Fetch current value of array datum and metadata */
! exec_eval_array_datum(estate, target,
! &arraytypeid,
! &arraytypmod,
! &arraytyplen,
! &arrayelemtypeid,
! &elemtyplen,
! &elemtypbyval,
! &elemtypalign,
! &oldarraydatum,
! &oldarrayisnull);
/*
* Evaluate the subscripts, switch into left-to-right order.
***
*** 4099,4104
--- 4107,4178
}
}
+
+ /*
+ * exec_eval_array_datum Get current value of PLpgSQL_datum
+ *
+ * The type oid, typmod, value in Datum format, and null flag are returned.
+ * The arraytyplen, elemtyplen, elemtypbyval, elemtypalign are returned.
+ *
+ * This routine is is used for array type variables - returns additional info
+ * about array elements - removes useless search inside ccache.
+ */
+ static void
+ exec_eval_array_datum(PLpgSQL_execstate *estate,
+ PLpgSQL_datum *datum,
+ Oid *arraytypoid,
+ int32 *arraytypmod,
+ int16 *arraytyplen,
+ Oid *elemtypeid,
+ int16 *elemtyplen,
+ bool *elemtybyval,
+ char *elemtypalign,
+ Datum *value,
+ bool *isnull)
+ {
+ PLpgSQL_var *var = (PLpgSQL_var *) datum;
+
+ Assert(datum->dtype == PLPGSQL_DTYPE_VAR);
+
+ if (var->array_desc == NULL)
+ {
+ PLpgSQL_array_desc *array_desc = (PLpgSQL_array_desc *) palloc(sizeof(PLpgSQL_array_desc));
+
+ array_desc->arraytypoid = var->datatype->typoid;
+ array_desc->arraytypmod = var->datatype->atttypmod;
+
+ /* If target is domain over array, reduce to base type */
+ array_desc->arraytypoid = getBaseTypeAndTypmod(array_desc->arraytypoid,
+ &array_desc->arraytypmod);
+
+ array_desc->elemtypoid = get_element_type(array_desc->arraytypoid);
+ if (!OidIsValid(array_desc->elemtypoid))
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("subscripted object is not an array")));
+
+ get_typlenbyvalalign(array_desc->elemtypoid,
+ &array_desc->elemtyplen,
+ &array_desc->elemtypbyval,
+ &array_desc->elemtypalign);
+ array_desc->arraytyplen = get_typlen(array_desc->arraytypoid);
+
+ var->array_desc = array_desc;
+ }
+
+ *arraytypoid = var->array_desc->arraytypoid;
+ *arraytypmod = var->array_desc->arraytypmod;
+ *arraytyplen = var->array_desc->arraytyplen;
+
+ *elemtypeid = var->array_desc->elemtypoid;
+ *elemtyplen = var->array_desc->elemtyplen;
+ *elemtybyval = var->array_desc->elemtypbyval;
+ *elemtypalign = var->array_desc->elemtypalign;
+
+ *value = var->value;
+ *isnull = var->isnull;
+ }
+
/*
* exec_ev
Re: [HACKERS] pika buildfarm member failure on isolationCheck tests
On Sun, Jun 19, 2011 at 09:10:02PM -0400, Robert Haas wrote: > Is this an open item for 9.1beta3? Yes. I've put it on the list. The SxactGlobalXmin and its refcount were getting out of sync with the actual transaction state. This is timing-dependent but I can reproduce it fairly reliably under concurrent workloads. It looks the problem comes from the change a couple days ago that removed the SXACT_FLAG_ROLLED_BACK flag and changed the SxactIsRolledBack checks to SxactIsDoomed. That's the correct thing to do everywhere else, but gets us in trouble here. We shouldn't be ignoring doomed transactions in SetNewSxactGlobalXmin, because they're not eligible for cleanup until the associated backend aborts the transaction and calls ReleasePredicateLocks. However, it isn't as simple as just removing the SxactIsDoomed check from SetNewSxactGlobalXmin. ReleasePredicateLocks calls SetNewSxactGlobalXmin *before* it releases the aborted transaction's SerializableXact (it pretty much has to, because we must drop and reacquire SerializableXactHashLock in between). But we don't want the aborted transaction included in the SxactGlobalXmin computation. It seems like we do need that SXACT_FLAG_ROLLED_BACK after all, even though it's only set for this brief interval. We need to be able to distinguish a transaction that's just been marked for death (doomed) from one that's already called ReleasePredicateLocks. Dan -- Dan R. K. Ports MIT CSAILhttp://drkp.net/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe
On Sun, Jun 19, 2011 at 10:13 PM, Simon Riggs wrote: > pre-existing bug I've been able to reproduce the bug on 9.0 stable as well, using the OP's test script. Multiple errors like this... ERROR: relation "pgbench_accounts" does not exist STATEMENT: alter table pgbench_accounts set (fillfactor = 100); Presume we want to fix both bugs, not just the most recent one so will continue with both patches. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: Fast GiST index build
New version of patch. There are some bugfixes, minor refactoring, comments (unfortunatelly, not all the code is covered by comments yet). Also "fastbuild" parameter was added to the GiST index. It allows to test index building with and without fast build without postgres recompile. -- With best regards, Alexander Korotkov. On Thu, Jun 16, 2011 at 10:35 PM, Alexander Korotkov wrote: > Oh, actually it's so easy. Thanks. > > -- > With best regards, > Alexander Korotkov. > > On Thu, Jun 16, 2011 at 10:26 PM, Heikki Linnakangas < > heikki.linnakan...@enterprisedb.com> wrote: > >> On 16.06.2011 21:13, Alexander Korotkov wrote: >> >>> My current idea is to measure number of IO accesses by pg_stat_statements >>> and measure CPU usage by /proc/PID/stat. Any thoughts? >>> >> >> Actually, you get both of those very easily with: >> >> set log_statement_stats=on >> >> LOG: QUERY STATISTICS >> DETAIL: ! system usage stats: >>! 0.000990 elapsed 0.00 user 0.00 system sec >>! [0.00 user 0.008000 sys total] >>! 0/0 [32/0] filesystem blocks in/out >>! 0/0 [0/959] page faults/reclaims, 0 [0] swaps >>! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent >>! 0/0 [10/1] voluntary/involuntary context switches >> STATEMENT: SELECT generate_series(1,100); >> >> >> >> -- >> Heikki Linnakangas >> EnterpriseDB http://www.enterprisedb.com >> > > gist_fast_build-0.2.0.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
