[HACKERS] Allow pg_archivecleanup to ignore extensions

[Greg Smith]

One bit of feedback I keep getting from people who archive their WAL 
files is that the fairly new pg_archivecleanup utility doesn't handle 
the case where those archives are compressed.  As the sort of users who 
are concerned about compression are also often ones with giant archives 
they struggle to cleanup, they would certainly appreciate having a 
bundled utility to take care of that. 

The attached patch provides an additional option to the utility to 
provide this capability.  It just strips a provided extension off any 
matching file it considers before running the test for whether it should 
be deleted or not.  It includes updates to the usage message and some 
docs about how this might be used.  Code by Jaime Casanova and myself.


Here's an example of it working:

$ psql -c "show archive_command"
 archive_command  


cp -i %p archive/%f < /dev/null && gzip archive/%f
[Yes, I know that can be written more cleanly.  I call external scripts 
with more serious error handling than you can put into a single command 
line for this sort of thing in production.]


$ psql -c "select pg_start_backup('test',true)"
$ psql -c "select pg_stop_backup()"
$ psql -c "checkpoint"
$ psql -c "select pg_switch_xlog()"

$ cd $PGDATA/archive
$ ls
00010025.gz
00010026.gz
00010027.gz
00010028.0020.backup.gz
00010028.gz
00010029.gz

$ pg_archivecleanup -d -x .gz `pwd`  
00010028.0020.backup
pg_archivecleanup: keep WAL file 
"/home/gsmith/pgwork/data/archivecleanup/archive/00010028" 
and later
pg_archivecleanup: removing file 
"/home/gsmith/pgwork/data/archivecleanup/archive/00010025.gz"
pg_archivecleanup: removing file 
"/home/gsmith/pgwork/data/archivecleanup/archive/00010027.gz"
pg_archivecleanup: removing file 
"/home/gsmith/pgwork/data/archivecleanup/archive/00010026.gz"

$ ls
00010028.0020.backup.gz
00010028.gz
00010029.gz

We recenty got some on-list griping that pg_standby doesn't handle 
archive files that are compressed, either.  Given how the job I'm 
working on this week is going, I'll probably have to add that feature 
next.  That's actually an easier source code hack than this one, because 
of how the pg_standby code modularizes the concept of a restore command.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us


diff --git a/contrib/pg_archivecleanup/pg_archivecleanup.c b/contrib/pg_archivecleanup/pg_archivecleanup.c
index 7989207..a95b659 100644
*** a/contrib/pg_archivecleanup/pg_archivecleanup.c
--- b/contrib/pg_archivecleanup/pg_archivecleanup.c
*** const char *progname;
*** 36,41 
--- 36,42 
  
  /* Options and defaults */
  bool		debug = false;		/* are we debugging? */
+ char	   *additional_ext = NULL;	/* Extension to remove from filenames */
  
  char	   *archiveLocation;	/* where to find the archive? */
  char	   *restartWALFileName; /* the file from which we can restart restore */
*** static void
*** 93,105 
--- 94,135 
  CleanupPriorWALFiles(void)
  {
  	int			rc;
+ 	int			chop_at;
  	DIR		   *xldir;
  	struct dirent *xlde;
+ 	char		walfile[MAXPGPATH];
  
  	if ((xldir = opendir(archiveLocation)) != NULL)
  	{
  		while ((xlde = readdir(xldir)) != NULL)
  		{
+ 			strncpy(walfile, xlde->d_name, MAXPGPATH);
+ 			/* 
+ 			 * Remove any specified additional extension from the filename
+ 			 * before testing it against the conditions below.
+ 			 */
+ 			if (additional_ext)
+ 			{
+ chop_at = strlen(walfile) - strlen(additional_ext);
+ /*
+  * Only chop if this is long enough to be a file name and the
+  * extension matches.
+  */
+ if ((chop_at >= (XLOG_DATA_FNAME_LEN - 1)) && 
+ 	(strcmp(walfile + chop_at,additional_ext)==0))
+ {
+ 	walfile[chop_at] = '\0';
+ 	/* 
+ 	 * This is too chatty even for regular debug output, but
+ 	 * leaving it in for program testing.
+ 	 */
+ 	if (false)
+ 		fprintf(stderr, 
+ 			"removed extension='%s' from file=%s result=%s\n",
+ 			additional_ext,xlde->d_name,walfile);
+ }
+ 			}
+ 
  			/*
  			 * We ignore the timeline part of the XLOG segment identifiers in
  			 * deciding whether a segment is still needed.	This ensures that
*** CleanupPriorWALFiles(void)
*** 113,122 
  			 * file. Note that this means files are not removed in the order
  			 * they were originally written, in case this worries you.
  			 */
! 			if (strlen(xlde->d_name) == XLOG_DATA_FNAME_LEN &&
! 			strspn(xlde->d_name, "0123456789ABCDEF") == XLOG_DATA_FNAME_LEN &&
! strcmp(xlde->d_name + 8, exclusiveCleanupFileName + 8) < 0)
  			{
  snprintf(WALFilePath, 

Re: [HACKERS] [pgsql-general 2011-1-21:] Are there any projects interested in object functionality? (+ rule bases)

[Nick Rudnick]

(my last two posts seemingly did not reach the HACKERS forum, so please 
let me resend the last one ;-) )


May I sum up?

o   in the recent there are no efforts known to experiment with 
reference types, methods, or rule inference on top of PostgreSQL -- 
advice that can be given mostly points to the given documented 
functionality


o   inside the PostgreSQL community, there is not many knowledge in 
circulation in regard of performance effects of using deeply nested data 
structures (with the possible exception of XML handling) or doing rule 
inference on top oof PostgreSQL -- but at least, there also are no 
substantial contraindications


o   extensions of PostgreSQL to support such a kind of usage have to be 
expected to be expected to be rejected from integration to the code base 
core -- i.e., if they are done, students have to be told «you can't 
expect this to become a part of PostgreSQL»


Is this understood correctly, especially the last point, or did 
Robert/Tom just specifically address syntactical conflicts (between 
schema and object semantics) with the point notation?


If not, it might be discouraging for lecture, as there might be interest 
to present something which at least might be imagined once to become a 
standard tool.


Otherwise, the striking lack of academical initiatives in the area of OO 
and rule inference on top of PostgreSQL appears to me as a demand to


a. check out academic sources, whether principle efficience issues of 
backend design discourage it so obviously that people do not even try it 
out


b. if this is not the case, to propose this professor to try to fill the 
gap... ;-) In this case, regarding method semantics extensions, avoiding 
conflicts with existent language constructs certainly will be 
preferable, as these will be small projects.


Cheers, Nick


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Implement genuine serializable isolation level.

[Heikki Linnakangas]

On 08.02.2011 05:03, Robert Haas wrote:

On Mon, Feb 7, 2011 at 5:11 PM, Heikki Linnakangas
  wrote:

Implement genuine serializable isolation level.


With gcc version 4.2.1 (Apple Inc. build 5664):

predicate.c: In function ‘CheckTargetForConflictsIn’:
predicate.c:3674: warning: ‘nexttargettag.locktag_field5’ may be used
uninitialized in this function
predicate.c:3674: warning: ‘nexttargettag.locktag_field4’ may be used
uninitialized in this function
predicate.c:3674: warning: ‘nexttargettag.locktag_field3’ may be used
uninitialized in this function
predicate.c:3674: warning: ‘nexttargettag.locktag_field2’ may be used
uninitialized in this function
predicate.c:3674: warning: ‘nexttargettag.locktag_field1’ may be used
uninitialized in this function


Thanks, fixed. I didn't get that warning on gcc 4.4.5, but apparently 
older gcc versions are more picky on that.


Also fixed a copy-pasto in my addition of pg_serial in the docs, pointed 
out by Kevin off-list.


--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Named restore points

[Jaime Casanova]

On Mon, Feb 7, 2011 at 10:59 PM, Robert Haas  wrote:
> On Fri, Feb 4, 2011 at 9:15 PM, Jaime Casanova  wrote:
>> ok, i will see you're reviewed version later today
>
> This patch is still marked as "Needs Review" in the CommitFest
> application, but I'm thinking perhaps it should be changed to Ready
> for Committer?  Are there any open issues?
>

only things i can found are:

> + static char recoveryStopNamedRestorePoint[MAXFNAMELEN];
>
> Is MAXFNAMELEN appropriate? AFAICS it is used for file name length. [Looking
> at code...] It seems to be used for backup label too so it is not so
> inappropriate.
>

which is just a question about if MAXFNAMELEN is the right length to use

and

> Finally, this is a nice feature iif we have a way to know what named restore
> points are available. DBAs need to take note of this list (that is not good)
> and the lazy ones will have a hard time to recover the right name (possibly
> with a xlog dump tool).
>
> So how could we store this information? Perhaps a file in
> $PGDATA/pg_xlog/restore_label that contains the label (and possibly the WAL
> location). Also it must have a way to transmit the restore_label when we add
> another restore point. I didn't implement this part (Jaime?) and it seems as
> important as the new xlog record type that is in the patch. It seems
> complicate but I don't have ideas. Anyone? The restore point names could be
> obtained by querying a function (say, pg_restore_point_names or
> pg_restore_point_list).
>

i still think this should be a separate tool or a dba written list,
the reason beign that with sql function we were not able track restore
points in archived segments... if you like i can try to build a simple
tool for this but don't think that is a showstopper, even without that
the feature is useful IMHO at least

-- 
Jaime Casanova         www.2ndQuadrant.com
Professional PostgreSQL: Soporte y capacitación de PostgreSQL

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: patch: fix performance problems with repated decomprimation of varlena values in plpgsql

[Pavel Stehule]

2011/2/8 Noah Misch :
> On Mon, Feb 07, 2011 at 11:16:18PM -0500, Robert Haas wrote:
>> So
>> can we just get rid of should_be_detoasted, and have exec_eval_datum()
>> or its callers instead test:
>>
>> !var->isnull && var->datatype->typbyval && var->datatype->typlen == -1
>> && VARATT_IS_EXTENDED(var->value)
>
> FWIW, this is what I meant by option 2 in my summary.
>
>> I haven't tested this, but it's not clear that'd be measurably slower
>> than checking a single Boolean.
>
> Pavel benchmarked this or something close, measuring a performance loss:
> http://archives.postgresql.org/message-id/aanlktikdhekc9r38w2ttzomdr8vdavanr3lhqfjke...@mail.gmail.com

I tested this in situation when Datum is detoasted on usage, not in
assignment. So impact will be less.

Regards

Pavel Stehule

>
> Tom also expressed concern over performance:
> http://archives.postgresql.org/message-id/24266.1295462...@sss.pgh.pa.us
>
> Not sure what's next.
>
> nm
>

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: patch: fix performance problems with repated decomprimation of varlena values in plpgsql

[Pavel Stehule]

2011/2/8 Robert Haas :
> On Sun, Feb 6, 2011 at 9:10 AM, Robert Haas  wrote:
>>> Suppose "foo" is toasted.  As the code stands in master, it gets detoasted 
>>> in
>>> text_lt().  Certainly we won't overwrite the source back in PL/pgSQL from 
>>> the
>>> detoast point in text_lt().
>>
>> Right, that much seems obvious...
>>
>>> Pavel's optimization requires that we identify the
>>> need to detoast the datum earlier and do so preemptively.
>>
>> I guess I need to look at the patch more.  I'm failing to understand
>> why that can't be done within one or two functions, without passing
>> around a new piece of state.
>
> So it looks like there are only two places that set
> should_be_detoasted to something other than false.
>
> plpgsql_exec_function does this:
>
>                    var->should_be_detoasted = !var->isnull &&
> !var->datatype->typbyval
>                                            && var->datatype->typlen == -1;
>
> And exec_assign_value does this, which appears to be the same test in
> a different guise:
>
>                if (!var->datatype->typbyval && !*isNull)
>                {
>                    var->freeval = true;
>                    var->should_be_detoasted = var->datatype->typlen == -1;
>                }
>
> Every other place where we set this flag, we appear to already know
> either that the value is null or that it can't be toasted anyhow.  So
> can we just get rid of should_be_detoasted, and have exec_eval_datum()
> or its callers instead test:
>
> !var->isnull && var->datatype->typbyval && var->datatype->typlen == -1
> && VARATT_IS_EXTENDED(var->value)
>
> I haven't tested this, but it's not clear that'd be measurably slower
> than checking a single Boolean.

Probably less than 1-3%. I tested situation where these checks was
when Datum was used. Assignment is less often, so some shortcut  there
these isn't not too important.

Pavel

p.s. There is lot of uneffectivity in assignment statement - different
typoid, typmod means IO cast, so 1-3% from these checks doesn't play a
big role.





>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Add ENCODING option to COPY

[Hitoshi Harada]

2011/2/8 Robert Haas :
> On Fri, Feb 4, 2011 at 1:54 PM, Hitoshi Harada  wrote:
>> 2011/2/5 Tom Lane :
>>> Hitoshi Harada  writes:
 2011/2/5 Tom Lane :
> Yeah, putting backend-only stuff into that header is a nonstarter.
>>>
 Do you mean you think it' all right to define
 pg_cached_encoding_conversion() in pg_conversion_fn.h?
>>>
>>> That seems pretty random too.  I still think you've designed this API
>>> badly and it'd be better to avoid exposing the FmgrInfo to callers
>>> by letting the function manage the cache internally.
>>
>> I'll try it in a few days, but only making struct that holds FmgrInfo
>> in a file local like tuplestorestate comes up with so far
>
> We've been through several iterations of this patch now and haven't
> come up with something committable.  I think it's time to mark this
> one Returned with Feedback and revisit this for 9.2.

As I've been thinking these days. The design isn't fixed yet even now.
Thanks for all the reviews.

Regards,


-- 
Hitoshi Harada

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pl/python explicit subtransactions

[Steve Singer]

On 11-02-06 11:40 AM, Jan Urbański wrote:


PFA an updated patch with documentation.



Yeah, changed them.


Those changes look fine.  The tests now pass.

I've attached a new version of the patch that fixes a few typos/wording 
issues I saw in the documentation.  I also changed the link to the 
python reference manual section on context managers. I think it is 
better to link to that versus the original PEP.


The documentation could probably still use more word-smithing but that 
can happen later.  I'm marking this as ready for a committer.






Thanks,
Jan


diff --git a/doc/src/sgml/plpython.sgml b/doc/src/sgml/plpython.sgml
index e05c293..9137ceb 100644
*** a/doc/src/sgml/plpython.sgml
--- b/doc/src/sgml/plpython.sgml
*** $$ LANGUAGE plpythonu;
*** 943,949 
  

  
!   
 Trapping Errors
  
 
--- 943,949 
  

  
!   
 Trapping Errors
  
 
*** $$ LANGUAGE plpythonu;
*** 969,974 
--- 969,1087 

   
  
+  
+   Explicit subtransactions
+   
+ Recovering from errors caused by database access as described
+ in  can lead to an undesirable situation
+ where some operations succeed before one of them fails and after recovering
+ from that error the data is left in an inconsistent state. PL/Python offers
+ a solution to this problem in the form of explicit subtransactions.
+   
+ 
+   
+Subtransaction context managers
+
+  Consider a function that implements a transfer between two accounts:
+ 
+ CREATE FUNCTION transfer_funds() RETURNS void AS $$
+ try:
+ plpy.execute("UPDATE accounts SET balance = balance - 100 WHERE account_name = 'joe'")
+ plpy.execute("UPDATE accounts SET balance = balance + 100 WHERE account_name = 'mary'")
+ except plpy.SPIError, e:
+ result = "error transferring funds: %s" % e.args
+ else:
+ result = "funds transferred correctly"
+ plpy.execute("INSERT INTO operations(result) VALUES ('%s')" % result)
+ $$ LANGUAGE plpythonu;
+ 
+  If the second UPDATE statement results in an exception
+  being raised, this function will report the error, but the result of the
+  first UPDATE will nevertheless be committed. In other
+  words, the funds will be withdrawn from Joe's account, but will not be
+  transferred to Mary's account.
+
+
+  To avoid such issues, you can wrap your plpy.execute
+  calls in an explicit subtransaction. The plpy module
+  provides a helper object to manage explicit subtransactions that gets
+  created with the plpy.subtransaction() function. 
+  Objects created by this function implement the http://www.python.org/doc//current/library/stdtypes.html#context-manager-types";>context manager interface.
+  
+  Using explicit subtransactions we can rewrite our function as:
+ 
+ CREATE FUNCTION transfer_funds2() RETURNS void AS $$
+ try:
+ with plpy.subtransaction():
+ plpy.execute("UPDATE accounts SET balance = balance - 100 WHERE account_name = 'joe'")
+ plpy.execute("UPDATE accounts SET balance = balance + 100 WHERE account_name = 'mary'")
+ except plpy.SPIError, e:
+ result = "error transferring funds: %s" % e.args
+ else:
+ result = "funds transferred correctly"
+ plpy.execute("INSERT INTO operations(result) VALUES ('%s')" % result)
+ $$ LANGUAGE plpythonu;
+ 
+  Note that the use of try/catch is still
+  required. Otherwise the exception would propagate to the top of the Python
+  stack and would cause the whole function to abort with
+  a PostgreSQL error.
+  The operations table would not have any row inserted
+  into it. The subtransaction context manager does not trap errors, it only
+  assures that all database operations executed inside its scope will be
+  atomically committed or rolled back.  A rollback of the subtransaction
+  block occurs on any kind of exception exit, not only ones caused by
+  errors originating from database access. A regular Python exception raised
+  inside an explicit subtransaction block would also cause the
+  subtransaction to be rolled back.
+
+
+  Another reason to use explicit subtransactions is that each
+  time plpy.execute or plpy.prepare is
+  used, it has to create its own internal subtransaction in order to be able
+  to recover from errors using the try/catch construct. If
+  you explicitly put your database access code in the scope of a
+  plpy.subtransaction() context manager, no additional
+  subtransactions will be created, which could improve the
+  performance of your function.
+
+   
+ 
+   
+Older Python versions
+ 
+
+  Context managers syntax using the with keyword was
+  implemented in Python 2.6. If using PL/Python with an older Python
+  version, it is still possible to use explicit subtransactions, although
+  not as transparently. The e

Re: [HACKERS] postponing some large patches to 9.2

[Steve Singer]

On 11-02-07 10:37 PM, Robert Haas wrote:


- The PL/python extravaganza.  I'm not really clear where we stand
with this.  There are a lot of patches here.



Some of the patches have been committed a few others are ready (or 
almost ready) for a committer.   The table function one  is the only one 
in 'waiting for author'


4 of the patches haven't yet received any review. Jan Urbanski has been 
pretty good about posting updated patches as the dependent patches get 
updated.  It would be good if a few people grabbed these. The individual 
patches tend to not be that large.






--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: patch: fix performance problems with repated decomprimation of varlena values in plpgsql

[Noah Misch]

On Mon, Feb 07, 2011 at 11:16:18PM -0500, Robert Haas wrote:
> So
> can we just get rid of should_be_detoasted, and have exec_eval_datum()
> or its callers instead test:
> 
> !var->isnull && var->datatype->typbyval && var->datatype->typlen == -1
> && VARATT_IS_EXTENDED(var->value)

FWIW, this is what I meant by option 2 in my summary.

> I haven't tested this, but it's not clear that'd be measurably slower
> than checking a single Boolean.

Pavel benchmarked this or something close, measuring a performance loss:
http://archives.postgresql.org/message-id/aanlktikdhekc9r38w2ttzomdr8vdavanr3lhqfjke...@mail.gmail.com

Tom also expressed concern over performance:
http://archives.postgresql.org/message-id/24266.1295462...@sss.pgh.pa.us

Not sure what's next.

nm

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

[Robert Haas]

On Mon, Feb 7, 2011 at 10:42 PM, Joachim Wieland  wrote:
>> i guess the huge amount of info is showing the patch is just for
>> debugging and will be removed before commit, right?
>
> That's right.

So how close are we to having a committable version of this?  Should
we push this out to 9.2?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Revised patches to add table function support to PL/Tcl (TODO item)

[Robert Haas]

On Tue, Dec 28, 2010 at 9:23 PM, Karl Lehenbauer
 wrote:
> On Dec 28, 2010, at 7:29 PM, Tom Lane wrote:
>> This patch appears to be changing a whole lot of stuff that in fact
>> pg_indent has never changed, so there's something wrong with the way you
>> are doing it.  It looks like a bad typedef list from here.
>
> You were right, Tom.  The problem was that typedefs "pltcl_interp_desc", 
> "pltcl_proc_key", and "pltcl_proc_ptr" weren't in 
> src/tools/pgindent/typedefs.list.  After adding them (and building and 
> installing the netbsd-based, patched indent), pgindent only changes a handful 
> of lines.
>
> pltcl-karl-try3-1-of-3-pgindent.patch patches typedefs.list with the three 
> missing typedefs and pltcl.c with the small changes made by pgindent (it 
> shifted some embedded comments left within their lines, mainly).
>
> As before, but "try3" now, pltcl-karl-try3-2-of-3-objects.patch converts 
> pltcl.c to use the "Tcl objects" C API.
>
> And as before, but "try3" now, pltcl-karl-try3-3-of-3-setof.patch adds 
> returning record and SETOF record.

This patch did not get reviewed, because the person who originally
planned to review it had a hardware failure that prevented him from
doing so.  Can anyone pick this up?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Tracking latest timeline in standby mode

[Robert Haas]

On Mon, Jan 24, 2011 at 2:00 AM, Fujii Masao  wrote:
> On Wed, Jan 5, 2011 at 5:08 AM, Heikki Linnakangas
>  wrote:
>> I finally got around to look at this. I wrote a patch to validate that the
>> TLI on xlog page header matches ThisTimeLineID during recovery, and noticed
>> quickly in testing that it doesn't catch all the cases I'd like to catch
>> :-(.
>
> The patch added into the CF hasn't solved this problem yet. Are you planning
> to solve it in 9.1? Or are you planning to just commit the patch for 9.1, and
> postpone the issue to 9.2 or later? I'm OK either way. Of course, the former
> is quite better, though.
>
> Anyway, you have to add the documentation about this feature.

This patch is erroneously marked Needs Review in the CommitFest
application, but I think really it's Waiting on Author, and has been
for a long time.  I'm thinking we should push this out to 9.2.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: pg_ctl failover Re: [HACKERS] Latches, signals, and waiting

[Robert Haas]

On Wed, Jan 19, 2011 at 1:01 AM, Fujii Masao  wrote:
> On Thu, Jan 13, 2011 at 9:08 PM, Fujii Masao  wrote:
>> I did s/failover/promote. Here is the updated patch.
>
> I rebased the patch to current git master.

This patch looks fine to me.  I will mark it Ready for Committer.

(Someone else please feel free to pick it up for the actual commit, if
you have cycles.)

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: patch: fix performance problems with repated decomprimation of varlena values in plpgsql

[Robert Haas]

On Sun, Feb 6, 2011 at 9:10 AM, Robert Haas  wrote:
>> Suppose "foo" is toasted.  As the code stands in master, it gets detoasted in
>> text_lt().  Certainly we won't overwrite the source back in PL/pgSQL from the
>> detoast point in text_lt().
>
> Right, that much seems obvious...
>
>> Pavel's optimization requires that we identify the
>> need to detoast the datum earlier and do so preemptively.
>
> I guess I need to look at the patch more.  I'm failing to understand
> why that can't be done within one or two functions, without passing
> around a new piece of state.

So it looks like there are only two places that set
should_be_detoasted to something other than false.

plpgsql_exec_function does this:

var->should_be_detoasted = !var->isnull &&
!var->datatype->typbyval
&& var->datatype->typlen == -1;

And exec_assign_value does this, which appears to be the same test in
a different guise:

if (!var->datatype->typbyval && !*isNull)
{
var->freeval = true;
var->should_be_detoasted = var->datatype->typlen == -1;
}

Every other place where we set this flag, we appear to already know
either that the value is null or that it can't be toasted anyhow.  So
can we just get rid of should_be_detoasted, and have exec_eval_datum()
or its callers instead test:

!var->isnull && var->datatype->typbyval && var->datatype->typlen == -1
&& VARATT_IS_EXTENDED(var->value)

I haven't tested this, but it's not clear that'd be measurably slower
than checking a single Boolean.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Add ENCODING option to COPY

[Robert Haas]

On Fri, Feb 4, 2011 at 1:54 PM, Hitoshi Harada  wrote:
> 2011/2/5 Tom Lane :
>> Hitoshi Harada  writes:
>>> 2011/2/5 Tom Lane :
 Yeah, putting backend-only stuff into that header is a nonstarter.
>>
>>> Do you mean you think it' all right to define
>>> pg_cached_encoding_conversion() in pg_conversion_fn.h?
>>
>> That seems pretty random too.  I still think you've designed this API
>> badly and it'd be better to avoid exposing the FmgrInfo to callers
>> by letting the function manage the cache internally.
>
> I'll try it in a few days, but only making struct that holds FmgrInfo
> in a file local like tuplestorestate comes up with so far

We've been through several iterations of this patch now and haven't
come up with something committable.  I think it's time to mark this
one Returned with Feedback and revisit this for 9.2.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Named restore points

[Robert Haas]

On Fri, Feb 4, 2011 at 9:15 PM, Jaime Casanova  wrote:
> ok, i will see you're reviewed version later today

This patch is still marked as "Needs Review" in the CommitFest
application, but I'm thinking perhaps it should be changed to Ready
for Committer?  Are there any open issues?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ALTER TABLE ... ADD FOREIGN KEY ... NOT ENFORCED

[Robert Haas]

On Thu, Feb 3, 2011 at 11:00 PM, Robert Haas  wrote:
> On Fri, Jan 14, 2011 at 6:15 AM, Simon Riggs  wrote:
>> Patch to implement the proposed feature attached, for CFJan2011.
>>
>> 2 sub-command changes:
>>
>> ALTER TABLE foo ADD FOREIGN KEY fkoo ... NOT VALID;
>>
>> ALTER TABLE foo VALIDATE CONSTRAINT fkoo;
>
> This patch, which seems to be the latest version, no longer applies,
> and has not been updated based on the previous provided review
> comments.
>
> Also, this diff hunk looks scary to me:
>
> +       if (must_use_query)
> +               ereport(ERROR,
> +                               (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
> +                                errmsg("cannot SELECT from primary
> key of relation \"%s\"",
> +                                               
> RelationGetRelationName(rel;
> +
>
> What's the justification for that?

Since this patch was reviewed on January 23rd by Marko Tiikkaja and
more briefly on February 3rd by me, and has not been updated, I am
marking it Returned with Feedback.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

[Joachim Wieland]

Hi Jaime,

thanks for your review!

On Sun, Feb 6, 2011 at 2:12 PM, Jaime Casanova  wrote:
> code review:
>
> something i found, and is a very simple one, is this warning (there's
> a similar issue in _StartMasterParallel with the buf variable)
> """
> pg_backup_directory.c: In function ‘_EndMasterParallel’:
> pg_backup_directory.c:856: warning: ‘status’ may be used uninitialized
> in this function
> """

Cool. My compiler didn't tell me about this.


> i guess the huge amount of info is showing the patch is just for
> debugging and will be removed before commit, right?

That's right.


> functional review:
>
> it works good most of the time, just a few points:
> - if i interrupt the process the connections stay, i guess it could
> catch the signal and finish the connections

Hm, well, recovering gracefully out of errors could be improved. In
your example you would signal the children implicitly because the
parent process dies and the pipes to the children would get broken as
well. Of course the parent could more actively terminate the children
but it might not be the best option to just kill them, as then there
will be a lot of "unexpected EOF" connections in the log. So if an
error condition comes up in the parent (as in your example, because
you canceled the process), then ideally the parent should signal the
children with a non-lethal signal and the children should catch this
"please terminate" signal and exit cleanly but as soon as possible. If
the error case comes up at the child however, then we'd need to make
sure that the user sees the error message from the child. This should
work well as-is but currently it could happen that the parent exists
before all of the children have exited. I'll investigate this a bit...


> - if i have an exclusive lock on a table and a worker starts dumping
> it, it fails because it can't take the lock but it just say "it was
> ok" and would prefer an error

I'm getting a clear

pg_dump: [Archivierer] could not lock table public.c: ERROR:  could
not obtain lock on relation "c"

but I'll look into this as well.

Regarding your other post:

> - there is no docs

True...

> - pg_dump and pg_restore are inconsistent:
>  pg_dump requires the directory to be provided with the -f option:
> pg_dump -Fd -f dir_dump
>  pg_restore pass the directory as an argument for -Fd: pg_restore -Fd dir_dump

Well, this is there with pg_dump and pg_restore currently as well. -F
is the switch for the format and it just takes "d" as the format. The
dir_dump is an option without any switch.

See the output for the --help switches:

Usage:
  pg_dump [OPTION]... [DBNAME]

Usage:
  pg_restore [OPTION]... [FILE]

So in either case you don't need to give a switch for what you have.
If you run pg_dump you don't give the switch for the database but you
need to give it for the output (-f) and with pg_restore you don't give
a switch for the file that you're restoring but you'd need to give -d
for restoring to a database.


Joachim

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] btree_gist (was: CommitFest progress - or lack thereof)

[Robert Haas]

On Fri, Feb 4, 2011 at 12:02 PM, Oleg Bartunov  wrote:
> Aha,
>
> Teodor sent it to the list Dec 28, see
> http://archives.postgresql.org/message-id/4D1A1677.80300%40sigaev.ru
>
> After a month I didn't see any activity on this patch, so I I added it to
> https://commitfest.postgresql.org/action/patch_view?id=350 Jan 21
>
> Now, I realised it was too late. Added to current commitfest.

I think this patch missed the deadline for the current CommitFest.
It's true that it was posted to the list in time, but it's just
madness to think we can do review in a meaningful way and get done in
a reasonable time if every patch that's ever been posted is fair game
to be added to the CommitFest at any point.  I believe it's a
generally accepted principle that adding things to the CommitFest
properly is the submitter's responsibility.

That having been said, this looks like a fairly mechanical change to a
contrib module that you and Teodor wrote.  So I'd say if you guys are
confident that it's correct, go ahead and commit.  If you need it
reviewed, or if you can't commit it in the next week or so, I think
it's going to have to wait for 9.2.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Where the Quals are actually 'List'ed

[Vaibhav Kaushal]

On Mon, 2011-02-07 at 11:16 -0500, Tom Lane wrote:
> Vaibhav Kaushal  writes:
> > Hi,
> > I find that ExecInitExpr creates a ExprState tree (and so say the
> > comments above the function in the source). Also, it seems to decide
> > which function would get called when the expression is to be evaluated
> > when ExecQual runs, by setting the function pointer, for example: 
> 
> > bstate->xprstate.evalfunc = (ExprStateEvalFunc)ExecEvalAnd;
> 
> > But ExecQual goes through a List, whereas, ExecInitExpr creates a tree. 
> > So is that same tree converted to a qual list or are we adding some more
> > information.
> 
> ExecInitExpr produces a List of state trees from a List of expr trees
> --- look at the last case in its switch.
> 
>   regards, tom lane
Thanks a lot for the help sir. Got the point :)

-Vaibhav(*_*)


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] postponing some large patches to 9.2

[Robert Haas]

On Mon, Feb 7, 2011 at 3:57 PM, Chris Browne  wrote:
> It sure looks to me like there are going to be a bunch of items that,
> based on the recognized policies, need to get deferred to 9.2, and the
> prospects for Sync Rep getting into 9.1 don't look notably good to me.
>
> It's definitely readily arguable that fairness requires that:
>
>  - Items not committable by 2011-02-15 be deferred to the 2011-Next fest
>
>   There are around 25 items right now that are sitting with [Waiting
>   for Author] and [Returned with Feedback] statuses.  They largely seem
>   like pretty fair game for "next fest."
>
>  - Large items that weren't included in the 2010-11 fest be considered
>   problematic to try to integrate into 9.1
>
>   There sure seem to be some large items in the 2011-01 fest, which I
>   thought wasn't supposed to be the case.

This discussion reveals that it's time to start making some
discussions about what can be accomplished for 9.1 and what must be
postponed to 9.2.  The big ones I think we should postpone are:

- Range Types.  This is a large patch which was submitted for the
first time to the last CommitFest of the cycle, and the first version
that had no open TODO items was posted yesterday, three-quarters of
the way through that last CommitFest.  Some good review has been done.
 While more is probably needed, I think we should feel good about
what's been accomplished and mark this one Returned with Feedback.

- ALTER EXTENSION UPGRADE.  This is another large patch which was
submitted for the first time to the last CommitFest of the cycle.  The
prerequisite patch to provide basic extension support has not been
committed yet, although it sounds like that will happen soon.
However, since that patch is undergoing a fair amount of surgery, it's
reasonably certain that this will require significant rebasing.  I
think it would also be an overstatement to say that we have consensus
on the design.  My feeling is that, unless Tom thinks that failing to
get this committed now is going to leave us with a mess later, we
should mark this one Returned with Feedback and revisit it for 9.2.

- FOR KEY LOCK tables.  This patch, unfortunately, has not gotten a
lot of review.  But it's a major, potentially destabilizing change
that was, like the last two, submitted for the first time to the last
CommitFest of the cycle.  Even if we assume for the sake of argument
that the code is unusually good for a feature of this type, I don't
think it's the right time to commit something like this.  I would
argue for putting this off until 9.2, though preferably with a bit
more review than it's gotten so far.

The other remaining "big" patches are:

- extension support for pg_upgrade.  Tom is planning to have this
committed within a day or so, per latest news.

- synchronous replication.  Based on some looking at this today, I am
somewhat doubtful about the possibility of me or anyone else beating
this completely into shape in time for 9.2, unless we choose to extend
the deadline by several weeks.  Simon said that he would have time to
finish this in the next two weeks, but, as noted, the CommitFest is
scheduled to be over in ONE week, and it looks to me like this is
still pretty rough.  However, there's a lot of good stuff in here, and
I think it might be practical to get some portion of it committed even
if we can't agree on all of it.  I recommend we give this one a little
more time to shake out before giving up on it.

- SQL/MED.  This patch unfortunately kind of stalled for a long time.
However, I believe that Heikki is now working actively on the core
patch, so I'm hopeful for some activity here soon.

- Writeable CTEs.  Tom has promised to look at this, but doesn't seem
to be in a hurry.

- Per-column collation.  This one has been lingering for a long time.
But Noah Misch recently did a pretty thorough review, and it's now
marked Ready for Committer.  Peter, are you planning to commit this?

- The PL/python extravaganza.  I'm not really clear where we stand
with this.  There are a lot of patches here.

There are a variety of smaller patches we need to make decisions
about, too.  But summarizing all of that here is going to be too much.
 I'll post to some of the other threads individually.

Thoughts on these?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Extensions support for pg_dump, patch v27

[Tom Lane]

Attached is an updated patch that incorporates all of the review I've
done so far on the core code.  This omits the contrib changes, which
I've not looked at in any detail, and the docs changes since I've not
yet updated the docs to match today's code changes.  User-visible
changes are that WITH NO DATA is gone, and instead there's
pg_extension_config_dump(regclass, text) as per today's discussion.
The latter is only lightly tested but seems to work as intended.

I believe the core code is now in pretty good shape; the only open issue
I have at the moment is that pg_dump will fail to suppress dumping of
USER MAPPING objects that are in an extension.  Which is something I'm
not too excited about anyway.  (The reason that's an issue is that I
reverted most of the changes in pg_dump.c in favor of using pg_dump's
already existing dependency mechanisms to suppress dumping unwanted
items.  The USER MAPPING code tries to bypass the DumpableObject
representation, which means it doesn't get filtered.)

The documentation still needs a good bit of work, but I hope to have
this committed within a day or so.

regards, tom lane



binrRC1pCyMTR.bin
Description: extensions.v31.patch.gz

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] We need to log aborted autovacuums

[Robert Haas]

On Sat, Feb 5, 2011 at 3:34 PM, Cédric Villemain
 wrote:
> Anyway, without GUC is fine too as it won't fill the /var/log itself !
> I am just not opposed to have new GUC in those areas (log && debug).

OK.  Committed without a new GUC, at least for now.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Implement genuine serializable isolation level.

[Robert Haas]

On Mon, Feb 7, 2011 at 5:11 PM, Heikki Linnakangas
 wrote:
> Implement genuine serializable isolation level.

With gcc version 4.2.1 (Apple Inc. build 5664):

predicate.c: In function ‘CheckTargetForConflictsIn’:
predicate.c:3674: warning: ‘nexttargettag.locktag_field5’ may be used
uninitialized in this function
predicate.c:3674: warning: ‘nexttargettag.locktag_field4’ may be used
uninitialized in this function
predicate.c:3674: warning: ‘nexttargettag.locktag_field3’ may be used
uninitialized in this function
predicate.c:3674: warning: ‘nexttargettag.locktag_field2’ may be used
uninitialized in this function
predicate.c:3674: warning: ‘nexttargettag.locktag_field1’ may be used
uninitialized in this function

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Varlena and binary

[Robert Haas]

On Mon, Feb 7, 2011 at 11:12 AM, Tom Lane  wrote:
> =?utf-8?q?Rados=C5=82aw_Smogura?=  writes:
>> I'm sending small patch for textsend. It reduces unnecessary copies, and
>> memory usage for duplication of varlena data. May you look?
>
> This code will break the day that text and bytea don't have the same
> internal representation, which seems likely to be soon.

Oh, really?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Error code for "terminating connection due to conflict with recovery"

[Robert Haas]

On Mon, Feb 7, 2011 at 4:11 PM, Simon Riggs  wrote:
>> So I guess the only remaining issue is whether we should distinguish
>> the error message text, as well as the error codes.  Tom was in favor
>> of that upthread, and I am too.  Right now we have:
> The error text is already differentiated by
> errdetail_recovery_conflict().

OK.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] exposing COPY API

[Andrew Dunstan]

On 02/07/2011 11:34 AM, Andrew Dunstan wrote:



On 02/04/2011 05:49 AM, Itagaki Takahiro wrote:

Here is a demonstration to support jagged input files. It's a patch
on the latest patch. The new added API is:

   bool NextLineCopyFrom(
 [IN] CopyState cstate,
 [OUT] char ***fields, [OUT] int *nfields, [OUT] Oid *tupleOid)

It just returns separated fields in the next line. Fortunately, I need
no extra code for it because it is just extracted from NextCopyFrom().

I'm willing to include the change into copy APIs,
but we still have a few issues. See below.



I've looked at this, and I think it will do what I want. I haven't had 
time to play with it, although I hope to soon.  AIUI, it basically 
hands back the raw parsed strings to the user, who then has the 
responsibility of constructing Datum and Nulls arrays to form the 
tuple.  That should be all I need. So +1 from me for including it. In 
fact, +10. And many thanks.



I think we need a better name though. NextCopyFromRawFields maybe.



Here is a patch against the latest revision of file_fdw to exercise this 
API. It includes some regression tests, and I think apart from one or 
two small details plus a requirement for documentation, is complete.


This work is also published at 



Here's an excerpt from the regression tests:

   CREATE FOREIGN TABLE jagged_text (
t   text[]
   ) SERVER file_server
   OPTIONS (format 'csv', filename
   '/home/andrew/pgl/pg_head/contrib/file_fdw/data/jagged.csv', header
   'true', textarray 'true');
   SELECT * FROM jagged_text;
 t
   
 {"one field"}
 {two,fields}
 {three,NULL,"fields of which one is null"}
 {"next line has no fields"}
 {}
   (5 rows)

   SELECT t[3] AS a, t[1] AS b, t[99] AS c  FROM jagged_text;
  a  |b| c
   -+-+---
 | one field   |
 | two |
 fields of which one is null | three   |
 | next line has no fields |
 | |
   (5 rows)

Overall, this API works quite nicely, and does exactly what I want, so 
again many thanks.


cheers

andrew


*** /dev/null
--- b/contrib/file_fdw/data/jagged.csv
***
*** 0 
--- 1,6 
+ header for a file with varying numbers of fields
+ one field
+ two,fields
+ three,,fields of which one is null
+ next line has no fields
+ 
*** a/contrib/file_fdw/file_fdw.c
--- b/contrib/file_fdw/file_fdw.c
***
*** 13,18 
--- 13,22 
  
  #include "postgres.h"
  
+ #include 
+ #include 
+ #include 
+ 
  #include "access/reloptions.h"
  #include "catalog/pg_foreign_table.h"
  #include "catalog/pg_foreign_server.h"
***
*** 26,31 
--- 30,36 
  #include "optimizer/cost.h"
  #include "parser/parsetree.h"
  #include "storage/fd.h"
+ #include "utils/array.h"
  #include "utils/builtins.h"
  
  PG_MODULE_MAGIC;
***
*** 61,78  static struct FileFdwOption valid_options[] = {
  
  	/* FIXME: implement force_not_null option */
  
! 	/* Centinel */
  	{ NULL,			InvalidOid }
  };
  
  /*
   * FDW-specific information for FdwExecutionState.
   */
  typedef struct FileFdwPrivate {
  	char		   *filename;
  	Relation		rel;		/* scan target relation */
! 	CopyState		cstate;		/* state of reaind file */
  	List		   *options;	/* merged generic options, excluding filename */
  } FileFdwPrivate;
  
  /*
--- 66,92 
  
  	/* FIXME: implement force_not_null option */
  
! 	/* Local option */
! 	{ "textarray",  ForeignTableRelationId },
! 
! 	/* Sentinel */
  	{ NULL,			InvalidOid }
  };
  
+ #define FILE_FDW_TEXTARRAY_STASH_INIT 64
  /*
   * FDW-specific information for FdwExecutionState.
   */
  typedef struct FileFdwPrivate {
  	char		   *filename;
+ 	booltextarray;  /* make a text array rather than a tuple */
  	Relation		rel;		/* scan target relation */
! 	CopyState		cstate;		/* state of read in file */
  	List		   *options;	/* merged generic options, excluding filename */
+ /* stash for processing text arrays - not used otherwise */
+ 	int text_array_stash_size;
+ 	Datum  *text_array_values;
+ 	bool   *text_array_nulls;
  } FileFdwPrivate;
  
  /*
***
*** 91,96  static void fileIterate(FdwExecutionState *festate, TupleTableSlot *slot);
--- 105,113 
  static void fileEndScan(FdwExecutionState *festate);
  static void fileReScan(FdwExecutionState *festate);
  
+ /* text array support */
+ static void makeTextArray(FileFdwPrivate *fdw_private,
+ 		   TupleTableSlot *slot, char **raw_fields, int nfields);
  /*
   * Helper functions
   */
***
*** 142,148  file_fdw_validator(PG_FUNCTION_

Re: [HACKERS] SSI patch version 14

[Kevin Grittner]

Heikki Linnakangas  wrote:
 
> Ok, committed.
 
Thanks for that, and for all the suggestions along the way!
 
Thanks also to Joe Conway for the review and partial commit in the
first CF for this release; to Jeff Davis for the reviews, pointing
out areas which needed work; and to Anssi Kääriäinen for testing
thoroughly enough to find some issues to fix!
 
And thanks especially to Dan Ports for joining me in this effort and
co-authoring this with me!  Besides the actual code contributions,
the ability to pound on the result with DBT-2 for days at a time was
invaluable.
 
WOO-HOO!
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SSI patch version 14

[Robert Haas]

On Mon, Feb 7, 2011 at 5:23 PM, Magnus Hagander  wrote:
> On Mon, Feb 7, 2011 at 23:14, Heikki Linnakangas
>  wrote:
>> On 06.02.2011 20:30, Kevin Grittner wrote:
>>>
>>> "Kevin Grittner"  wrote:
>>>
 I'm working on it now, and hope to have it all settled down today.
>>>
>>> Done and pushed to the git repo.  Branch serializable on:
>>>
>>> git://git.postgresql.org/git/users/kgrittn/postgres.git
>>>
>>> Heikki: I'm back to not having any outstanding work on the patch.
>>
>> Ok, committed.
>>
>> Thank you for all this! It has been a huge effort from you and Dan, and a
>> great feature as a result.
>
> +1!

Indeed, congratulations Kevin!

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SSI patch version 14

[Magnus Hagander]

On Mon, Feb 7, 2011 at 23:14, Heikki Linnakangas
 wrote:
> On 06.02.2011 20:30, Kevin Grittner wrote:
>>
>> "Kevin Grittner"  wrote:
>>
>>> I'm working on it now, and hope to have it all settled down today.
>>
>> Done and pushed to the git repo.  Branch serializable on:
>>
>> git://git.postgresql.org/git/users/kgrittn/postgres.git
>>
>> Heikki: I'm back to not having any outstanding work on the patch.
>
> Ok, committed.
>
> Thank you for all this! It has been a huge effort from you and Dan, and a
> great feature as a result.

+1!

(or more)

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Simon Riggs]

On Mon, 2011-02-07 at 11:50 -0800, Josh Berkus wrote:
> >> I just spoke to my manager at EnterpriseDB and he cleared my schedule
> >> for the next two days to work on this.  So I'll go hack on this now.
> >> I haven't read the patch yet so I don't know for sure how quite I'll
> >> be able to get up to speed on it, so if someone who is more familiar
> >> with this code wants to grab the baton away from me, feel free.
> >> Otherwise, I'll see what I can do with it.
> > 
> > Presumably you have a reason for declaring war? I'm sorry for that, I
> > really am.
> 
> How is clearing out his whole schedule to help review & fix the patch
> declaring war?  You have an odd attitude towards assistance, Simon.

It seems likely that Robert had not read my reply where I said I had
time to work on this project before posting. In that case, I withdraw my
comments and apologise.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/books/
 PostgreSQL Development, 24x7 Support, Training and Services
 


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SSI patch version 14

[Heikki Linnakangas]

On 06.02.2011 20:30, Kevin Grittner wrote:

"Kevin Grittner"  wrote:


I'm working on it now, and hope to have it all settled down today.


Done and pushed to the git repo.  Branch serializable on:

git://git.postgresql.org/git/users/kgrittn/postgres.git

Heikki: I'm back to not having any outstanding work on the patch.


Ok, committed.

Thank you for all this! It has been a huge effort from you and Dan, and 
a great feature as a result.


--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Spread checkpoint sync

[Greg Smith]

Kevin Grittner wrote:

There are occasional posts from those wondering why their read-only
queries are so slow after a bulk load, and why they are doing heavy
writes.  (I remember when I posted about that, as a relative newbie,
and I know I've seen others.)
  


Sure; I created http://wiki.postgresql.org/wiki/Hint_Bits a while back 
specifically to have a resource to explain that mystery to offer 
people.  But there's a difference between having a performance issue 
that people don't understand, and having a real bottleneck you can't get 
rid of.  My experience is that people who have hint bit issues run into 
them as a minor side-effect of a larger vacuum issue, and that if you 
get that under control they're only a minor detail in comparison.  Makes 
it hard to get too excited about optimizing them.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us
"PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Simon Riggs]

On Mon, 2011-02-07 at 15:25 -0500, Robert Haas wrote:

> I would certainly appreciate it if
> everyone could at least credit me with acting in good faith.

I think you are, if that helps.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/books/
 PostgreSQL Development, 24x7 Support, Training and Services
 


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Varlena and binary

[Radosław Smogura]

Just from curious may I ask in which direction this will go, and how this will 
affect performance of text and binary format?

Actually I started to make smaller improvements, and I think about one big to 
encode text (when client and server encoding are different) directly to 
StringInfo, without intermediate buffer.

Thanks in advice
Radek

Tom Lane  Monday 07 February 2011 17:12:07
> =?utf-8?q?Rados=C5=82aw_Smogura?=  writes:
> > I'm sending small patch for textsend. It reduces unnecessary copies, and
> > memory usage for duplication of varlena data. May you look?
> 
> This code will break the day that text and bytea don't have the same
> internal representation, which seems likely to be soon.  Barring some
> compelling evidence of a major performance improvement obtainable this
> way, I don't think we want this patch.
> 
>   regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Chris Browne]

dp...@pgadmin.org (Dave Page) writes:
> On Mon, Feb 7, 2011 at 6:55 PM, Robert Haas  wrote:
>> On Mon, Feb 7, 2011 at 12:43 PM, Tom Lane  wrote:
>>> Robert Haas  writes:
 ... Well, the current CommitFest ends in one week, ...
>>>
>>> Really?  I thought the idea for the last CF of a development cycle was
>>> that it kept going till we'd dealt with everything.  Arbitrarily
>>> rejecting stuff we haven't dealt with doesn't seem fair.
>>
>> Uh, we did that with 8.4 and it was a disaster.  The CommitFest lasted
>> *five months*. We've been doing schedule-based CommitFests ever since
>> and it's worked much better.
>
> Rejecting stuff because we haven't gotten round to dealing with it in
> such a short period of time is a damn good way to limit the number of
> contributions we get. I don't believe we've agreed at any point that
> the last commitfest should be the same time length as the others (when
> we originally came up with the commitfest idea, it certainly wasn't
> expected), and deciding that without giving people advanced notice is
> a really good way to piss them off and encourage them to go work on
> other things.
>
> If we're going to put a time limit on this - and I think we should -
> we should publish a date ASAP, that gives everyone a fair chance to
> finish their work - say, 4 weeks.
>
> Then, if we want to make the last commitfest the same length as the
> others next year, we can make that decision and document those plans.

There *is* a problem that there doesn't seem to be enough time to
readily allow development of larger features without people getting
stuck fighting with the release periods.  But that's not the problem
taking place here.  It was documented, last May, that the final
CommitFest for 9.1 was to complete 2011-02-15, and there did seem to be
agreement on that.

It sure looks to me like there are going to be a bunch of items that,
based on the recognized policies, need to get deferred to 9.2, and the
prospects for Sync Rep getting into 9.1 don't look notably good to me.

Looking at things statistically, the 9.1 commitfests have had the
following numbers of items:

  #1 - 2010-09 - 52, of which 26 were committed
  #2 - 2010-11 - 43, of which 23 were committed
  #3 - 2011-01 - 98, of which 35 have been committed, and 10 are
 considered ready to commit.

It may appear unfair to not offer everyone a "fair chance to finish
their work," but it's not as if the date wasn't published Plenty Long
Ago. and well-publicized.

But deferring the end of the CommitFest would be Not Fair to those that
*did* get their proposed changes ready for the preceding Fests.  We
cannot evade unfairness.

It's definitely readily arguable that fairness requires that:

 - Items not committable by 2011-02-15 be deferred to the 2011-Next fest

   There are around 25 items right now that are sitting with [Waiting
   for Author] and [Returned with Feedback] statuses.  They largely seem
   like pretty fair game for "next fest."

 - Large items that weren't included in the 2010-11 fest be considered
   problematic to try to integrate into 9.1

   There sure seem to be some large items in the 2011-01 fest, which I
   thought wasn't supposed to be the case.

We shouldn't just impose policy for the sake of imposing policy, but I
do recall Really Long CommitFests being pretty disastrous.  And there's
*SO* much outstanding in this particular fest that it's getting past
time for doing some substantial triage so that reviewer attentions may
be directed towards the items most likely to be acceptable for 9.1.

I hate to think that 9.1 won't include Simon's SR material, but that may
have to be.
-- 
http://www3.sympatico.ca/cbbrowne/slony.html
"It's a pretty rare beginner who isn't clueless.  If beginners weren't
clueless, the infamous Unix learning cliff wouldn't be a problem."
-- david parsons

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[Tom Lane]

Dimitri Fontaine  writes:
> Tom Lane  writes:
>> If you're worried about that, then it's questionable whether ALTER
>> EXTENSION SET SCHEMA makes sense at all, ever.  I don't see any reason
>> to think that an extension is more fragile for this purpose than any
>> other random SQL dependencies.  Also, an extension being relocatable
>> doesn't seem to me to guarantee that it can cope with its dependencies
>> moving around; they're really independent properties.

> Well a relocatable extension certainly supports SET SCHEMA just fine, or
> it would not have the property.  Then your conclusion is right.  My
> comment was about what happens when things are setup the other way.

Yes, I understood that.

> We have earthdistance that depends on cube.  Let's pretend that
> earthdistance is not relocatable.  I think we should then consider (when
> both are installed) that cube is not relocatable, whatever its control
> file says.  That's because not relocatable means that the install script
> is using the @extschema@ place holder and the fragility there is known
> quite high: the install script and some installed objects do depend on
> @extschema@.

But @extschema@ isn't affected by where the other modules are.

The basic issue here is that we might have wired something about the
referenced schemas into one of the objects in the dependent extension,
for example via

create function foo() ... SET search_path FROM CURRENT;

I agree that this is a risk.  My point is that you can do that without
any extension, and you're just as much at risk.  If you think that this
is something we must protect against, I think ripping out ALTER
EXTENSION SET SCHEMA altogether is the only real answer.  I see no
argument whatsoever why we should lock down extensions and only extensions
against this risk.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Error code for "terminating connection due to conflict with recovery"

[Simon Riggs]

On Wed, 2011-02-02 at 21:21 -0500, Robert Haas wrote:
> On Tue, Feb 1, 2011 at 3:17 AM, Simon Riggs  wrote:
> > ERRCODE_DATABASE_DROPPED57P04   looks best
> 
> So I guess the only remaining issue is whether we should distinguish
> the error message text, as well as the error codes.  Tom was in favor
> of that upthread, and I am too.  Right now we have:
> 
> else if (RecoveryConflictPending && RecoveryConflictRetryable)
> {
> 
> pgstat_report_recovery_conflict(RecoveryConflictReason);
> ereport(FATAL,
> 
> (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
>   errmsg("terminating connection due to
> conflict with recovery"),
>  errdetail_recovery_conflict()));
> }
> else if (RecoveryConflictPending)
> {
> /* Currently there is only one non-retryable
> recovery conflict */
> Assert(RecoveryConflictReason ==
> PROCSIG_RECOVERY_CONFLICT_DATABASE);
> 
> pgstat_report_recovery_conflict(RecoveryConflictReason);
> ereport(FATAL,
> (errcode(ERRCODE_DATABASE_DROPPED),
>   errmsg("terminating connection due to
> conflict with recovery"),
>  errdetail_recovery_conflict()));
> }
> 
> The simplest thing to do seems to be to make the second one read
> "terminating connection because the database has been dropped".

The error text is already differentiated by
errdetail_recovery_conflict().

-- 
 Simon Riggs   http://www.2ndQuadrant.com/books/
 PostgreSQL Development, 24x7 Support, Training and Services
 


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Dave Page]

On Mon, Feb 7, 2011 at 9:46 PM, Robert Haas  wrote:
> On Mon, Feb 7, 2011 at 3:34 PM, Dave Page  wrote:
>> On Mon, Feb 7, 2011 at 9:25 PM, Robert Haas  wrote:
>>> You're moving the bar.  It DOES say that the CommitFest will end on
>>> February 15th.  Now, if we want to have a discussion about changing
>>> that, let's have that discussion (perhaps on a thread where the
>>> subject has something to do with the topic), but we DID talk about
>>> this, it WAS agreed, and it's been sitting there on the wiki for
>>> something like 8 months.  Obviously, there will continue to be
>>> polishing after the CommitFest is over, but that's not the same thing
>>> as saying we're going to lengthen the CommitFest itself.
>>
>> I'm not moving the bar - I'm talking practically. Regardless of when
>> we consider the commitfest itself over, development and commit work of
>> new features has always continued until beta 1, and that has not
>> changed as far as I'm aware.
>
> I don't think that's really true.  Go back and read the output of 'git
> log REL9_0_BETA1'.  It's bug fixes, rearrangements of things that were
> committed but turned out to be controversial, documentation work,
> release note editing, pgindent crap...  sure, it wasn't a totally hard
> freeze, but it was pretty solid slush.  We did a good job not letting
> things drag out, and FWIW I think that was a good decision.  I don't
> remember too many people being unhappy about their patches getting
> punted, either.  There were one or two, but generally we punted things
> that needed major rework or just weren't getting updated in a timely
> fashion, and that, combined with a lot of hard work on Tom's part
> among others, worked fine.

I guess we disagree on what we consider to be "development" then. Just
looking back to April, I see various committers whacking things around
that look to me like the fine tuning and completion of earlier
patches.

Oh - and just so we're clear... I too want us to get the release out
promptly, I'm just concerned that we don't blindside developers.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Robert Haas]

On Mon, Feb 7, 2011 at 3:34 PM, Dave Page  wrote:
> On Mon, Feb 7, 2011 at 9:25 PM, Robert Haas  wrote:
>> You're moving the bar.  It DOES say that the CommitFest will end on
>> February 15th.  Now, if we want to have a discussion about changing
>> that, let's have that discussion (perhaps on a thread where the
>> subject has something to do with the topic), but we DID talk about
>> this, it WAS agreed, and it's been sitting there on the wiki for
>> something like 8 months.  Obviously, there will continue to be
>> polishing after the CommitFest is over, but that's not the same thing
>> as saying we're going to lengthen the CommitFest itself.
>
> I'm not moving the bar - I'm talking practically. Regardless of when
> we consider the commitfest itself over, development and commit work of
> new features has always continued until beta 1, and that has not
> changed as far as I'm aware.

I don't think that's really true.  Go back and read the output of 'git
log REL9_0_BETA1'.  It's bug fixes, rearrangements of things that were
committed but turned out to be controversial, documentation work,
release note editing, pgindent crap...  sure, it wasn't a totally hard
freeze, but it was pretty solid slush.  We did a good job not letting
things drag out, and FWIW I think that was a good decision.  I don't
remember too many people being unhappy about their patches getting
punted, either.  There were one or two, but generally we punted things
that needed major rework or just weren't getting updated in a timely
fashion, and that, combined with a lot of hard work on Tom's part
among others, worked fine.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] PgEast 2011: 3 days until close of CFP

[Joshua D. Drake]

Hello hackers,

Just FYI, the CFP for PgEast in NYC closes in three days.

https://www.postgresqlconference.org/talk_types

Sincerely,

JD
-- 
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering
http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Robert Haas]

On Mon, Feb 7, 2011 at 3:14 PM, Dimitri Fontaine  wrote:
> Robert Haas  writes:
>> I'm not trying to bypass compromising, and I don't know what makes you
>> think otherwise.  I am trying to ensure that the CommitFest wraps up
>
> Well, I'm too tired to allow myself posting such comments, sorry to have
> left the previous mail through.

Thanks, I understand.

> More than one commit fest saw its time
> frame extended for 1 or 2 weeks already, I think, all I'm saying is that
> this one will certainly not be an exception, and that's for the best.

We've actually done really well.  The last CommitFest in 9.0 wrapped
up on 2/17 (two days late), and the others were mostly right on time
as well.  The CommitFests for 9.1 ended on: 8/15 (on time), 10/26 (9
days late, but there was no activity on the last two of those days, so
say 7 days late), and 12/21 (six days late).  As far as I can tell,
the difference primarily has to do with who manages the CommitFests
and how aggressively they follow up on patches that are dropped.  The
last CommitFest we have that really ran late was the final CommitFest
of the 8.4 cycle, and it was that event that led me to accept Josh
Berkus's invitation to be a CF manager for the first 9.0 CommitFest.
Because five month CommitFests with the tree frozen are awful and
sucky for everyone except the people who are getting extra time to
finish their patches, and they aren't really that great for those
people either.

As far as I am concerned, everything from now until we've released a
stable beta with no known issues is time that I can't spend doing
development.  So I'd like to minimize that time - not by arbitrarily
throwing patches out the window - but by a combination of postponing
patches that are not done and working my ass off to finish as much as
possible.

> Be sure I appreciate the efforts you're putting into the mix!

Thanks.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Dave Page]

On Mon, Feb 7, 2011 at 9:25 PM, Robert Haas  wrote:
> You're moving the bar.  It DOES say that the CommitFest will end on
> February 15th.  Now, if we want to have a discussion about changing
> that, let's have that discussion (perhaps on a thread where the
> subject has something to do with the topic), but we DID talk about
> this, it WAS agreed, and it's been sitting there on the wiki for
> something like 8 months.  Obviously, there will continue to be
> polishing after the CommitFest is over, but that's not the same thing
> as saying we're going to lengthen the CommitFest itself.

I'm not moving the bar - I'm talking practically. Regardless of when
we consider the commitfest itself over, development and commit work of
new features has always continued until beta 1, and that has not
changed as far as I'm aware.

> I think, though, that we need to be explicit about what we're doing,
> and why we're doing it.  I have been working hard on this CommitFest
> for a long time (since approximately a month before it started) at the
> cost of development projects I would have liked to have worked on,
> because I knew we were going to be overwhelmed with patches.  I have
> helped as many people as I can with as many patches as I have been
> able to.  I think that finishing on time (or at least as close to on
> time as we can manage) is important to our success as a development
> community, just as having good features is.  We don't have to agree on
> what the best thing to do is, but I would certainly appreciate it if
> everyone could at least credit me with acting in good faith.

Oh, I have absolutely no doubt you're working in good faith, and
personally I thank you for the hard work you've put in. I just
disagree with your interpretation of the timetable.


-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Joshua D. Drake]

On Mon, 2011-02-07 at 12:24 -0800, Josh Berkus wrote:

> +1.
> 
> I, for one, would vote against extending beta if Sync Rep isn't ready
> yet.  There's plenty of other "big features" in 9.1, and Sync Rep will
> benefit from having additional development time given the number of
> major spec points we only cleared up a few weeks ago.
> 
> I think the majority of our users would prefer a 9.1 in May to one that
> has Sync Rep and is delivered in September.  If they had a choice.
> 

+1

JD

-- 
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering
http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Robert Haas]

On Mon, Feb 7, 2011 at 3:06 PM, Dave Page  wrote:
>>> Rejecting stuff because we haven't gotten round to dealing with it in
>>> such a short period of time is a damn good way to limit the number of
>>> contributions we get. I don't believe we've agreed at any point that
>>> the last commitfest should be the same time length as the others
>>
>> News to me.
>>
>> http://wiki.postgresql.org/wiki/PostgreSQL_9.1_Development_Plan
>
> Yes, and? It doesn't say beta 1 at the after a month of the last
> commitfest, which is the milestone which marks the end of development.
> It says alpha 4, and possibly more alphas. It's pretty clear that it
> is expected that development and polishing will continue past the 20th
> February.

You're moving the bar.  It DOES say that the CommitFest will end on
February 15th.  Now, if we want to have a discussion about changing
that, let's have that discussion (perhaps on a thread where the
subject has something to do with the topic), but we DID talk about
this, it WAS agreed, and it's been sitting there on the wiki for
something like 8 months.  Obviously, there will continue to be
polishing after the CommitFest is over, but that's not the same thing
as saying we're going to lengthen the CommitFest itself.

I think we need to step back a few paces here and talk about what
we're trying to accomplish by making some change to the proposed and
agreed CommitFest schedule.  If there's a concern that some patches
haven't been thoroughly reviewed at this point, then I think that's a
fair concern, and let's talk about which ones they are and see what we
can do about it.  I don't believe that's the case, and it's certainly
not the case for sync rep, which was submitted in an unpolished state
by Simon's own admission, reviewed and discussed, then sat for three
weeks without an update.  So perhaps the concern is that sync rep is a
make or break for this release.  OK, then fine, let's talk about
whether it's worth slipping the release for that feature.  I have no
problem with either of those conversations, and I'm happy to offer my
opinions and listen to the opinions of others, and we can make some
decision.

I think, though, that we need to be explicit about what we're doing,
and why we're doing it.  I have been working hard on this CommitFest
for a long time (since approximately a month before it started) at the
cost of development projects I would have liked to have worked on,
because I knew we were going to be overwhelmed with patches.  I have
helped as many people as I can with as many patches as I have been
able to.  I think that finishing on time (or at least as close to on
time as we can manage) is important to our success as a development
community, just as having good features is.  We don't have to agree on
what the best thing to do is, but I would certainly appreciate it if
everyone could at least credit me with acting in good faith.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Josh Berkus]

On 2/7/11 11:41 AM, Robert Haas wrote:
> However, I don't want to prolong
> the CommitFest indefinitely in the face of patches that the authors
> are not actively working on or can't finish in the time available, or
> where there is no consensus that the proposed change is what we want.
> I believe that this, too, is a generally accepted principle in our
> community, not something I just made up.

+1.

I, for one, would vote against extending beta if Sync Rep isn't ready
yet.  There's plenty of other "big features" in 9.1, and Sync Rep will
benefit from having additional development time given the number of
major spec points we only cleared up a few weeks ago.

I think the majority of our users would prefer a 9.1 in May to one that
has Sync Rep and is delivered in September.  If they had a choice.

Speaking of which, time to do some reviewing ...

-- 
  -- Josh Berkus
 PostgreSQL Experts Inc.
 http://www.pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Dimitri Fontaine]

Robert Haas  writes:
> I'm not trying to bypass compromising, and I don't know what makes you
> think otherwise.  I am trying to ensure that the CommitFest wraps up

Well, I'm too tired to allow myself posting such comments, sorry to have
left the previous mail through.  More than one commit fest saw its time
frame extended for 1 or 2 weeks already, I think, all I'm saying is that
this one will certainly not be an exception, and that's for the best.

Be sure I appreciate the efforts you're putting into the mix!

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Dave Page]

On Mon, Feb 7, 2011 at 8:59 PM, Robert Haas  wrote:
> On Mon, Feb 7, 2011 at 2:56 PM, Dave Page  wrote:
>> On Mon, Feb 7, 2011 at 6:55 PM, Robert Haas  wrote:
>>> On Mon, Feb 7, 2011 at 12:43 PM, Tom Lane  wrote:
 Robert Haas  writes:
> ... Well, the current CommitFest ends in one week, ...

 Really?  I thought the idea for the last CF of a development cycle was
 that it kept going till we'd dealt with everything.  Arbitrarily
 rejecting stuff we haven't dealt with doesn't seem fair.
>>>
>>> Uh, we did that with 8.4 and it was a disaster.  The CommitFest lasted
>>> *five months*. We've been doing schedule-based CommitFests ever since
>>> and it's worked much better.
>>
>> Rejecting stuff because we haven't gotten round to dealing with it in
>> such a short period of time is a damn good way to limit the number of
>> contributions we get. I don't believe we've agreed at any point that
>> the last commitfest should be the same time length as the others
>
> News to me.
>
> http://wiki.postgresql.org/wiki/PostgreSQL_9.1_Development_Plan

Yes, and? It doesn't say beta 1 at the after a month of the last
commitfest, which is the milestone which marks the end of development.
It says alpha 4, and possibly more alphas. It's pretty clear that it
is expected that development and polishing will continue past the 20th
February.


-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Kevin Grittner]

Robert Haas  wrote:
> Dave Page  wrote:
>> Robert Haas  wrote:
>>> Tom Lane  wrote:
 Robert Haas  writes:
> ... Well, the current CommitFest ends in one week, ...

 Really?  I thought the idea for the last CF of a development
 cycle was that it kept going till we'd dealt with everything. 
 Arbitrarily rejecting stuff we haven't dealt with doesn't seem
 fair.
>>>
>>> Uh, we did that with 8.4 and it was a disaster.  The CommitFest
>>> lasted *five months*. We've been doing schedule-based
>>> CommitFests ever since and it's worked much better.
>>
>> Rejecting stuff because we haven't gotten round to dealing with
>> it in such a short period of time is a damn good way to limit the
>> number of contributions we get. I don't believe we've agreed at
>> any point that the last commitfest should be the same time length
>> as the others
> 
> News to me.
> 
> http://wiki.postgresql.org/wiki/PostgreSQL_9.1_Development_Plan
 
I believe that with tighter management of the process, it should be
possible to reduce the average delay between someone writing a
feature and that feature appearing in a production release by about
two months without compromising quality.  Getting hypothetical for a
moment, delaying release of 50 features for two months to allow
release of one feature ten months earlier is likely to frustrate a
lot more people than having the train leave the station on time and
putting that one feature into the next release.
 
My impression was that Robert is trying to find a way to help get
Simon's patch into this release without holding everything up for
it.  In my book, that's not a declaration of war; it's community
spirit.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[Dimitri Fontaine]

Tom Lane  writes:
> If you're worried about that, then it's questionable whether ALTER
> EXTENSION SET SCHEMA makes sense at all, ever.  I don't see any reason
> to think that an extension is more fragile for this purpose than any
> other random SQL dependencies.  Also, an extension being relocatable
> doesn't seem to me to guarantee that it can cope with its dependencies
> moving around; they're really independent properties.

Well a relocatable extension certainly supports SET SCHEMA just fine, or
it would not have the property.  Then your conclusion is right.  My
comment was about what happens when things are setup the other way.

We have earthdistance that depends on cube.  Let's pretend that
earthdistance is not relocatable.  I think we should then consider (when
both are installed) that cube is not relocatable, whatever its control
file says.  That's because not relocatable means that the install script
is using the @extschema@ place holder and the fragility there is known
quite high: the install script and some installed objects do depend on
@extschema@.  Moving the dependencies underneath it in this case looks
to me more than a risk: we know we're breaking things.

What you're saying (or what I'm reading at least) is that if
earthdistance is relocatable, you don't have faith that it means we can
actually move cube without collateral damages.  Well, the author said it
would cope fine, and in this case I see no reason not to believe him.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SQL/MED - file_fdw

[Andrew Dunstan]

On 02/07/2011 01:39 AM, Itagaki Takahiro wrote:




file_fdw uses CopyFromErrorCallback() to give errors the proper context.  The
function uses template strings like "COPY %s, line %d", where %s is the name of
the relation being copied.  Presumably file_fdw and other features using this
API would wish to customize that error message prefix, and the relation name
might not be apropos at all.  How about another argument to BeginCopyFrom,
specifying an error prefix to be stashed in the CopyState?

I changed "COPY %s, .." to "relation %s, ..." because the first string is
the relation name anyway. We could have another prefix argument, but I think
it has little information for errors.

We also have many "COPY" in other messages, but they won't be used actually
because the are messages for invalid arguments and file_fdw will have own
validater function. All invalid arguments will be filtered in CREATE commands.



These changes have broken the regression tests. The attached patches 
(one for the core regression tests and one for file_fdw) fix that.


But I don't know that your change is terribly helpful. I rather like 
Noah's idea better, if we need to make a change.


cheers

andrew
diff --git a/contrib/file_fdw/output/file_fdw.source b/contrib/file_fdw/output/file_fdw.source
index f8ce4ca..7021ad8 100644
--- a/contrib/file_fdw/output/file_fdw.source
+++ b/contrib/file_fdw/output/file_fdw.source
@@ -89,7 +89,7 @@ SELECT * FROM agg_csv c JOIN agg_text t ON (t.a = c.a) ORDER BY c.a;
 -- error context report tests
 SELECT * FROM agg_bad;   -- ERROR
 ERROR:  invalid input syntax for type real: "aaa"
-CONTEXT:  COPY agg_bad, line 3, column b: "aaa"
+CONTEXT:  relation agg_bad, line 3, column b: "aaa"
 -- misc query tests
 \t on
 EXPLAIN (VERBOSE, COSTS FALSE) SELECT * FROM agg_csv;
diff --git a/src/test/regress/expected/alter_table.out b/src/test/regress/expected/alter_table.out
index 3280065..eea0b09 100644
--- a/src/test/regress/expected/alter_table.out
+++ b/src/test/regress/expected/alter_table.out
@@ -1002,7 +1002,7 @@ copy test("pg.dropped.1") to stdout;
 ERROR:  column "pg.dropped.1" of relation "test" does not exist
 copy test from stdin;
 ERROR:  extra data after last expected column
-CONTEXT:  COPY test, line 1: "10	11	12"
+CONTEXT:  relation test, line 1: "10	11	12"
 select * from test;
  b | c 
 ---+---
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 15cbe02..d9630f8 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -35,17 +35,17 @@ ERROR:  column "d" specified more than once
 -- missing data: should fail
 COPY x from stdin;
 ERROR:  invalid input syntax for integer: ""
-CONTEXT:  COPY x, line 1, column a: ""
+CONTEXT:  relation x, line 1, column a: ""
 COPY x from stdin;
 ERROR:  missing data for column "e"
-CONTEXT:  COPY x, line 1: "2000	230	23	23"
+CONTEXT:  relation x, line 1: "2000	230	23	23"
 COPY x from stdin;
 ERROR:  missing data for column "e"
-CONTEXT:  COPY x, line 1: "2001	231	\N	\N"
+CONTEXT:  relation x, line 1: "2001	231	\N	\N"
 -- extra data: should fail
 COPY x from stdin;
 ERROR:  extra data after last expected column
-CONTEXT:  COPY x, line 1: "2002	232	40	50	60	70	80"
+CONTEXT:  relation x, line 1: "2002	232	40	50	60	70	80"
 -- various COPY options: delimiters, oids, NULL string
 COPY x (b, c, d, e) from stdin with oids delimiter ',' null 'x';
 COPY x from stdin WITH DELIMITER AS ';' NULL AS '';
diff --git a/src/test/regress/expected/domain.out b/src/test/regress/expected/domain.out
index 7d72791..daadcef 100644
--- a/src/test/regress/expected/domain.out
+++ b/src/test/regress/expected/domain.out
@@ -49,7 +49,7 @@ INSERT INTO basictest values ('88', 'haha', 'short', '123.1212');-- Truncate
 -- Test copy
 COPY basictest (testvarchar) FROM stdin; -- fail
 ERROR:  value too long for type character varying(5)
-CONTEXT:  COPY basictest, line 1, column testvarchar: "notsoshorttext"
+CONTEXT:  relation basictest, line 1, column testvarchar: "notsoshorttext"
 COPY basictest (testvarchar) FROM stdin;
 select * from basictest;
  testint4 | testtext | testvarchar | testnumeric 
@@ -130,7 +130,7 @@ select testint4arr[1], testchar4arr[2:2] from domarrtest;
 COPY domarrtest FROM stdin;
 COPY domarrtest FROM stdin;	-- fail
 ERROR:  value too long for type character varying(4)
-CONTEXT:  COPY domarrtest, line 1, column testchar4arr: "{qwerty,w,e}"
+CONTEXT:  relation domarrtest, line 1, column testchar4arr: "{qwerty,w,e}"
 select * from domarrtest;
   testint4arr  |testchar4arr 
 ---+-
@@ -173,14 +173,14 @@ INSERT INTO nulltest values ('a', 'b', 'c', NULL, 'd'); -- Good
 -- Test copy
 COPY nulltest FROM stdin; --fail
 ERROR:  null value in column "col3" violates not-null constraint
-CONTEXT:  COPY nulltest, line 1: "a	b	\N	d	d"
+CONTEXT:  relation nulltest, line 1: "a	b	\N	d	d"
 COPY nulltest FROM stdin; --fail
 ERROR:  d

Re: [HACKERS] Sync Rep for 2011CF1

[Dave Page]

On Mon, Feb 7, 2011 at 6:55 PM, Robert Haas  wrote:
> On Mon, Feb 7, 2011 at 12:43 PM, Tom Lane  wrote:
>> Robert Haas  writes:
>>> ... Well, the current CommitFest ends in one week, ...
>>
>> Really?  I thought the idea for the last CF of a development cycle was
>> that it kept going till we'd dealt with everything.  Arbitrarily
>> rejecting stuff we haven't dealt with doesn't seem fair.
>
> Uh, we did that with 8.4 and it was a disaster.  The CommitFest lasted
> *five months*. We've been doing schedule-based CommitFests ever since
> and it's worked much better.

Rejecting stuff because we haven't gotten round to dealing with it in
such a short period of time is a damn good way to limit the number of
contributions we get. I don't believe we've agreed at any point that
the last commitfest should be the same time length as the others (when
we originally came up with the commitfest idea, it certainly wasn't
expected), and deciding that without giving people advanced notice is
a really good way to piss them off and encourage them to go work on
other things.

If we're going to put a time limit on this - and I think we should -
we should publish a date ASAP, that gives everyone a fair chance to
finish their work - say, 4 weeks.

Then, if we want to make the last commitfest the same length as the
others next year, we can make that decision and document those plans.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Robert Haas]

On Mon, Feb 7, 2011 at 2:56 PM, Dave Page  wrote:
> On Mon, Feb 7, 2011 at 6:55 PM, Robert Haas  wrote:
>> On Mon, Feb 7, 2011 at 12:43 PM, Tom Lane  wrote:
>>> Robert Haas  writes:
 ... Well, the current CommitFest ends in one week, ...
>>>
>>> Really?  I thought the idea for the last CF of a development cycle was
>>> that it kept going till we'd dealt with everything.  Arbitrarily
>>> rejecting stuff we haven't dealt with doesn't seem fair.
>>
>> Uh, we did that with 8.4 and it was a disaster.  The CommitFest lasted
>> *five months*. We've been doing schedule-based CommitFests ever since
>> and it's worked much better.
>
> Rejecting stuff because we haven't gotten round to dealing with it in
> such a short period of time is a damn good way to limit the number of
> contributions we get. I don't believe we've agreed at any point that
> the last commitfest should be the same time length as the others

News to me.

http://wiki.postgresql.org/wiki/PostgreSQL_9.1_Development_Plan

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[David E. Wheeler]

On Feb 7, 2011, at 10:23 AM, Tom Lane wrote:

> On reflection, the set of extensions that an extension depends on is
> obviously a property of the extension, which means it ought to be
> specified in the extension's control file, not in the CREATE EXTENSION
> command.  So now I'm thinking something like
> 
>   requires = 'foo, bar, baz'
> 
> in the file. 

And that takes us one step closer to PGXN's META.json file. Here's the spec:

  http://pgxn.org/meta/spec.txt

It includes a "prereqs" section, which looks like this:

   "prereqs": {
  "runtime": {
 "requires": {
"citext": 0,
"plpgsql": 0,
"PostgreSQL": "8.0.0"
 },
 "recommends": {
"PostgreSQL": "8.4.0"
 }
  }
   },


Best,

David

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Josh Berkus]

>> I just spoke to my manager at EnterpriseDB and he cleared my schedule
>> for the next two days to work on this.  So I'll go hack on this now.
>> I haven't read the patch yet so I don't know for sure how quite I'll
>> be able to get up to speed on it, so if someone who is more familiar
>> with this code wants to grab the baton away from me, feel free.
>> Otherwise, I'll see what I can do with it.
> 
> Presumably you have a reason for declaring war? I'm sorry for that, I
> really am.

How is clearing out his whole schedule to help review & fix the patch
declaring war?  You have an odd attitude towards assistance, Simon.

-- 
  -- Josh Berkus
 PostgreSQL Experts Inc.
 http://www.pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Robert Haas]

On Mon, Feb 7, 2011 at 2:09 PM, Dimitri Fontaine  wrote:
> Robert Haas  writes:
>> done in the time available is another thing entirely.  I do NOT want
>> to still be working on the items for this CommitFest in June - that's
>> about when I'd like to be releasing beta3.
>
> Except that's not how we work here.  You want to change that with
> respect to the release management process and schedule (or lack
> thereof).  Tradition and current practice say you need to reach
> consensus to be able to bypass compromising.
>
> Good luck with that.

I'm not trying to bypass compromising, and I don't know what makes you
think otherwise.  I am trying to ensure that the CommitFest wraps up
in a timely fashion, which is something we have done consistently for
every CommitFest in the 9.0 and 9.1 cycles to date, including the last
CommitFest of the 9.0 cycle.  It is not somehow a deviation from past
community practice to boot patches that can't be finished up in the
time available during the CommitFest.  That has been routine practice
for a long time.

I have worked very hard on this CommitFest, both to line up patch
reviewers and to review myself.  I want to make sure that every patch
gets a good, thorough review before the CommitFest is over.  I think
there is general consensus that this is important and that we will
lose contributors if we don't do it.  However, I don't want to prolong
the CommitFest indefinitely in the face of patches that the authors
are not actively working on or can't finish in the time available, or
where there is no consensus that the proposed change is what we want.
I believe that this, too, is a generally accepted principle in our
community, not something I just made up.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[Tom Lane]

Dimitri Fontaine  writes:
> That said, we should do something about ALTER EXTENSION SET SCHEMA and
> the relocatable property.  I'm thinking that an extension stops being
> relocatable as soon as any of its reverse dependencies (all the tree) is
> not relocatable.

If you're worried about that, then it's questionable whether ALTER
EXTENSION SET SCHEMA makes sense at all, ever.  I don't see any reason
to think that an extension is more fragile for this purpose than any
other random SQL dependencies.  Also, an extension being relocatable
doesn't seem to me to guarantee that it can cope with its dependencies
moving around; they're really independent properties.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Joshua D. Drake]

On Mon, 2011-02-07 at 17:59 +, Simon Riggs wrote:
> On Mon, 2011-02-07 at 12:39 -0500, Robert Haas wrote:
> 
> > I just spoke to my manager at EnterpriseDB and he cleared my schedule
> > for the next two days to work on this.  So I'll go hack on this now.
> > I haven't read the patch yet so I don't know for sure how quite I'll
> > be able to get up to speed on it, so if someone who is more familiar
> > with this code wants to grab the baton away from me, feel free.
> > Otherwise, I'll see what I can do with it.
> 
> Presumably you have a reason for declaring war? I'm sorry for that, I
> really am.

Simon,

My impression was that Robert had received a release from current
responsibilities to help you with your patch, not that he was declaring
war or some such thing. I believe we all want SyncRep to be successful. 

Sincerely,

JD


-- 
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering
http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Dimitri Fontaine]

Robert Haas  writes:
> done in the time available is another thing entirely.  I do NOT want
> to still be working on the items for this CommitFest in June - that's
> about when I'd like to be releasing beta3.

Except that's not how we work here.  You want to change that with
respect to the release management process and schedule (or lack
thereof).  Tradition and current practice say you need to reach
consensus to be able to bypass compromising.

Good luck with that.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] limiting hint bit I/O

[Cédric Villemain]

2011/2/7 Cédric Villemain :
> 2011/2/7 Robert Haas :
>> On Mon, Feb 7, 2011 at 10:48 AM, Bruce Momjian  wrote:
>>> Robert Haas wrote:
 On Sat, Feb 5, 2011 at 4:31 PM, Bruce Momjian  wrote:
 > Uh, in this C comment:
 >
 > + ? ? ? ?* or not we want to take the time to write it. ?We allow up to 
 > 5% of
 > + ? ? ? ?* otherwise-not-dirty pages to be written due to hint bit 
 > changes,
 >
 > 5% of what? ?5% of all buffers? ?5% of all hint-bit-dirty ones? ?Can you
 > clarify this in the patch?

 5% of buffers that are hint-bit-dirty but not otherwise dirty.  ISTM
 that's exactly what the comment you just quoted says on its face, but
 I'm open to some other wording you want to propose.
>>>
>>> How about:
>>>
>>>        otherwise-not-dirty -> only-hint-bit-dirty
>>>
>>> So 95% of your hint bit modificates are discarded if the pages is not
>>> otherwise dirtied?  That seems pretty radical.
>>
>> No, it's more subtle than that, although I admit it *is* radical.
>> There are three ways that pages can get written out to disk:
>>
>> 1. Checkpoints.
>> 2. Background writer activity.
>> 3. Backends writing out dirty buffers because there are no clean
>> buffers available to allocate.
>>
>> What the latest version of the patch implements is:
>>
>> 1. Checkpoints no longer write only-hint-bit-dirty pages to disk.
>> Since a checkpoint doesn't evict pages from memory, the hint bits are
>> still there to be written out (or not) by (2) or (3), below.
>>
>> 2. When the background writer's cleaning scan hits an
>> only-hint-bit-dirty page, it writes it, same as before.  This
>> definitely doesn't result in the loss of any hint bits.
>>
>> 3. When a backend writes out a dirty buffer itself, because there are
>> no clean buffers available to allocate, it initially writes them.  But
>> if there are more than 100 such pages per block of 2000 allocations,
>> it recycles any after the first 100 without writing them.
>>
>> In normal operation, I suspect that there will be very little impact
>> from this change.  The change described in #1 may slightly reduce the
>> size of some checkpoints, but it's unclear that it will be enough to
>> be material.  The change described in #3 will probably also not
>> matter, because, in a well-tuned system, the background writer should
>> be set aggressively enough to provide a supply of clean pages, and
>> therefore backends shouldn't be doing many writes themselves, and
>> therefore most buffer allocations will be of already-clean pages, and
>> the logic described in #3 will probably never kick in.  Even if they
>> are writing a lot of buffers themselves, the logic in #3 still won't
>> kick in if many of the pages being written are actually dirty - it
>> will only matter if the backends are writing out lots and lots of
>> pages *solely because they are only-hint-bit-dirty*.
>>
>> Where I expect this to make a big difference is on sequential scans of
>> just-loaded tables.  In that case, the BufferAccessStrategy machinery
>> will force the backend to reuse the same buffers over and over again,
>> and all of those pages will be only-hint-bit-dirty.  So the backend
>> has to do a write for every page it allocates, and even though those
>> writes are being absorbed by the OS cache, it's still slow.  With this
>> patch, what will happen is that the backend will write about 100
>> pages, then perform the next 1900 allocations without writing, then
>> write another 100 pages, etc.  So at the end of the scan, instead of
>> having written an amount of data equal to the size of the table, we
>> will have written 5% of that amount, and 5% of the hint bits will be
>> on disk.  Each subsequent scan will get another 5% of the hint bits on
>> disk until after 20 scans they are all set.  So the work of setting
>> the hint bits is spread out across the first 20 table scans instead of
>> all being done the first time through.
>>
>> Clearly, there's further jiggering that can be done here.  But the
>> overall goal is simply that some of our users don't seem to like it
>> when the first scan of a newly loaded table generates a huge storm of
>> *write* traffic.  Given that the hint bits appear to be quite
>> important from a performance perspective (see benchmark numbers
>> upthread),
>
> those are not real benchmarks, just quick guess to check behavior.
> (and I agree it looks good, but I also got inconsistent results, the
> patched postgresql hardly reach the same speed of the original
> 9.1devel even after 200 hundreds select of your testcase)
>
>
>> we don't really have the option of just not writing them -
>> but we can try to not to do it all at once, if we think that's an
>> improvement, which I think is likely.
>>
>> Overall, I'm inclined to move this patch to the next CommitFest and
>> forget about it for now.  I don't think we're going to get enough
>> testing of this in the next week to be really confident that it's
>> right.  I might be willing to co

Re: [HACKERS] limiting hint bit I/O

[Cédric Villemain]

2011/2/7 Robert Haas :
> On Mon, Feb 7, 2011 at 10:48 AM, Bruce Momjian  wrote:
>> Robert Haas wrote:
>>> On Sat, Feb 5, 2011 at 4:31 PM, Bruce Momjian  wrote:
>>> > Uh, in this C comment:
>>> >
>>> > + ? ? ? ?* or not we want to take the time to write it. ?We allow up to 
>>> > 5% of
>>> > + ? ? ? ?* otherwise-not-dirty pages to be written due to hint bit 
>>> > changes,
>>> >
>>> > 5% of what? ?5% of all buffers? ?5% of all hint-bit-dirty ones? ?Can you
>>> > clarify this in the patch?
>>>
>>> 5% of buffers that are hint-bit-dirty but not otherwise dirty.  ISTM
>>> that's exactly what the comment you just quoted says on its face, but
>>> I'm open to some other wording you want to propose.
>>
>> How about:
>>
>>        otherwise-not-dirty -> only-hint-bit-dirty
>>
>> So 95% of your hint bit modificates are discarded if the pages is not
>> otherwise dirtied?  That seems pretty radical.
>
> No, it's more subtle than that, although I admit it *is* radical.
> There are three ways that pages can get written out to disk:
>
> 1. Checkpoints.
> 2. Background writer activity.
> 3. Backends writing out dirty buffers because there are no clean
> buffers available to allocate.
>
> What the latest version of the patch implements is:
>
> 1. Checkpoints no longer write only-hint-bit-dirty pages to disk.
> Since a checkpoint doesn't evict pages from memory, the hint bits are
> still there to be written out (or not) by (2) or (3), below.
>
> 2. When the background writer's cleaning scan hits an
> only-hint-bit-dirty page, it writes it, same as before.  This
> definitely doesn't result in the loss of any hint bits.
>
> 3. When a backend writes out a dirty buffer itself, because there are
> no clean buffers available to allocate, it initially writes them.  But
> if there are more than 100 such pages per block of 2000 allocations,
> it recycles any after the first 100 without writing them.
>
> In normal operation, I suspect that there will be very little impact
> from this change.  The change described in #1 may slightly reduce the
> size of some checkpoints, but it's unclear that it will be enough to
> be material.  The change described in #3 will probably also not
> matter, because, in a well-tuned system, the background writer should
> be set aggressively enough to provide a supply of clean pages, and
> therefore backends shouldn't be doing many writes themselves, and
> therefore most buffer allocations will be of already-clean pages, and
> the logic described in #3 will probably never kick in.  Even if they
> are writing a lot of buffers themselves, the logic in #3 still won't
> kick in if many of the pages being written are actually dirty - it
> will only matter if the backends are writing out lots and lots of
> pages *solely because they are only-hint-bit-dirty*.
>
> Where I expect this to make a big difference is on sequential scans of
> just-loaded tables.  In that case, the BufferAccessStrategy machinery
> will force the backend to reuse the same buffers over and over again,
> and all of those pages will be only-hint-bit-dirty.  So the backend
> has to do a write for every page it allocates, and even though those
> writes are being absorbed by the OS cache, it's still slow.  With this
> patch, what will happen is that the backend will write about 100
> pages, then perform the next 1900 allocations without writing, then
> write another 100 pages, etc.  So at the end of the scan, instead of
> having written an amount of data equal to the size of the table, we
> will have written 5% of that amount, and 5% of the hint bits will be
> on disk.  Each subsequent scan will get another 5% of the hint bits on
> disk until after 20 scans they are all set.  So the work of setting
> the hint bits is spread out across the first 20 table scans instead of
> all being done the first time through.
>
> Clearly, there's further jiggering that can be done here.  But the
> overall goal is simply that some of our users don't seem to like it
> when the first scan of a newly loaded table generates a huge storm of
> *write* traffic.  Given that the hint bits appear to be quite
> important from a performance perspective (see benchmark numbers
> upthread),

those are not real benchmarks, just quick guess to check behavior.
(and I agree it looks good, but I also got inconsistent results, the
patched postgresql hardly reach the same speed of the original
9.1devel even after 200 hundreds select of your testcase)


> we don't really have the option of just not writing them -
> but we can try to not to do it all at once, if we think that's an
> improvement, which I think is likely.
>
> Overall, I'm inclined to move this patch to the next CommitFest and
> forget about it for now.  I don't think we're going to get enough
> testing of this in the next week to be really confident that it's
> right.  I might be willing to commit with some more moderate amount of
> testing if we were right at the beginning of a development cycle,
> figuring that we'd shak

Re: [HACKERS] More extension issues: ownership and search_path

[Dimitri Fontaine]

Tom Lane  writes:
> On reflection, the set of extensions that an extension depends on is
> obviously a property of the extension, which means it ought to be
> specified in the extension's control file, not in the CREATE EXTENSION
> command.  So now I'm thinking something like
>
>   requires = 'foo, bar, baz'

+1

And that can change at upgrade time, of course, but that's another
story.  Ditto for recommends and conflict dependency types, that's
material for 9.2 at best.

> in the file.  We could even imagine autoloading such dependencies if
> they're not already installed, but that's a frammish for later.  (One
> objection to autoloading is it's not clear which schema to drop the
> other extensions into.)

Well I don't see why it wouldn't be the same schema in case of auto
solving dependencies, but I don't see a pressing need to have automatic
dependency following at install time (we're still in the realm of dpkg,
we'll get into apt-get next) :)

That said, we should do something about ALTER EXTENSION SET SCHEMA and
the relocatable property.  I'm thinking that an extension stops being
relocatable as soon as any of its reverse dependencies (all the tree) is
not relocatable.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Spread checkpoint sync

[Kevin Grittner]

Greg Smith  wrote:

> As a larger statement on this topic, I'm never very excited about
> redesigning here starting from any point other than "saw a
> bottleneck doing  on a production system".  There's a long list
> of such things already around waiting to be addressed, and I've
> never seen any good evidence of work related to hint bits being on
> it.  Please correct me if you know of some--I suspect you do from
> the way you're brining this up.

There are occasional posts from those wondering why their read-only
queries are so slow after a bulk load, and why they are doing heavy
writes.  (I remember when I posted about that, as a relative newbie,
and I know I've seen others.)

I think worst case is probably:

- Bulk load data.
- Analyze (but don't vacuum) the new data.
- Start a workload with a lot of small, concurrent random reads.
- Watch performance tank when the write cache gluts.

This pattern is why we've adopted a pretty strict rule in our shop
that we run VACUUM FREEZE ANALYZE between a bulk load and putting
the database back into production.  It's probably a bigger issue for
those who can't do that.

-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: RangeTypes

[Peter Eisentraut]

On sön, 2011-01-30 at 14:52 -0800, Jeff Davis wrote:
>   * naming issues:
> - period -> tsrange ?
> - periodtz -> tstzrange ?
> - intrange -> int4range

Have you considered a grammar approach like for arrays, so that you
would write something like

CREATE TABLE ... (
foo RANGE OF int
);

instead of explicitly creating a range type for every scalar type in
existence?  I think that that might be easier to use in the common case.

I guess the trick might be how to store and pass the operator class and
some other parameters.




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[Tom Lane]

I wrote:
> ... So where I think we're going to end up
> is adding a clause along the line of "USING list-of-extension-names"
> to CREATE EXTENSION, storing those dependencies explicitly, and having
> the CREATE EXTENSION code set search_path to the target schema followed
> by the target schema(s) of the USING extensions.

On reflection, the set of extensions that an extension depends on is
obviously a property of the extension, which means it ought to be
specified in the extension's control file, not in the CREATE EXTENSION
command.  So now I'm thinking something like

requires = 'foo, bar, baz'

in the file.  We could even imagine autoloading such dependencies if
they're not already installed, but that's a frammish for later.  (One
objection to autoloading is it's not clear which schema to drop the
other extensions into.)

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[Tom Lane]

Dimitri Fontaine  writes:
> Tom Lane  writes:
>> I think we'd better add an extowner column to pg_extension.

> Agreed.  There's no need to have it now but we will add it at some
> point.  So if now is when that works the best for you, I'm happy to see
> that happen :)

BTW, on trying this I notice that pg_dump's default approach to
ownership doesn't work because of the lack of an ALTER EXTENSION
OWNER TO command.  I'm going to go ahead and add extowner to the catalog
anyway, because it's easy and I'm convinced we're going to want it
later.  But I don't feel like writing ALTER EXTENSION OWNER TO right
now, so pg_dump will continue its current behavior of creating the
extension as the user running the script.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Robert Haas]

On Mon, Feb 7, 2011 at 12:59 PM, Simon Riggs  wrote:
> On Mon, 2011-02-07 at 12:39 -0500, Robert Haas wrote:
>
>> I just spoke to my manager at EnterpriseDB and he cleared my schedule
>> for the next two days to work on this.  So I'll go hack on this now.
>> I haven't read the patch yet so I don't know for sure how quite I'll
>> be able to get up to speed on it, so if someone who is more familiar
>> with this code wants to grab the baton away from me, feel free.
>> Otherwise, I'll see what I can do with it.
>
> Presumably you have a reason for declaring war? I'm sorry for that, I
> really am.

What the hell are you talking about?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Robert Haas]

On Mon, Feb 7, 2011 at 12:43 PM, Tom Lane  wrote:
> Robert Haas  writes:
>> ... Well, the current CommitFest ends in one week, ...
>
> Really?  I thought the idea for the last CF of a development cycle was
> that it kept going till we'd dealt with everything.  Arbitrarily
> rejecting stuff we haven't dealt with doesn't seem fair.

Uh, we did that with 8.4 and it was a disaster.  The CommitFest lasted
*five months*. We've been doing schedule-based CommitFests ever since
and it's worked much better.  I agree it's unfair to reject things
without looking at them, and I'd like to avoid that if at all
possible, but punting things because they need more work than can be
done in the time available is another thing entirely.  I do NOT want
to still be working on the items for this CommitFest in June - that's
about when I'd like to be releasing beta3.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Simon Riggs]

On Mon, 2011-02-07 at 12:39 -0500, Robert Haas wrote:

> I just spoke to my manager at EnterpriseDB and he cleared my schedule
> for the next two days to work on this.  So I'll go hack on this now.
> I haven't read the patch yet so I don't know for sure how quite I'll
> be able to get up to speed on it, so if someone who is more familiar
> with this code wants to grab the baton away from me, feel free.
> Otherwise, I'll see what I can do with it.

Presumably you have a reason for declaring war? I'm sorry for that, I
really am.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/books/
 PostgreSQL Development, 24x7 Support, Training and Services
 


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[Tom Lane]

"David E. Wheeler"  writes:
> On Feb 7, 2011, at 9:51 AM, Tom Lane wrote:
>> Interesting point.  It's all right at the moment because I tweaked
>> pg_dump_sort.c so that procedural languages are dumped before modules.
>> But maybe we should convert the PLs to modules.

> s/modules/extensions/?

Yeah, I keep saying "module".  Memo to self: grep the whole patch for
"module" before committing.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A different approach to extension NO USER DATA feature

[Tom Lane]

"David E. Wheeler"  writes:
> On Feb 7, 2011, at 8:57 AM, Tom Lane wrote:
>> Yeah, this is another approach that one could take instead of having
>> per-row flags.  I'm not sure that it's better, much less so much better
>> that we should force extensions to do it that way and not the other.
>> But it's definitely another argument against making a hard-wired
>> assumption that there will be a flag column.

> Would the flag column usually be invisible, like the system oid column?

No, it's just a column.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[David E. Wheeler]

On Feb 7, 2011, at 9:51 AM, Tom Lane wrote:

> Interesting point.  It's all right at the moment because I tweaked
> pg_dump_sort.c so that procedural languages are dumped before modules.
> But maybe we should convert the PLs to modules.

s/modules/extensions/?

David


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[David E. Wheeler]

On Feb 7, 2011, at 9:20 AM, Dimitri Fontaine wrote:

> Also, I didn't bite this bullet, but maybe we should provide core PLs as
> extension.  Then CREATE LANGUAGE would maybe get deprecated and only
> valid when used in an extension's script — or the next patch (UPGRADE)
> will take care of create a plpythonu extension then attaching the PL
> into it.

I anticipate dependencies becoming a big deal. I already have ideas for 
extensions that depend on citext, for example (domains for time zone, email 
address, etc.). And yeah, some of those might depend on procedural languages. 
FWIW, I've been putting PL prereqs in META.json files on PGXN. pgTAP, for 
example, requires PL/pgSQL:

  http://master.pgxn.org/dist/pgTAP/pgTAP-0.25.0.json

Best,

David


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[Tom Lane]

Dimitri Fontaine  writes:
> Tom Lane  writes:
>> Quite aside from search path, cross-extension dependencies simply aren't
>> going to work unless pg_dump knows about them so it can load the
>> extensions in the right order.  I had forgotten about the earthdistance
>> case, but given that I think we probably can't put this issue off.

> Well in fact the ordering already works because some earthdistance
> objects depend on some cube objects, and pg_dump sees that in pg_depend.

Really?  I think it's more likely just luckily working because of the
alphabetical-ordering default.  To make it work reliably off of those
dependencies, we'd need some code to pull up the dependency links from
the individual objects to the module level, and we haven't got that.
I'd prefer to make the module dependencies explicit anyway.

> Imagine that you wrote a set of plpgsql and plpythonu functions that you
> want to maintain as an extension.  You certainly want to mark that this
> extension depends on the procedural languages being installed, right?

Interesting point.  It's all right at the moment because I tweaked
pg_dump_sort.c so that procedural languages are dumped before modules.
But maybe we should convert the PLs to modules.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A different approach to extension NO USER DATA feature

[David E. Wheeler]

On Feb 7, 2011, at 8:57 AM, Tom Lane wrote:

> Yeah, this is another approach that one could take instead of having
> per-row flags.  I'm not sure that it's better, much less so much better
> that we should force extensions to do it that way and not the other.
> But it's definitely another argument against making a hard-wired
> assumption that there will be a flag column.

Would the flag column usually be invisible, like the system oid column?

Best,

David


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Tom Lane]

Robert Haas  writes:
> ... Well, the current CommitFest ends in one week, ...

Really?  I thought the idea for the last CF of a development cycle was
that it kept going till we'd dealt with everything.  Arbitrarily
rejecting stuff we haven't dealt with doesn't seem fair.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SSI patch version 14

[Kevin Grittner]

Kevin Grittner  wrote:
> "Kevin Grittner"  wrote:
>> Jeff Davis wrote:
>>  
>>> What does PredicateLockTuple do that needs a share lock? Does a
>>> pin suffice?
>> 
>> If one process is reading a tuple and another is writing it
>> (e.g., UPDATE or DELETE) the concern is that we need to be able
>> to guarantee that either the predicate lock appears in time for
>> the writer to see it on the call to
>> CheckForSerializableConflictIn or the reader sees the MVCC
>> changes in CheckForSerializableConflictOut. It's OK if the
>> conflict is detected both ways, but if it's missed both ways then
>> we could have a false negative, allowing anomalies to slip
>> through. It didn't seem to me on review that acquiring the
>> predicate lock after releasing the shared buffer lock (and just
>> having the pin) would be enough to ensure that a write which
>> followed that would see the predicate lock.
>> 
>> reader has pin and shared lock
>> writer increments pin count
>> reader releases shared lock
>> writer acquires exclusive lock
>> writer checks predicate lock and fails to see one
>> reader adds predicate lock
>> we have a problem
>  
> Hmmm...  Or do we?  If both sides were careful to record what
> they're doing before checking for a conflict, the pin might be
> enough. I'll check for that.  In at least one of those moves I was
> moving the predicate lock acquisition from after the conflict
> check to before, but maybe I didn't need to move it quite so
> far
 
Some of these calls are placed where there is no reasonable choice
but to do them while holding a buffer lock.  There are some which
*might* be able to be moved out, but I'm inclined to say that should
be on a list of possible performance improvements in future
releases.  My concern is that the code paths are complicated enough
to make it hard to confidently desk-check such changes, we don't yet
have a good way to test whether such a change is introducing a race
condition.  The src/test/isolation tests are good at proving the
fundamental correctness of the logic in predicate.c, and the DBT-2
stress tests Dan ran are good at flushing out race conditions within
predicate.c code; but we don't have a test which is good at pointing
out race conditions based on *placement of the calls* to predicate.c
from other code.
 
Until we have such tests, I think we should be cautious about
risking possible hard-to-reproduce correctness violations to shave a
few nanoseconds off the time a shared buffer lock is held. 
Particularly since we're so close to the end of the release cycle.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Robert Haas]

On Mon, Feb 7, 2011 at 12:28 PM, Robert Haas  wrote:
> On Mon, Feb 7, 2011 at 11:33 AM, Simon Riggs  wrote:
>> On Mon, 2011-02-07 at 09:55 -0500, Robert Haas wrote:
>>> On Sun, Jan 30, 2011 at 11:44 AM, Robert Haas  wrote:
>>> > Time's running short - do you have an updated patch?
>>>
>>> This patch hasn't been updated in more than three weeks.  I assume
>>> this should now be marked Returned with Feedback, and we'll revisit
>>> synchronous replication for 9.2?
>>
>> I have time to complete that in next two weeks, but you are right I
>> haven't had it in last few weeks.
>
> Well, the current CommitFest ends in one week, and we need to leave
> time for someone (Heikki, most likely) to review, so there's really
> only a couple of days left.
>
> Bruce's suggestion of having someone else pick it up seems like it
> might work.  The obvious candidates are probably Heikki Linnakangas,
> Tom Lane, Fujii Masao, and (if you squint a little) me.  I am not
> clear that any of those people have the necessary time available
> immediately, however.

I just spoke to my manager at EnterpriseDB and he cleared my schedule
for the next two days to work on this.  So I'll go hack on this now.
I haven't read the patch yet so I don't know for sure how quite I'll
be able to get up to speed on it, so if someone who is more familiar
with this code wants to grab the baton away from me, feel free.
Otherwise, I'll see what I can do with it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Robert Haas]

On Mon, Feb 7, 2011 at 11:33 AM, Simon Riggs  wrote:
> On Mon, 2011-02-07 at 09:55 -0500, Robert Haas wrote:
>> On Sun, Jan 30, 2011 at 11:44 AM, Robert Haas  wrote:
>> > Time's running short - do you have an updated patch?
>>
>> This patch hasn't been updated in more than three weeks.  I assume
>> this should now be marked Returned with Feedback, and we'll revisit
>> synchronous replication for 9.2?
>
> I have time to complete that in next two weeks, but you are right I
> haven't had it in last few weeks.

Well, the current CommitFest ends in one week, and we need to leave
time for someone (Heikki, most likely) to review, so there's really
only a couple of days left.

Bruce's suggestion of having someone else pick it up seems like it
might work.  The obvious candidates are probably Heikki Linnakangas,
Tom Lane, Fujii Masao, and (if you squint a little) me.  I am not
clear that any of those people have the necessary time available
immediately, however.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Range Types

[Dimitri Fontaine]

Jeff Davis  writes:
> Ok, but what should the parameter to CREATE TYPE ... AS RANGE be then?
>
> CREATE TYPE foo AS RANGE (
>   SUBTYPE = ...
>   SUBTYPE_BTREE_OPERATOR_CLASS = ...
> );
>
> is a little verbose. Ideas?

I would think

  CREATE TYPE foo AS RANGE (bar) USING (btree_ops);

The USING clause is optional, because you generally have a default btree
opclass for the datatype.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[Dimitri Fontaine]

Tom Lane  writes:
> No, I've hacked the code enough already that merging would be painful.
> I'll keep working on it.

I supposed so much, but got to ask :)

> Oh, duh, I'd forgotten about the OverrideSearchPath usage.  So never
> mind the above claim.  But I still think it'd be a good idea to ensure
> that the search path is the same during dump/reload as it was the first
> time, and the current arrangement isn't going to ensure that.

Right.  Something automated would be best (no user intervention), it
would force extension scripts to be self-contained or to explicitly
declare all their external dependencies.  I'm in fact doping out
entirely my previous SET idea.

>> So while we said this is 9.2 material, if you want to tackle the whole
>> search_path at restore time issue (I did only the creation namespace,
>> thinking it would be enough) fully, you need dependency too.
>
> Quite aside from search path, cross-extension dependencies simply aren't
> going to work unless pg_dump knows about them so it can load the
> extensions in the right order.  I had forgotten about the earthdistance
> case, but given that I think we probably can't put this issue off.

Well in fact the ordering already works because some earthdistance
objects depend on some cube objects, and pg_dump sees that in pg_depend.

>> I think we should then register some core components as extensions for
>> the sake of interdependencies here, too.
>
> Maybe that sort of refactoring could be done in 9.2 or further out.
> I don't see it happening for 9.1.  I'm not really sure what the value
> is anyway.

Imagine that you wrote a set of plpgsql and plpythonu functions that you
want to maintain as an extension.  You certainly want to mark that this
extension depends on the procedural languages being installed, right?

Also, I didn't bite this bullet, but maybe we should provide core PLs as
extension.  Then CREATE LANGUAGE would maybe get deprecated and only
valid when used in an extension's script — or the next patch (UPGRADE)
will take care of create a plpythonu extension then attaching the PL
into it.

Again, pg_depend already allows pg_dump to create plpythonu before it
creates the extension that depends on it, though.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] OpenVMS - an effort which needs guidance and support.

[Chris Browne]

peder...@ccsscorp.com ("Bill Pedersen") writes:
> I look forward to hearing from people in the PostgreSQL community as well as
> from others interested in this effort.

To a number of us, it's academically interesting, though, as we don't
have VMS systems, it's not likely to be super-easy to assist in the
matter.

It certainly would  be interesting to see how easy or difficult the port
would be.  I suspect that's a more interesting port than, say, Digital
UNIX, these days.
-- 
(reverse (concatenate 'string "ofni.sesabatadxunil" "@" "enworbbc"))
http://linuxfinances.info/info/slony.html
On the other hand, you have different fingers. 

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] limiting hint bit I/O

[Robert Haas]

On Mon, Feb 7, 2011 at 10:48 AM, Bruce Momjian  wrote:
> Robert Haas wrote:
>> On Sat, Feb 5, 2011 at 4:31 PM, Bruce Momjian  wrote:
>> > Uh, in this C comment:
>> >
>> > + ? ? ? ?* or not we want to take the time to write it. ?We allow up to 5% 
>> > of
>> > + ? ? ? ?* otherwise-not-dirty pages to be written due to hint bit changes,
>> >
>> > 5% of what? ?5% of all buffers? ?5% of all hint-bit-dirty ones? ?Can you
>> > clarify this in the patch?
>>
>> 5% of buffers that are hint-bit-dirty but not otherwise dirty.  ISTM
>> that's exactly what the comment you just quoted says on its face, but
>> I'm open to some other wording you want to propose.
>
> How about:
>
>        otherwise-not-dirty -> only-hint-bit-dirty
>
> So 95% of your hint bit modificates are discarded if the pages is not
> otherwise dirtied?  That seems pretty radical.

No, it's more subtle than that, although I admit it *is* radical.
There are three ways that pages can get written out to disk:

1. Checkpoints.
2. Background writer activity.
3. Backends writing out dirty buffers because there are no clean
buffers available to allocate.

What the latest version of the patch implements is:

1. Checkpoints no longer write only-hint-bit-dirty pages to disk.
Since a checkpoint doesn't evict pages from memory, the hint bits are
still there to be written out (or not) by (2) or (3), below.

2. When the background writer's cleaning scan hits an
only-hint-bit-dirty page, it writes it, same as before.  This
definitely doesn't result in the loss of any hint bits.

3. When a backend writes out a dirty buffer itself, because there are
no clean buffers available to allocate, it initially writes them.  But
if there are more than 100 such pages per block of 2000 allocations,
it recycles any after the first 100 without writing them.

In normal operation, I suspect that there will be very little impact
from this change.  The change described in #1 may slightly reduce the
size of some checkpoints, but it's unclear that it will be enough to
be material.  The change described in #3 will probably also not
matter, because, in a well-tuned system, the background writer should
be set aggressively enough to provide a supply of clean pages, and
therefore backends shouldn't be doing many writes themselves, and
therefore most buffer allocations will be of already-clean pages, and
the logic described in #3 will probably never kick in.  Even if they
are writing a lot of buffers themselves, the logic in #3 still won't
kick in if many of the pages being written are actually dirty - it
will only matter if the backends are writing out lots and lots of
pages *solely because they are only-hint-bit-dirty*.

Where I expect this to make a big difference is on sequential scans of
just-loaded tables.  In that case, the BufferAccessStrategy machinery
will force the backend to reuse the same buffers over and over again,
and all of those pages will be only-hint-bit-dirty.  So the backend
has to do a write for every page it allocates, and even though those
writes are being absorbed by the OS cache, it's still slow.  With this
patch, what will happen is that the backend will write about 100
pages, then perform the next 1900 allocations without writing, then
write another 100 pages, etc.  So at the end of the scan, instead of
having written an amount of data equal to the size of the table, we
will have written 5% of that amount, and 5% of the hint bits will be
on disk.  Each subsequent scan will get another 5% of the hint bits on
disk until after 20 scans they are all set.  So the work of setting
the hint bits is spread out across the first 20 table scans instead of
all being done the first time through.

Clearly, there's further jiggering that can be done here.  But the
overall goal is simply that some of our users don't seem to like it
when the first scan of a newly loaded table generates a huge storm of
*write* traffic.  Given that the hint bits appear to be quite
important from a performance perspective (see benchmark numbers
upthread), we don't really have the option of just not writing them -
but we can try to not to do it all at once, if we think that's an
improvement, which I think is likely.

Overall, I'm inclined to move this patch to the next CommitFest and
forget about it for now.  I don't think we're going to get enough
testing of this in the next week to be really confident that it's
right.  I might be willing to commit with some more moderate amount of
testing if we were right at the beginning of a development cycle,
figuring that we'd shake out any warts as the cycle went along, but
this isn't seeming like the right time for this kind of a change.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Range Types

[Jeff Davis]

On Mon, 2011-02-07 at 13:33 +0100, Dimitri Fontaine wrote:
> Hi,
> 
> Jeff Davis  writes:
> >   * Should pg_range reference the btree opclass or the compare function
> > directly?
> 
> I would say yes.  We use the btree opclass in other similar situations.

Ok, but what should the parameter to CREATE TYPE ... AS RANGE be then?

CREATE TYPE foo AS RANGE (
  SUBTYPE = ...
  SUBTYPE_BTREE_OPERATOR_CLASS = ...
);

is a little verbose. Ideas?

> Is there any reason to restrict who's get to use the feature?  I don't
> see any…

Mostly just paranoia. If they define a strange canonical function, maybe
that would cause a problem. Then again, they would have to define that
in C to cause a problem anyway. I'll leave it as superuser-only for now,
and see if anyone else raises potential problems.

> >   * Should the SQL (inlinable) function "length", which relies on
> > polymorphic "-", be immutable, strict, or volatile?
> 
> I would think stable: polymorphic means that the function
> implementing the "-" operator depends on the argument.  I don't recall
> that it depends on them in a volatile way… except if you change the
> operator definition, which is possible to do (so not immutable).

I was concerned about someone defining "-" with a stable or volatile
function as the definition. I'm not sure if that is a reasonable concern
or not.

> >   * Later we might consider whether we should include btree_gist in
> > core, to make range types more useful with exclusion constraints
> > out-of-the-box. This should be left for later, I'm just including this
> > for the archives so it doesn't get lost.
> 
> I would expect the extension to have something to offer here.

Yes. With extensions and PGXN, I would hope that installing btree_gist
would not be much of a problem. However, I eventually intend to submit
features like "RANGE KEY", a language extension that would need
something like btree_gist to work very well at all. Technically
btree_gist is not required, but in practice it is necessary to use
ranges and exclusion constraints together effectively.

Regards,
Jeff Davis


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A different approach to extension NO USER DATA feature

[Tom Lane]

Florian Pflug  writes:
> We could avoid the need for a per-row "system_data" flag if we required
> extensions to split user-editable and system-provided configuration data
> into different tables. For convenient access to the configuration data,
> the extension could let the user-editable table inherit from the 
> system-provided one, or use a view to combine the two.

Yeah, this is another approach that one could take instead of having
per-row flags.  I'm not sure that it's better, much less so much better
that we should force extensions to do it that way and not the other.
But it's definitely another argument against making a hard-wired
assumption that there will be a flag column.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[Tom Lane]

Dimitri Fontaine  writes:
> Tom Lane  writes:
>> I think we'd better add an extowner column to pg_extension.

> Agreed.  There's no need to have it now but we will add it at some
> point.  So if now is when that works the best for you, I'm happy to see
> that happen :)

> Would it help that I prepare some of those modifications, as patches
> over the extension's patch that you started from?

No, I've hacked the code enough already that merging would be painful.
I'll keep working on it.

>> Another is the search_path setting.  Currently this is actually rather
>> broken since pg_dump makes no effort at all to ensure that search_path
>> is the same at reload time as it was when the extension was created,
>> and thus the contained objects could easily go into the wrong schema.

> Well there's some code to place the extension's schema at the first
> place in the search_path before executing the script, already.

Oh, duh, I'd forgotten about the OverrideSearchPath usage.  So never
mind the above claim.  But I still think it'd be a good idea to ensure
that the search path is the same during dump/reload as it was the first
time, and the current arrangement isn't going to ensure that.

> So while we said this is 9.2 material, if you want to tackle the whole
> search_path at restore time issue (I did only the creation namespace,
> thinking it would be enough) fully, you need dependency too.

Quite aside from search path, cross-extension dependencies simply aren't
going to work unless pg_dump knows about them so it can load the
extensions in the right order.  I had forgotten about the earthdistance
case, but given that I think we probably can't put this issue off.

> I think we should then register some core components as extensions for
> the sake of interdependencies here, too.

Maybe that sort of refactoring could be done in 9.2 or further out.
I don't see it happening for 9.1.  I'm not really sure what the value
is anyway.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Range Types - cache lookup failed for function

[Jeff Davis]

On Sun, 2011-02-06 at 20:10 +0100, Erik Rijkers wrote:
> I was trying
> where intrange @> integer
> 
> which admittedly is not in the documentation,
> but does already half work, and would be really
> convenient to have.  As it stands the construct
> seems to fail after ANALYZE, when there is more
> than 1 row:

Thank you for the report! I actually did make some mention of that in
the documentation, albeit brief (in the operators table, using timestamp
as an example).

I have a fix for it. There may still be an issue with the constructors
like range(1,2), so I'll look into it a little more, but an updated
patch should come soon.

Regards,
Jeff Davis


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A different approach to extension NO USER DATA feature

[Florian Pflug]

On Feb6, 2011, at 19:23 , Tom Lane wrote:
> After a bit of thought I believe that we can fix this if we are willing
> to teach pg_dump explicitly about extension configuration tables.
> The behavior we want for those is for the table schema definition to
> never be dumped (the table should always be created by CREATE EXTENSION),
> but for some subset of the table data to get dumped, excluding any
> system-provided rows.  An extension that wants to make use of that
> ability would probably need to add a boolean column "system_data" or
> something similar to its configuration tables.  Having done so,
> the extension's SQL script could call a function that identifies the
> configuration table and tells how to decide which rows to dump,
> along the lines of

We could avoid the need for a per-row "system_data" flag if we required
extensions to split user-editable and system-provided configuration data
into different tables. For convenient access to the configuration data,
the extension could let the user-editable table inherit from the 
system-provided one, or use a view to combine the two.

We'd then only need a per-table flag which tells pg_dump to dump the
data (but not the structure), so instead of a

  pg_extension_partial_dump (table_name regclass, where_condition text)

we'd have

  pg_extension_userdata(table_name regclass).

Apart from getting rid of the slightly ugly userdata flag-column, this
would also make it easier for users to distinguish between user-editable
and system-provided configuration options.

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More extension issues: ownership and search_path

[Dimitri Fontaine]

Tom Lane  writes:
> One is ownership.  Since we don't record the identity of the user who
> created an extension, there's no way for pg_dump to ensure that it's
> recreated by the same user.  I think we'll regret that in future even
> if you don't think it's problematic today.  In particular, I foresee
> eventually allowing non-superusers to load extensions, so I think this
> is going to follow pretty much the same trajectory as procedural
> languages, which we originally did not track ownership for.  In short,
> I think we'd better add an extowner column to pg_extension.

Agreed.  There's no need to have it now but we will add it at some
point.  So if now is when that works the best for you, I'm happy to see
that happen :)

Would it help that I prepare some of those modifications, as patches
over the extension's patch that you started from?

> Another is the search_path setting.  Currently this is actually rather
> broken since pg_dump makes no effort at all to ensure that search_path
> is the same at reload time as it was when the extension was created,
> and thus the contained objects could easily go into the wrong schema.

Well there's some code to place the extension's schema at the first
place in the search_path before executing the script, already.

> But even without thinking about dump/reload, it seems to me that it
> would be a good thing for reproducibility if CREATE EXTENSION were to
> forcibly set search_path before running the extension's SQL script.
>
> The level-zero version of that would be to do the equivalent of
>   SET LOCAL search_path = @extschema@
> such that the path only contains the target schema and nothing else.

Spelled this way, I could see attaching SET to CREATE EXTENSION the same
way we did for CREATE FUNCTION.  I'm not too sure about what other set
of GUCs would be useful to support here, but that would be a good
mechanism to use I would say.

> The trouble with this simplistic approach is that it doesn't work nicely
> if one extension depends on another --- you probably want the search
> path to include the schema(s) the required extensions got installed
> into.  Of course inter-extension dependencies aren't going to work
> anyway unless pg_dump knows about them so it can make sure to load the
> extensions in the right order.  So where I think we're going to end up
> is adding a clause along the line of "USING list-of-extension-names"
> to CREATE EXTENSION, storing those dependencies explicitly, and having
> the CREATE EXTENSION code set search_path to the target schema followed
> by the target schema(s) of the USING extensions.  Not sure if this is
> worth doing immediately or can be left for 9.2.  At least in the contrib
> modules, there are no interdependencies, and I don't know whether any
> exist in the wider world of existing extensions.

We have a interdependency in contrib, earthdistance depends on cube
already.  In skytools, PGQ is composed of several parts that are
packaged as a single extension now, but whose sources are maintained in
separate parts.  Other than that, I think tricky scripts that depends on
some objects of the extension to already be usable will be simpler to
solve with splitting the extensions and adding some dependency.

So while we said this is 9.2 material, if you want to tackle the whole
search_path at restore time issue (I did only the creation namespace,
thinking it would be enough) fully, you need dependency too.

I think we should then register some core components as extensions for
the sake of interdependencies here, too.  \dx would then list PostgreSQL
itself and its (major) version, and the installed PL would need to be
there, and maybe some "features" too — but the way we handle bugfix only
minor upgrade makes that useless for us I think.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] exposing COPY API

[Andrew Dunstan]

On 02/04/2011 05:49 AM, Itagaki Takahiro wrote:

Here is a demonstration to support jagged input files. It's a patch
on the latest patch. The new added API is:

   bool NextLineCopyFrom(
 [IN] CopyState cstate,
 [OUT] char ***fields, [OUT] int *nfields, [OUT] Oid *tupleOid)

It just returns separated fields in the next line. Fortunately, I need
no extra code for it because it is just extracted from NextCopyFrom().

I'm willing to include the change into copy APIs,
but we still have a few issues. See below.



I've looked at this, and I think it will do what I want. I haven't had 
time to play with it, although I hope to soon.  AIUI, it basically hands 
back the raw parsed strings to the user, who then has the responsibility 
of constructing Datum and Nulls arrays to form the tuple.  That should 
be all I need. So +1 from me for including it. In fact, +10. And many 
thanks.



I think we need a better name though. NextCopyFromRawFields maybe.


On Fri, Feb 4, 2011 at 16:53, Andrew Dunstan  wrote:

The problem with COPY FROM is that nobody's come up with a good syntax for
allowing it as a FROM target. Doing what I want via FDW neatly gets us
around that problem. But I'm quite OK with doing the hard work inside the
COPY code - that's what my working prototype does in fact.

I think it is not only syntax issue. I found an issue that we hard to
support FORCE_NOT_NULL option for extra fields. See FIXME in the patch.
It is a fundamental problem to support jagged fields.


I don't think we need to worry about it. The caller will have access to 
the raw strings so they can handle it. In fact, I'd take out that bit of 
code in NextCopyLine_From, and replace it with a comment about how it's 
the caller's responsibility to handle.



One thing I'd like is to to have file_fdw do something we can't do another
way. currently it doesn't, so it's nice but uninteresting.

BTW, how do you determine which field is shifted in your broken CSV file?
For example, the case you find "AB,CD,EF" for 2 columns tables.
I could provide a raw CSV reader for jagged files, but you still have to
cook the returned fields into a proper tuple...




I answered this previously, but in the case of a text array it won't 
even arise - the array will have however many fields have been read.



cheers

andrew


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Simon Riggs]

On Mon, 2011-02-07 at 09:55 -0500, Robert Haas wrote:
> On Sun, Jan 30, 2011 at 11:44 AM, Robert Haas  wrote:
> > Time's running short - do you have an updated patch?
> 
> This patch hasn't been updated in more than three weeks.  I assume
> this should now be marked Returned with Feedback, and we'll revisit
> synchronous replication for 9.2?

Hi Robert,

I have time to complete that in next two weeks, but you are right I
haven't had it in last few weeks.

Cheers

-- 
 Simon Riggs   http://www.2ndQuadrant.com/books/
 PostgreSQL Development, 24x7 Support, Training and Services
 


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A different approach to extension NO USER DATA feature

[Tom Lane]

Dimitri Fontaine  writes:
> Tom Lane  writes:
>> one I'd been thinking about a bit was OIDs of modules this one depends
>> on.  The current design doesn't cope very well with modules that depend
>> on other ones.

> Or even at all.  I guess here "modules" is referring to shared object
> libraries, right?  Or are you already thinking about extension that
> depend on other extensions, like earthdistance depends on cube?

Sorry, I meant module == extension.  If it's not intended that we try to
support dependent extensions yet, I'd be fine with leaving that for 9.2.
However, if earthdistance already has such a dependency, maybe we can't
put that issue off.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sync Rep for 2011CF1

[Bruce Momjian]

Robert Haas wrote:
> On Sun, Jan 30, 2011 at 11:44 AM, Robert Haas  wrote:
> > Time's running short - do you have an updated patch?
> 
> This patch hasn't been updated in more than three weeks.  I assume
> this should now be marked Returned with Feedback, and we'll revisit
> synchronous replication for 9.2?

Seems it is time for someone else to take the patch and complete it? 
Who can do that?

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A different approach to extension NO USER DATA feature

[Dimitri Fontaine]

Tom Lane  writes:
> one I'd been thinking about a bit was OIDs of modules this one depends
> on.  The current design doesn't cope very well with modules that depend
> on other ones.

Or even at all.  I guess here "modules" is referring to shared object
libraries, right?  Or are you already thinking about extension that
depend on other extensions, like earthdistance depends on cube?

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Where the Quals are actually 'List'ed

[Tom Lane]

Vaibhav Kaushal  writes:
> Hi,
> I find that ExecInitExpr creates a ExprState tree (and so say the
> comments above the function in the source). Also, it seems to decide
> which function would get called when the expression is to be evaluated
> when ExecQual runs, by setting the function pointer, for example: 

> bstate->xprstate.evalfunc = (ExprStateEvalFunc)ExecEvalAnd;

> But ExecQual goes through a List, whereas, ExecInitExpr creates a tree. 
> So is that same tree converted to a qual list or are we adding some more
> information.

ExecInitExpr produces a List of state trees from a List of expr trees
--- look at the last case in its switch.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Varlena and binary

[Tom Lane]

=?utf-8?q?Rados=C5=82aw_Smogura?=  writes:
> I'm sending small patch for textsend. It reduces unnecessary copies, and 
> memory usage for duplication of varlena data. May you look?

This code will break the day that text and bytea don't have the same
internal representation, which seems likely to be soon.  Barring some
compelling evidence of a major performance improvement obtainable this
way, I don't think we want this patch.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PERFORM] pgbench to the MAXINT

[Greg Smith]

The update on the work to push towards a bigger pgbench is that I now 
have the patch running and generating databases larger than any 
previously possible scale:


$ time pgbench -i -s 25000 pgbench
...
25 tuples done.
...
real258m46.350s
user14m41.970s
sys0m21.310s

$ psql -d pgbench -c "select 
pg_size_pretty(pg_relation_size('pgbench_accounts'));"

pg_size_pretty

313 GB

$ psql -d pgbench -c "select 
pg_size_pretty(pg_relation_size('pgbench_accounts_pkey'));"

pg_size_pretty

52 GB

$ time psql -d pgbench -c "select count(*) from pgbench_accounts"
  count   


25

real18m48.363s
user0m0.010s
sys0m0.000s

The only thing wrong with the patch sent already needed to reach this 
point was this line:


for (k = 0; k < naccounts * scale; k++)

Which needed a (int64) cast for the multiplied value in the middle there.

Unfortunately the actual test itself doesn't run yet.  Every line I see 
when running the SELECT-only test says:


client 0 sending SELECT abalance FROM pgbench_accounts WHERE aid = 1;

So something about the updated random generation code isn't quite right 
yet.  Now that I have this monster built, I'm going to leave it on the 
server until I can sort that out, which hopefully will finish up in the 
next day or so.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us
"PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] More extension issues: ownership and search_path

[Tom Lane]

There are some things that the current extensions patch leaves
indeterminate during a dump and reload cycle, which strikes me
as a bad thing.

One is ownership.  Since we don't record the identity of the user who
created an extension, there's no way for pg_dump to ensure that it's
recreated by the same user.  I think we'll regret that in future even
if you don't think it's problematic today.  In particular, I foresee
eventually allowing non-superusers to load extensions, so I think this
is going to follow pretty much the same trajectory as procedural
languages, which we originally did not track ownership for.  In short,
I think we'd better add an extowner column to pg_extension.

Another is the search_path setting.  Currently this is actually rather
broken since pg_dump makes no effort at all to ensure that search_path
is the same at reload time as it was when the extension was created,
and thus the contained objects could easily go into the wrong schema.
But even without thinking about dump/reload, it seems to me that it
would be a good thing for reproducibility if CREATE EXTENSION were to
forcibly set search_path before running the extension's SQL script.

The level-zero version of that would be to do the equivalent of
SET LOCAL search_path = @extschema@
such that the path only contains the target schema and nothing else.

The trouble with this simplistic approach is that it doesn't work nicely
if one extension depends on another --- you probably want the search
path to include the schema(s) the required extensions got installed
into.  Of course inter-extension dependencies aren't going to work
anyway unless pg_dump knows about them so it can make sure to load the
extensions in the right order.  So where I think we're going to end up
is adding a clause along the line of "USING list-of-extension-names"
to CREATE EXTENSION, storing those dependencies explicitly, and having
the CREATE EXTENSION code set search_path to the target schema followed
by the target schema(s) of the USING extensions.  Not sure if this is
worth doing immediately or can be left for 9.2.  At least in the contrib
modules, there are no interdependencies, and I don't know whether any
exist in the wider world of existing extensions.

Comments?

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] limiting hint bit I/O

[Bruce Momjian]

Robert Haas wrote:
> On Sat, Feb 5, 2011 at 4:31 PM, Bruce Momjian  wrote:
> > Uh, in this C comment:
> >
> > + ? ? ? ?* or not we want to take the time to write it. ?We allow up to 5% 
> > of
> > + ? ? ? ?* otherwise-not-dirty pages to be written due to hint bit changes,
> >
> > 5% of what? ?5% of all buffers? ?5% of all hint-bit-dirty ones? ?Can you
> > clarify this in the patch?
> 
> 5% of buffers that are hint-bit-dirty but not otherwise dirty.  ISTM
> that's exactly what the comment you just quoted says on its face, but
> I'm open to some other wording you want to propose.

How about:

otherwise-not-dirty -> only-hint-bit-dirty

So 95% of your hint bit modificates are discarded if the pages is not
otherwise dirtied?  That seems pretty radical.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers