Re: [HACKERS] Extension of Thick Indexes

2009-03-20 Thread Gokulakannan Somasundaram
 It would be helpful to explain how this solves the lack of atomicity of
 visibility data updates, which last time I checked was the killer
 problem for this feature.


Hmmm... To put it more clearly, this problem occurs when there is a thick
index on a mutable function(marked as immutable). In order to avoid the
problem, i wrote the code that would not support functional indexes, it
would only support the normal ones. I  think the main argument against Thick
Index was
 - Visibility Map, which supports Index only Scans partially but by
occupying lesser space and doesn't have the functional index issue. Since
the main advantage of Thick Index was Index Only Scans, the community
preferred to wait for Visibility map

Heikki is working on the Visibility map and i think his observations might
help on Thick Index project.

Thanks,
Gokul.


Re: [HACKERS] Have \d show child tables that inherit from the specified parent

2009-03-20 Thread damien clochard

Greg Sabino Mullane wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160



This one is very basic, it just shows the child tables of a specific
table when you type \d tablename in psql :


I'm not so jazzed about this, as I work on systems that have literally
hundreds of child tables.


hi everyone.

Thanks for your comments.

I must admit that i didn't asked myself much questions when i 
implemented that feature. As i said i picked that item almost randomly 
in the todo list. I choosed to show child table on \d instead \d+ in 
order to be consistant with the fact that \d shows the mother of the 
specified table.


That being said i recognize that people who have dozens of child tables 
might get annoyed by this patch.


Here's 3 ideas that could make things better :

1- When using \d , only show the numbers of child tables

2- Only display the list of child table when \d+ is used. (mode ???)

3- Display the child table names in line, instead of one line per table.

Here's what it would look like :

# \d mother
 Table public.mother
 Column |  Type   |  Modifiers
+-+-
 id | integer | not null default nextval('mother_id_seq'::regclass)
This table has 5 child tables. Use \d+ to display them.

#\d+ mother
Table public.mother
 Column |  Type   |  Modifiers
+-+-
 id | integer | not null default nextval('mother_id_seq'::regclass)
Child tables : daughter1, daughter2, daughter3, daughter4, daughter5


Does it look better ?

--
damien clochard
dalibo.org | dalibo.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Extension of Thick Indexes

2009-03-20 Thread Heikki Linnakangas

Gokulakannan Somasundaram wrote:

It would be helpful to explain how this solves the lack of atomicity of
visibility data updates, which last time I checked was the killer
problem for this feature.


Hmmm... To put it more clearly, this problem occurs when there is a thick
index on a mutable function(marked as immutable). In order to avoid the
problem, i wrote the code that would not support functional indexes, it
would only support the normal ones. I  think the main argument against Thick
Index was
 - Visibility Map, which supports Index only Scans partially but by
occupying lesser space and doesn't have the functional index issue. Since
the main advantage of Thick Index was Index Only Scans, the community
preferred to wait for Visibility map

Heikki is working on the Visibility map and i think his observations might
help on Thick Index project.


The common ground between thick indexes and visibility map based 
index-only-scans is the method to pass Datums from the index to the 
executor, and all the executor and planner changes to take advantage of 
that. In fact, even without thick indexes or visibility map, that would 
provide some gain: we could use the data from the index to filter rows 
before going to the heap to check visibility. For example if you have a 
wide table with a text column, and there's an index on the text column, 
it would be faster to do a full scan on the index for a predicate like 
textcol LIKE '%foobar%', than a sequential scan of the heap, assuming 
there's only few matches.


The thick index approach has a lot of limitations compared to using 
visibility map: How to deal with functional indexes? How to deal with 
datatypes with broken comparison functions? And it needs support from 
each indexam separately, and is outright impossible in something like an 
bitmap index. These have all been discussed before, but I believe the 
executor and indexam API changes required are similar to that with the 
visibility map. That part of the patch I'm very interested in, though I 
haven't looked at it at all yet.


--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Extension of Thick Indexes

2009-03-20 Thread Shrish Purohit
Hi All,

Some brief information about the thick index patch.

The patch adds snapshot (MVCC) information to the indexes to enable them
being used independently.  With this information, the indexes need not refer
to the heap data to check an index key's visibility. Various functions such
as IndexTupleSatisfiesMVCC() are uesd to verify if the MVCC satisfies a
snapshot.

The following data structure is added to the thick index key data structure:
struct SnapshotFieldsData
 {
   TransactionId t_xmin;
   TransactionId t_xmax;
   TransactionId t_cid;
   uint16 infomask;
 };

Thus the pg_index catalog is added with flag to indicate thick index.
(FormData_pg_index ) is modified accordingly.

The IndexScanDescData structure, which stores information about how an index
can be scanned, was modified to include:
SnapshotIndexParams  sindex_params; 
// params of thick index that can be used for scanning
void *opaque;

The following data data-structure is used by the database execution engine
to scan thick index:

 typedef struct SnapshotIndexParamsData
 {
   bool isIndexOnlyScan;
// true if index should be used without accessing the table.
   bool is_minmax_scan;
// true if this the query contains min/max aggregate
functions.
   bool is_count_only_scan;
// if the query contains only count(), then true. 
// If true, then index rows are not cached.
   bool cache_stack;
// If true, remembers path from root to leaf in the stack (a
member 
// var defined below). Used For optimization of updates of a row, which 
// requires delete and insert.  
// While deleting a key from index, its stack is tracked, and the same stack

// is used for inserting the updated key.
   CmdType  operation;
// insert, update, delete, .
   HeapTuplehtup;
// reference to the heap tuple that's being operated
   ScanKey  insertion_skey;  /* Insertion Scan Key */
   void*  stack;
// reference to the stack mentioned in comments of cache_stack.
   TupleTableSlot* slot;
//  stores heap data structures containing index keys
 } SnapshotIndexParamsData;

IndexScan data structure, which stores information about how an index is
scanned, is updated to include following members:
Bool indexOnlyScan;
Bool is_minmax_scan;
Bool is_count_only_scan;

The Postgres optimizer figures out various ways to access each relation used
by the input SQL statement in set_plain_rel_pathlist() function, which
indirectly invokes create_index_path(). Create_index_path()  does the
following:
Identifies index path to access a table, figures out if index covers all the
columns used by the SQL statement to initialize index_covers_all_clauses
sets the cost of accessing table using index (or just using index only scan)
by invoking cost_index() function.

Functions index_covers_rel_clauses, index_covers_having_quals,
index_covers_target_list are used to set flag index_covers_all_clauses. 
index_covers_all_clauses flag indicates that index covers all clauses and
can be used In IndexOnlyScan mode.

--How updates and deletes are performed? 

find_old_slot(estate) identifies the old_slot information from estates'
ExprContexts. ExecUpdate uses oldslot, newslot and function
ExecUpdateIndexTuples is used to update the index. ExecUpdateIndexTuples
prepares iscan IndexScanDescriptor which is used to scan index.
Index is scanded by index_getnext(iscan,ForwardScanDirection).for b-tree
index it internall calls _bt_readpage. for update and delete operations
_bt_readpage calls SnapshotIndexDeleteTuple which updates mvcc information
of the index tuple based on MVCC information includeed in the slot.
ExecUpdateIndexTuples then inserts the new index tuple. 

FormIndexDatum stores the MVCC information along with values []. 
index_form_tuple is also modified accordingly to store MVCC information. 

Functions StoreMinimalIndexTuple and GetNextMinimalIndexTuple are used to
store index key column values into mintup and to get values from mintup
respectively.

For countOnlyScans we count same static slot, while for minmax_scan first
non-null tuple as the tuple with minium/maximum value.


Thanks,
Shrish Purohit | Software Engineer | Persistent Systems
shrish_puro...@persistent.co.in  | Cell: +91 98509 59940 | Tel: +91 (20)
3023 4493
Innovation in software product design, development and delivery-
www.persistentsys.com

-Original Message-
From: Heikki Linnakangas [mailto:heikki.linnakan...@enterprisedb.com] 
Sent: Friday, March 20, 2009 2:01 PM
To: Gokulakannan Somasundaram
Cc: Alvaro Herrera; Josh Berkus; Shrish Purohit; Amit Gupta;
pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Extension of Thick Indexes

Gokulakannan Somasundaram wrote:
 It would be helpful to explain how this solves the lack of atomicity of
 visibility data updates, which last time I checked was the 

Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Robert Treat
On Tuesday 17 March 2009 09:38:59 Bruce Momjian wrote:
 You are assuming that only commit-fest work is required to get us to
 beta.  You might remember the long list of open items I faced in January
 that I have whittled down, but I still have about twenty left.


I think part of the perception of the project sitting around doing nothing 
isn't so much that you/tom are doing nothing, but others who were doing 
review or coding features are now caught in limbo. One of the things the 
commitfest has been successful at is helping delegate code review. Perhaps we 
need to take a fresh look at your list of twenty things and see what can be 
delegated out to others. 

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] cs_CZ vs regression tests, part N

2009-03-20 Thread Peter Eisentraut

Tom Lane wrote:

It's still broken:
http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=gothic_mothdt=2009-03-17%2021:06:01

I remain of the opinion that supporting the regression tests in a locale
that works like this is more trouble than it's worth.


We looked into this and it turns out that this is actually a C locale 
vs. all-other-locales issue; not specific to Czech.  More specifically, 
it is related to the sorting of upper-case vs. lower-case letters.  A 
_1.out file is an appropriate answer here.


(Hmm, no one except Zdenek testing locales yet on the build farm?  Can't 
easily tell from the index page.)


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] cs_CZ vs regression tests, part N

2009-03-20 Thread Andrew Dunstan



Peter Eisentraut wrote:

Tom Lane wrote:

It's still broken:
http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=gothic_mothdt=2009-03-17%2021:06:01 



I remain of the opinion that supporting the regression tests in a locale
that works like this is more trouble than it's worth.


We looked into this and it turns out that this is actually a C locale 
vs. all-other-locales issue; not specific to Czech.  More 
specifically, it is related to the sorting of upper-case vs. 
lower-case letters.  A _1.out file is an appropriate answer here.


(Hmm, no one except Zdenek testing locales yet on the build farm?  
Can't easily tell from the index page.)





Yes, dungbeetle is ... I took tcl out of its config until we had this 
sorted out. Are you going to commit a _1.out file?


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Bruce Momjian
Robert Treat wrote:
 On Tuesday 17 March 2009 09:38:59 Bruce Momjian wrote:
  You are assuming that only commit-fest work is required to get us to
  beta.  You might remember the long list of open items I faced in January
  that I have whittled down, but I still have about twenty left.
 
 
 I think part of the perception of the project sitting around doing nothing 
 isn't so much that you/tom are doing nothing, but others who were doing 
 review or coding features are now caught in limbo. One of the things the 
 commitfest has been successful at is helping delegate code review. Perhaps we 
 need to take a fresh look at your list of twenty things and see what can be 
 delegated out to others. 

Yep, I agree.  The problem is that last time I put out a list that
wasn't clensed I got a lot of compaints so I am only going to put out a
list that is 100% accurate, and that will take hours to produce, time I
don't have now because I am working on the release notes.

And things are getting addressed, like the pg_restore -j/-m flag and GIN
index changes, so there is movement, there just isn't documented
movement.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread David Fetter
On Tue, Mar 17, 2009 at 09:38:59AM -0400, Bruce Momjian wrote:
 Robert Haas wrote:
   Well, we have been working on stuff for the past month so it was not
   like we were waiting on SE-PG to move forward.
  
  Stuff related to the CommitFest?
  
  AFAICS, the only committer who has done any significant review or
  committing of patches in the last month is Heikki, who extensively
  reworked and then committed infrastructure changes for recovery on
  February 18th (2 days shy of a month ago) and then extensively
  reviewed Hot Standby and SE-PostgreSQL.  It's really, really good that
  those patches have finally received some extensive review, both
  because now some version of each of them will likely make it into 8.5,
  and because now we have only a handful of patches left that Tom has
  said are pretty close to being committable.  But I don't see how you
  can say it didn't delay the release.
 
 You are assuming that only commit-fest work is required to get us to
 beta.  You might remember the long list of open items I faced in January
 that I have whittled down, but I still have about twenty left.

What are those, and are there any you can delegate, or at least shout
out for help on?

 Tom has done work fixing optimizer bugs introduced in 8.4.  I have had
 EnterpriseDB work to do and am working on the release notes now.  The
 bottom line is that there is lots of cleanup required to get to beta
 independent of the last commit fest work.

How much of this cleanup work can be distributed?

 I agree if we had said no to those patches we could be farther
 now, but I am not sure how much farther.

One way to find out is to make a list of all the things that happen
and see how to get more people, productively, on it :)

Cheers,
David.
-- 
David Fetter da...@fetter.org http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter  XMPP: david.fet...@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] cs_CZ vs regression tests, part N

2009-03-20 Thread Zdenek Kotala

Peter Eisentraut píše v pá 20. 03. 2009 v 15:33 +0200:

 (Hmm, no one except Zdenek testing locales yet on the build farm?  Can't 
 easily tell from the index page.)

It seems to me like good idea to add column with locale list tested on a
animal.

Zdenek


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Tom Lane
Bruce Momjian br...@momjian.us writes:
 Robert Treat wrote:
 ... Perhaps we 
 need to take a fresh look at your list of twenty things and see what can be 
 delegated out to others. 

 Yep, I agree.  The problem is that last time I put out a list that
 wasn't clensed I got a lot of compaints so I am only going to put out a
 list that is 100% accurate, and that will take hours to produce, time I
 don't have now because I am working on the release notes.

I complained to Bruce about this off-list a couple days ago --- doing
the release notes before putting up the TODO list is backwards,
precisely because it serializes everybody else on the release note work.

Personally I've been distracted by some Red Hat responsibilities, but
am now clear of the main one and will get back to reviewing the last
couple of patches today.  I think we should be focusing on getting
things done with the idea that we are trying to be ready for beta
in a week or two.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] xpath processing brain dead

2009-03-20 Thread Andrew Dunstan



Andrew Dunstan wrote:



Hannu Krosing wrote:

Is it just that in you _can't_ use Xpath on fragments, and you _need_ to
pass full documents to Xpath ?
At least this is my reading of Xpath standard.

  


I think that's possibly overstating it., unless I have missed 
something (W3 standards are sometimes not much more clear than the SQL 
standards ;-( )


For instance, there's this, that implies at least that the tree might 
not be a document:


   A / at the beginning of a path expression is an abbreviation for
   the initial step fn:root(self::node()) treat as document-node()/
   (however, if the / is the entire path expression, the trailing /
   is omitted from the expansion.) The effect of this initial step is
   to begin the path at the root node of the tree that contains the
   context node. If the context item is not a node, a type error is
   raised [err:XPTY0020]. At evaluation time, if the root node above
   the context node is not a document node, a dynamic error is raised
   [err:XPDY0050].

The problem is that we certainly do have to provide a context node 
(the standard is clear about that), and unless we want to convert a 
non-document to a node-set as James suggested and then apply the xpath 
expression to each node in the node-set, we have no way of sanely 
specifying the context node.




No-one has come up with an answer to this, so I propose to remove the 
hackery. That leaves the question of what to do when the xml is not a 
well formed document ... raise an error?


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 10:34 AM, Bruce Momjian br...@momjian.us wrote:
 Robert Treat wrote:
 On Tuesday 17 March 2009 09:38:59 Bruce Momjian wrote:
  You are assuming that only commit-fest work is required to get us to
  beta.  You might remember the long list of open items I faced in January
  that I have whittled down, but I still have about twenty left.
 

 I think part of the perception of the project sitting around doing nothing
 isn't so much that you/tom are doing nothing, but others who were doing
 review or coding features are now caught in limbo. One of the things the
 commitfest has been successful at is helping delegate code review. Perhaps we
 need to take a fresh look at your list of twenty things and see what can be
 delegated out to others.

 Yep, I agree.  The problem is that last time I put out a list that
 wasn't clensed I got a lot of compaints so I am only going to put out a
 list that is 100% accurate, and that will take hours to produce, time I
 don't have now because I am working on the release notes.

I don't want to be a grump, but this is a false dichotomy.  What you
put out last time was a dump of 700 emails, much of which was
irrelevant and most of the rest of which was duplicative of itself or
the CommitFest wiki.  Now, it may be true that even if your list was
80% accurate, people would still have complained about the other 20%,
but we don't know that, because the actual list was at best 10%
accurate, and of course people are going to complain about that.

I personally think that the way pgsql-hackers organizes itself using
email is completely insane.  The only reason that you need to write
the release notes instead of, say, me, is because the only information
on what needs to go into them is buried in a thicket of CVS commit
messages that I am not nearly brave enough to attempt to penetrate.  I
suggested putting them in CVS yesterday; Tom didn't like that, but
what about a wiki page or a database?  grep 'release notes'
/last/six/months/of/email can't possibly be the best way to do this.
Given any sort of list to work from, even one that is totally
disorganized and written in broken English, I can't believe this is
more than an hour or two of work, and I'd be more than happy to take a
crack at it (I'm probably not the only one, either).

Similarly, the only reason we don't have a workable TODO list is
because you're attempting to extract it from a disorganized jumble of
email after the fact, instead of maintaining it publicly and adding
and removing items along the way.  It might be slightly more work to
think up a reasonable label for an action item at the time you learn
about it than to just dump the email in a folder, but I think the time
you didn't have to spend sorting through it later would more than make
up for it.  Plus then items could be worked on along the way instead
of waiting until the bitter end when the TODO list materializes and we
all say Oh, really?.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Heikki Linnakangas

Robert Haas wrote:

Similarly, the only reason we don't have a workable TODO list is
because you're attempting to extract it from a disorganized jumble of
email after the fact, instead of maintaining it publicly and adding
and removing items along the way.  It might be slightly more work to
think up a reasonable label for an action item at the time you learn
about it than to just dump the email in a folder, but I think the time
you didn't have to spend sorting through it later would more than make
up for it.  Plus then items could be worked on along the way instead
of waiting until the bitter end when the TODO list materializes and we
all say Oh, really?.


The TODO list is already on the Wiki. I've edited it a few times when 
I've spotted TODO-worthy ideas on the mailing lists.


--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Alvaro Herrera
Robert Haas escribió:

 I personally think that the way pgsql-hackers organizes itself using
 email is completely insane.

Note that during the 8.4 timeframe we've stolen a lot of work from
Bruce.  The TODO list was moved to the wiki, for one; the patch queue
was also moved to the wiki.  Now the FAQ has moved to wiki (and has
already seen lots of improvement, so it was clearly a good move).
Previously this was all handled as email boxes, so while some
inefficiences in the process still remain, we're a lot better than we
were in the 8.3 cycle.

 Similarly, the only reason we don't have a workable TODO list is
 because you're attempting to extract it from a disorganized jumble of
 email after the fact, instead of maintaining it publicly and adding
 and removing items along the way.

We do have an alternative open items list,
http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Open_Items
However, it's incomplete.  It is a bit sad that nobody can complete it,
because Bruce has taken pgpatches out of the air.  (Of course, anybody
could go fetch all the pgsql-hackers emails and dig up the remaining
open items to add them there, but that would be duplicative of the
effort Bruce has already put into his own queue).

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Bruce Momjian

This is about the reaction I expected, and is again so far off the mark
that I will just continue doing what I think is best.

Why doesn't someone offer to take my mbox file and generate a list from
that?

---

Robert Haas wrote:
 On Fri, Mar 20, 2009 at 10:34 AM, Bruce Momjian br...@momjian.us wrote:
  Robert Treat wrote:
  On Tuesday 17 March 2009 09:38:59 Bruce Momjian wrote:
   You are assuming that only commit-fest work is required to get us to
   beta. ?You might remember the long list of open items I faced in January
   that I have whittled down, but I still have about twenty left.
  
 
  I think part of the perception of the project sitting around doing nothing
  isn't so much that you/tom are doing nothing, but others who were doing
  review or coding features are now caught in limbo. One of the things the
  commitfest has been successful at is helping delegate code review. Perhaps 
  we
  need to take a fresh look at your list of twenty things and see what can be
  delegated out to others.
 
  Yep, I agree. ?The problem is that last time I put out a list that
  wasn't clensed I got a lot of compaints so I am only going to put out a
  list that is 100% accurate, and that will take hours to produce, time I
  don't have now because I am working on the release notes.
 
 I don't want to be a grump, but this is a false dichotomy.  What you
 put out last time was a dump of 700 emails, much of which was
 irrelevant and most of the rest of which was duplicative of itself or
 the CommitFest wiki.  Now, it may be true that even if your list was
 80% accurate, people would still have complained about the other 20%,
 but we don't know that, because the actual list was at best 10%
 accurate, and of course people are going to complain about that.
 
 I personally think that the way pgsql-hackers organizes itself using
 email is completely insane.  The only reason that you need to write
 the release notes instead of, say, me, is because the only information
 on what needs to go into them is buried in a thicket of CVS commit
 messages that I am not nearly brave enough to attempt to penetrate.  I
 suggested putting them in CVS yesterday; Tom didn't like that, but
 what about a wiki page or a database?  grep 'release notes'
 /last/six/months/of/email can't possibly be the best way to do this.
 Given any sort of list to work from, even one that is totally
 disorganized and written in broken English, I can't believe this is
 more than an hour or two of work, and I'd be more than happy to take a
 crack at it (I'm probably not the only one, either).
 
 Similarly, the only reason we don't have a workable TODO list is
 because you're attempting to extract it from a disorganized jumble of
 email after the fact, instead of maintaining it publicly and adding
 and removing items along the way.  It might be slightly more work to
 think up a reasonable label for an action item at the time you learn
 about it than to just dump the email in a folder, but I think the time
 you didn't have to spend sorting through it later would more than make
 up for it.  Plus then items could be worked on along the way instead
 of waiting until the bitter end when the TODO list materializes and we
 all say Oh, really?.
 
 ...Robert

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Bruce Momjian
Alvaro Herrera wrote:
 Robert Haas escribi?:
 
  I personally think that the way pgsql-hackers organizes itself using
  email is completely insane.
 
 Note that during the 8.4 timeframe we've stolen a lot of work from
 Bruce.  The TODO list was moved to the wiki, for one; the patch queue
 was also moved to the wiki.  Now the FAQ has moved to wiki (and has
 already seen lots of improvement, so it was clearly a good move).
 Previously this was all handled as email boxes, so while some
 inefficiences in the process still remain, we're a lot better than we
 were in the 8.3 cycle.

Please take as much work from me as you can.  However, looking at the
TODO commits, I am still the one doing most of the changes:

http://wiki.postgresql.org/index.php?title=Todoaction=history

  Similarly, the only reason we don't have a workable TODO list is
  because you're attempting to extract it from a disorganized jumble of
  email after the fact, instead of maintaining it publicly and adding
  and removing items along the way.
 
 We do have an alternative open items list,
 http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Open_Items
 However, it's incomplete.  It is a bit sad that nobody can complete it,
 because Bruce has taken pgpatches out of the air.  (Of course, anybody
 could go fetch all the pgsql-hackers emails and dig up the remaining
 open items to add them there, but that would be duplicative of the
 effort Bruce has already put into his own queue).

Yep, anyone can do what I am doing now;  there is no magic in my
fingers.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Alvaro Herrera
Bruce Momjian escribió:
 Alvaro Herrera wrote:

  Note that during the 8.4 timeframe we've stolen a lot of work from
  Bruce.  The TODO list was moved to the wiki, for one; the patch queue
  was also moved to the wiki.  Now the FAQ has moved to wiki (and has
  already seen lots of improvement, so it was clearly a good move).
  Previously this was all handled as email boxes, so while some
  inefficiences in the process still remain, we're a lot better than we
  were in the 8.3 cycle.
 
 Please take as much work from me as you can.  However, looking at the
 TODO commits, I am still the one doing most of the changes:
 
   http://wiki.postgresql.org/index.php?title=Todoaction=history

Yes, but there are commits from others too.  I think the TODO is seen as
a delicate document and other people is reluctant to edit it.


  We do have an alternative open items list,
  http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Open_Items
  However, it's incomplete.  It is a bit sad that nobody can complete it,
  because Bruce has taken pgpatches out of the air.
 
 Yep, anyone can do what I am doing now;  there is no magic in my
 fingers.

So are you going to publish your mbox?

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Bruce Momjian
Alvaro Herrera wrote:
 Bruce Momjian escribi?:
  Alvaro Herrera wrote:
 
   Note that during the 8.4 timeframe we've stolen a lot of work from
   Bruce.  The TODO list was moved to the wiki, for one; the patch queue
   was also moved to the wiki.  Now the FAQ has moved to wiki (and has
   already seen lots of improvement, so it was clearly a good move).
   Previously this was all handled as email boxes, so while some
   inefficiences in the process still remain, we're a lot better than we
   were in the 8.3 cycle.
  
  Please take as much work from me as you can.  However, looking at the
  TODO commits, I am still the one doing most of the changes:
  
  http://wiki.postgresql.org/index.php?title=Todoaction=history
 
 Yes, but there are commits from others too.  I think the TODO is seen as
 a delicate document and other people is reluctant to edit it.
 
 
   We do have an alternative open items list,
   http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Open_Items
   However, it's incomplete.  It is a bit sad that nobody can complete it,
   because Bruce has taken pgpatches out of the air.
  
  Yep, anyone can do what I am doing now;  there is no magic in my
  fingers.
 
 So are you going to publish your mbox?

After the complaints last time, no.  If everyone says they are not going
to complain, I will, or email it to people who ask for it (and if you
ask for it, don't complain either).

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Alvaro Herrera
Bruce Momjian escribió:
 Alvaro Herrera wrote:
  Bruce Momjian escribi?:

We do have an alternative open items list,
http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Open_Items
However, it's incomplete.  It is a bit sad that nobody can complete it,
because Bruce has taken pgpatches out of the air.
   
   Yep, anyone can do what I am doing now;  there is no magic in my
   fingers.
  
  So are you going to publish your mbox?
 
 After the complaints last time, no.  If everyone says they are not going
 to complain, I will, or email it to people who ask for it (and if you
 ask for it, don't complain either).

Well, I spent some time pointing out items to prune, and several others
did too.  Are you saying that we wasted our time doing that?

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Bruce Momjian
Alvaro Herrera wrote:
 Bruce Momjian escribi?:
  Alvaro Herrera wrote:
   Bruce Momjian escribi?:
 
 We do have an alternative open items list,
 http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Open_Items
 However, it's incomplete.  It is a bit sad that nobody can complete 
 it,
 because Bruce has taken pgpatches out of the air.

Yep, anyone can do what I am doing now;  there is no magic in my
fingers.
   
   So are you going to publish your mbox?
  
  After the complaints last time, no.  If everyone says they are not going
  to complain, I will, or email it to people who ask for it (and if you
  ask for it, don't complain either).
 
 Well, I spent some time pointing out items to prune, and several others
 did too.  Are you saying that we wasted our time doing that?

Yes, I did get some helpful feedback from publishing the list, but most
felt it was a waste in showing it (including Robert Haas), so I have no
desire to repeat that.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 1:08 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
 The TODO list is already on the Wiki. I've edited it a few times when I've
 spotted TODO-worthy ideas on the mailing lists.

Well, Bruce and Tom both made reference to something that Bruce was
going to produce along these lines...  I think what we are talking
about is open items for 8.4beta, not dev projects in general.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 1:08 PM, Alvaro Herrera
alvhe...@commandprompt.com wrote:
 I personally think that the way pgsql-hackers organizes itself using
 email is completely insane.
 Note that during the 8.4 timeframe we've stolen a lot of work from
 Bruce.  The TODO list was moved to the wiki, for one; the patch queue
 was also moved to the wiki.  Now the FAQ has moved to wiki (and has
 already seen lots of improvement, so it was clearly a good move).
 Previously this was all handled as email boxes, so while some
 inefficiences in the process still remain, we're a lot better than we
 were in the 8.3 cycle.

Good point.

 Similarly, the only reason we don't have a workable TODO list is
 because you're attempting to extract it from a disorganized jumble of
 email after the fact, instead of maintaining it publicly and adding
 and removing items along the way.

 We do have an alternative open items list,
 http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Open_Items
 However, it's incomplete.  It is a bit sad that nobody can complete it,
 because Bruce has taken pgpatches out of the air.  (Of course, anybody
 could go fetch all the pgsql-hackers emails and dig up the remaining
 open items to add them there, but that would be duplicative of the
 effort Bruce has already put into his own queue).

I don't even understand why we're interested in doing this.  If the
patches weren't important enough for someone to add them to the
CommitFest wiki in October, why are we delaying the release to hunt
for them in March?  I personally spent hours and hours of time in late
October dredging up every patch that looked anywhere close to
reviewable (not committable, reviewable!) and put them all on the
wiki.  Now, it's certainly possible that I missed something, but
there's been plenty of time between then and now and I think there's
only been 1 or 2 complaints about things being overlooked (and those
were very, very old patches from before I started reading the mailing
list regularly).

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Alvaro Herrera
Robert Haas escribió:

 I don't even understand why we're interested in doing this.  If the
 patches weren't important enough for someone to add them to the
 CommitFest wiki in October, why are we delaying the release to hunt
 for them in March?

The problem is not patches that were not committed, but rather loose
ends in patches that were.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 1:10 PM, Bruce Momjian br...@momjian.us wrote:
 This is about the reaction I expected, and is again so far off the mark
 that I will just continue doing what I think is best.

Would you like to explain WHY it's off the mark?

 Why doesn't someone offer to take my mbox file and generate a list from
 that?

A list of what?  CVS commit messages that needed to be added to the
release notes?  Patches that were committed but need follow-up work?
Patches that were missed and never added to the Wiki?  Other
release-related tasks that need to be done?

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Alvaro Herrera
Robert Haas escribió:
 On Fri, Mar 20, 2009 at 1:40 PM, Alvaro Herrera
 alvhe...@commandprompt.com wrote:
  Robert Haas escribió:
  I don't even understand why we're interested in doing this.  If the
  patches weren't important enough for someone to add them to the
  CommitFest wiki in October, why are we delaying the release to hunt
  for them in March?
  The problem is not patches that were not committed, but rather loose
  ends in patches that were.
 
 There seems to be a reasonably well-maintained list of open items here:
 
 http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Open_Items

Hey, dude, I pointed that URL in a followup to a message from you,
earlier today.

 Of course, if this list is radically incomplete, then it doesn't help
 much, but does anyone think that's the case?

We don't know -- Bruce's list may contain something, but since it's
hidden we can't do anything.  Maybe it is all already-completed items,
or redundant with the list on wiki, or maybe they're things that can be
postponed for 8.5.  No way to tell.

My guess is that Bruce's queue contains a reasonably small number of
minor fixups that need to happen before release.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 3:13 PM, Alvaro Herrera
alvhe...@commandprompt.com wrote:
 Robert Haas escribió:
 On Fri, Mar 20, 2009 at 1:40 PM, Alvaro Herrera
 alvhe...@commandprompt.com wrote:
  Robert Haas escribió:
  I don't even understand why we're interested in doing this.  If the
  patches weren't important enough for someone to add them to the
  CommitFest wiki in October, why are we delaying the release to hunt
  for them in March?
  The problem is not patches that were not committed, but rather loose
  ends in patches that were.

 There seems to be a reasonably well-maintained list of open items here:

 http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Open_Items

 Hey, dude, I pointed that URL in a followup to a message from you,
 earlier today.

So you did.  Sorry...  I seem to be rubbing everyone the wrong way
today.  (Did I hear someone say, who said anything about today?)

 Of course, if this list is radically incomplete, then it doesn't help
 much, but does anyone think that's the case?

 We don't know -- Bruce's list may contain something, but since it's
 hidden we can't do anything.  Maybe it is all already-completed items,
 or redundant with the list on wiki, or maybe they're things that can be
 postponed for 8.5.  No way to tell.

 My guess is that Bruce's queue contains a reasonably small number of
 minor fixups that need to happen before release.

Well, I guess there's no point in speculating.  We'll just have to
wait and see.  Bruce seems to feel that what he's doing is something
that no one else has done or is willing to do, and I guess I'm
wondering whether that's really the case.  I think a lot of people
have worked hard to make sure that all of the patches that were
submitted got dealt with and the open items resulting from those
patches were closed.  There is always more to do, of course.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Bruce Momjian
Robert Haas wrote:
  Of course, if this list is radically incomplete, then it doesn't help
  much, but does anyone think that's the case?
 
  We don't know -- Bruce's list may contain something, but since it's
  hidden we can't do anything. ?Maybe it is all already-completed items,
  or redundant with the list on wiki, or maybe they're things that can be
  postponed for 8.5. ?No way to tell.
 
  My guess is that Bruce's queue contains a reasonably small number of
  minor fixups that need to happen before release.
 
 Well, I guess there's no point in speculating.  We'll just have to
 wait and see.  Bruce seems to feel that what he's doing is something
 that no one else has done or is willing to do, and I guess I'm
 wondering whether that's really the case.  I think a lot of people
 have worked hard to make sure that all of the patches that were
 submitted got dealt with and the open items resulting from those
 patches were closed.  There is always more to do, of course.

I did offer to post my mbox file so people could see what I have as open
8.4 items, but the no complaining requirement seems to have eliminated
volunteers.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 3:23 PM, Bruce Momjian br...@momjian.us wrote:
 I did offer to post my mbox file so people could see what I have as open
 8.4 items, but the no complaining requirement seems to have eliminated
 volunteers.

IIRC, the biggest problem we had last time (apart from the
complaining) was that there was no easy way to refer to particular
items in the list.  If you could find a way to make the numbering
stable (first 8 characters of the SHA-1 hash of the contents, a la
git? drop every message into a file whose name is the initial number
of that messages, and keep the numbers the same as files are
removed?), it would be easier for people tell you which items could be
removed and why.  I think you suggested referencing by subject lines
last time, but the problem is that you had so many duplicates that it
was hard to explain which ones should be kept and which one should be
left, especially because you were making a lot of edits at the same
time (subject A three down from subject B ceases to be clear when
subject B has in the meantime been removed, or when one of the
intervening messages has been removed, thus making the one three down
a different message with the same subject).

Lest I be accused of offering suggestions in place of actual help, if
you send me whatever script you're using to put this on the web now I
would be willing to take a crack at making the changes described
above.  I really don't believe dumping everything into an mbox file is
the best way of keeping track of open to-do items - but if that's the
only choice, then we might as well try to make it as painless as
possible.  (I offered to write a webapp to attempt to improve upon the
CommitFest wiki a while ago, so we could eventually do things like...
run a command to dump the latest version of every outstanding patch
into a given directory... and a whole lot of people offered
suggestions for features...  but I haven't been able to connect up
with Magnus, so that's gone nowhere.)

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Bruce Momjian
Robert Haas wrote:
 On Fri, Mar 20, 2009 at 3:23 PM, Bruce Momjian br...@momjian.us wrote:
  I did offer to post my mbox file so people could see what I have as open
  8.4 items, but the no complaining requirement seems to have eliminated
  volunteers.
 
 IIRC, the biggest problem we had last time (apart from the
 complaining) was that there was no easy way to refer to particular
 items in the list.  If you could find a way to make the numbering
 stable (first 8 characters of the SHA-1 hash of the contents, a la
 git? drop every message into a file whose name is the initial number
 of that messages, and keep the numbers the same as files are
 removed?), it would be easier for people tell you which items could be
 removed and why.  I think you suggested referencing by subject lines
 last time, but the problem is that you had so many duplicates that it
 was hard to explain which ones should be kept and which one should be
 left, especially because you were making a lot of edits at the same
 time (subject A three down from subject B ceases to be clear when
 subject B has in the meantime been removed, or when one of the
 intervening messages has been removed, thus making the one three down
 a different message with the same subject).
 
 Lest I be accused of offering suggestions in place of actual help, if
 you send me whatever script you're using to put this on the web now I
 would be willing to take a crack at making the changes described
 above.  I really don't believe dumping everything into an mbox file is
 the best way of keeping track of open to-do items - but if that's the
 only choice, then we might as well try to make it as painless as
 possible.  (I offered to write a webapp to attempt to improve upon the
 CommitFest wiki a while ago, so we could eventually do things like...
 run a command to dump the latest version of every outstanding patch
 into a given directory... and a whole lot of people offered
 suggestions for features...  but I haven't been able to connect up
 with Magnus, so that's gone nowhere.)

You can use my emails to make a master list --- there is no need to make
mine the master.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 3:50 PM, Bruce Momjian br...@momjian.us wrote:
 You can use my emails to make a master list --- there is no need to make
 mine the master.

OK, good enough.  Can you post a link to the raw mbox file?

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Bruce Momjian
Robert Haas wrote:
 On Fri, Mar 20, 2009 at 3:50 PM, Bruce Momjian br...@momjian.us wrote:
  You can use my emails to make a master list --- there is no need to make
  mine the master.
 
 OK, good enough.  Can you post a link to the raw mbox file?

OK, done:

http://momjian.us/cgi-bin/pgsql/open

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 4:03 PM, Bruce Momjian br...@momjian.us wrote:
 Robert Haas wrote:
 On Fri, Mar 20, 2009 at 3:50 PM, Bruce Momjian br...@momjian.us wrote:
  You can use my emails to make a master list --- there is no need to make
  mine the master.

 OK, good enough.  Can you post a link to the raw mbox file?

 OK, done:

        http://momjian.us/cgi-bin/pgsql/open

The Download in mbox format link doesn't work.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Open 8.4 item list

2009-03-20 Thread Bruce Momjian
Here are some of the emails I consider open for 8.4:

http://momjian.us/cgi-bin/pgsql/open

(Same email, using updated subject line.)

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Open 8.4 item list

2009-03-20 Thread Bruce Momjian
Bruce Momjian wrote:
 Here are some of the emails I consider open for 8.4:
 
   http://momjian.us/cgi-bin/pgsql/open
 
 (Same email, using updated subject line.)

mbox download URL fixed.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Joshua D. Drake
On Fri, 2009-03-20 at 16:03 -0400, Robert Haas wrote:
 On Fri, Mar 20, 2009 at 4:03 PM, Bruce Momjian br...@momjian.us wrote:
  Robert Haas wrote:
  On Fri, Mar 20, 2009 at 3:50 PM, Bruce Momjian br...@momjian.us wrote:
   You can use my emails to make a master list --- there is no need to make
   mine the master.
 
  OK, good enough.  Can you post a link to the raw mbox file?
 
  OK, done:
 
 http://momjian.us/cgi-bin/pgsql/open
 
 The Download in mbox format link doesn't work.

Worked for me.

Joshua D. Drake


 
 ...Robert
 
-- 
PostgreSQL - XMPP: jdr...@jabber.postgresql.org
   Consulting, Development, Support, Training
   503-667-4564 - http://www.commandprompt.com/
   The PostgreSQL Company, serving since 1997


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 4:06 PM, Joshua D. Drake j...@commandprompt.com wrote:
 On Fri, 2009-03-20 at 16:03 -0400, Robert Haas wrote:
 On Fri, Mar 20, 2009 at 4:03 PM, Bruce Momjian br...@momjian.us wrote:
  Robert Haas wrote:
  On Fri, Mar 20, 2009 at 3:50 PM, Bruce Momjian br...@momjian.us wrote:
   You can use my emails to make a master list --- there is no need to make
   mine the master.
 
  OK, good enough.  Can you post a link to the raw mbox file?
 
  OK, done:
 
         http://momjian.us/cgi-bin/pgsql/open

 The Download in mbox format link doesn't work.

 Worked for me.

Ah, it's working now.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Open 8.4 item list

2009-03-20 Thread Bruce Momjian
Bruce Momjian wrote:
 Bruce Momjian wrote:
  Here are some of the emails I consider open for 8.4:
  
  http://momjian.us/cgi-bin/pgsql/open
  
  (Same email, using updated subject line.)
 
 mbox download URL fixed.

Let me add that my item tracking is more of a blob that expands and
contracts every day, rather than some fixed list.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] hstore improvements?

2009-03-20 Thread Andrew Gierth
 Josh == Josh Berkus j...@agliodbs.com writes:

  Tom Lane wrote:
  Josh Berkus j...@agliodbs.com writes:
  As an hstore user, I'd be fine with simply limiting it to 64K (or,
  heck, 8K) and throwing an error.  I'd also be fine with limiting
  keys to 255 bytes, although we'd have to warn people.
  Yeah, 255 might well be more of a problem than the other limit.  We
  could move to something like 10/22 or 12/20 split, which would give
  us 1KB/4MB or 4KB/1MB limits.

 Josh Anything you like.  What I'm saying is that I think I use
 Josh hstore more heavily than most people, and that if the limits
 Josh were as low as 255b/8K it wouldn't hurt me any.

 Josh I suppose 1K/4MB would allow OO-types to use hstore as an
 Josh object store, so you'll make them happy with a new foot gun.
 Josh Why not?

I decided to obviate the entire question and remove the limits
altogether (while still keeping the overhead the same, i.e. 8 bytes
per entry).

-- 
Andrew.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Open 8.4 item list

2009-03-20 Thread Merlin Moncure
On Fri, Mar 20, 2009 at 4:08 PM, Bruce Momjian br...@momjian.us wrote:
 Bruce Momjian wrote:
 Bruce Momjian wrote:
  Here are some of the emails I consider open for 8.4:
 
      http://momjian.us/cgi-bin/pgsql/open
 
  (Same email, using updated subject line.)

 mbox download URL fixed.

 Let me add that my item tracking is more of a blob that expands and
 contracts every day, rather than some fixed list.

how many of these things are really blocking 8.4? or is that not known?

merlin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Open 8.4 item list

2009-03-20 Thread Bruce Momjian
Merlin Moncure wrote:
 On Fri, Mar 20, 2009 at 4:08 PM, Bruce Momjian br...@momjian.us wrote:
  Bruce Momjian wrote:
  Bruce Momjian wrote:
   Here are some of the emails I consider open for 8.4:
  
   ? ? http://momjian.us/cgi-bin/pgsql/open
  
   (Same email, using updated subject line.)
 
  mbox download URL fixed.
 
  Let me add that my item tracking is more of a blob that expands and
  contracts every day, rather than some fixed list.
 
 how many of these things are really blocking 8.4? or is that not known?

Not known, and changes every day.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Pilot Project: transformationHook

2009-03-20 Thread Pavel Stehule
Hello

I am sending samples of possible future modules based on transformationHook

regards
Pavel Stehule


tmodules.tgz
Description: GNU Zip compressed data
*** ./src/backend/parser/parse_expr.c.orig	2009-03-17 16:20:18.0 +0100
--- ./src/backend/parser/parse_expr.c	2009-03-17 16:24:43.0 +0100
***
*** 36,41 
--- 36,44 
  
  bool		Transform_null_equals = false;
  
+ /* Hook for plugins to get control in transformExpr */
+ ParseExprTransform_hook_type ParseExprTransform_hook = NULL;
+ 
  static Node *transformParamRef(ParseState *pstate, ParamRef *pref);
  static Node *transformAExprOp(ParseState *pstate, A_Expr *a);
  static Node *transformAExprAnd(ParseState *pstate, A_Expr *a);
***
*** 100,105 
--- 103,117 
  Node *
  transformExpr(ParseState *pstate, Node *expr)
  {
+ 	if (ParseExprTransform_hook)
+ 		return (*ParseExprTransform_hook) (pstate, expr);
+ 	else
+ 		return standard_transformExpr(pstate, expr);
+ }
+  
+ Node *
+ standard_transformExpr(ParseState *pstate, Node *expr)
+ {
  	Node	   *result = NULL;
  
  	if (expr == NULL)
*** ./src/include/parser/parse_expr.h.orig	2009-03-17 16:25:34.0 +0100
--- ./src/include/parser/parse_expr.h	2009-03-17 16:38:29.0 +0100
***
*** 18,23 
--- 18,29 
  /* GUC parameters */
  extern bool Transform_null_equals;
  
+ /* Hook for parser plugin to get control in transformExpr */
+ typedef Node *(*ParseExprTransform_hook_type) (ParseState *pstate, Node *expr);
+ extern PGDLLIMPORT ParseExprTransform_hook_type ParseExprTransform_hook;
+ 
  extern Node *transformExpr(ParseState *pstate, Node *expr);
+ extern Node *standard_transformExpr(ParseState *pstate, Node *expr);
+ 
  
  #endif   /* PARSE_EXPR_H */

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 1:40 PM, Alvaro Herrera
alvhe...@commandprompt.com wrote:
 Robert Haas escribió:
 I don't even understand why we're interested in doing this.  If the
 patches weren't important enough for someone to add them to the
 CommitFest wiki in October, why are we delaying the release to hunt
 for them in March?
 The problem is not patches that were not committed, but rather loose
 ends in patches that were.

There seems to be a reasonably well-maintained list of open items here:

http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Open_Items

The only thing I can recall that is outstanding but not mentioned here
is the controversy over the behavior of the various \d commands.  But
even that is something that can be changed after getting feedback from
beta-users and beta-testers, and we might even have a better idea what
to do about it at that point.  Once we release, we're probably stuck
with whatever the behavior is at that point, but I think we've got
enough time between now and then that we don't need to get too worried
about it now.

Of course, if this list is radically incomplete, then it doesn't help
much, but does anyone think that's the case?  My impression is that
most of the major open items (e.g. follow-on cleanup patch for
column-level permissions, Kevin Grittner's planner issues) were tied
up some time ago.  If there are other outstanding issues, why can't
they just wait to 8.5?  I guess I'm just confused as to how this
process works (I'm new around here?).  As far as I can tell, the
committers are very careful about not committing stuff to the tree, so
that means that the tree pretty much always works and doesn't usually
contain too much that's half-baked.  So it would seem like that would
make going to beta mostly a matter of finishing all the committing,
and maybe addressing the documentation issues mentioned on the open
items list linked above.  I gather from Bruce's comments, and Tom's,
that there's more to it than that, but I'm sort of in the dark on the
details.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Open 8.4 item list

2009-03-20 Thread Jaime Casanova
On 3/20/09, Bruce Momjian br...@momjian.us wrote:
 Here are some of the emails I consider open for 8.4:

http://momjian.us/cgi-bin/pgsql/open


Item #3 Extending grant insert on tables to sequences

is not for 8.4, i haven't reviewed it to adjust the patch after column
privileges was applied. i will update it and put a patch for 8.5 ASAP

-- 
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] gettext, plural form and translation

2009-03-20 Thread Sergey Burladyan
Alvaro Herrera alvhe...@commandprompt.com writes:

 Care to submit a patch?

this is it, i divide it into two, first is change source and second is change
ru.po file for psql.

changelog:

 gettext-plural-test.patch
 - check ngettext in configure (HAVE_NGETTEXT), show warning if not. must be
 error, i agree with Peter, i think gettext without support of plural form can't
 compile .po file with it :(, but not sure, so for test it is only warning
 - new macros _P(s,p,n) for ngettext
 - HAVE_NGETTEXT always 1 in pg_config.h.win32
 - psql, remove (1 row), switch this string into _P(...) macros

 gettext-plural-ru-test.patch:
 - correct translation for 1 rows message

*** a/config/programs.m4
--- b/config/programs.m4
***
*** 193,198  AC_DEFUN([PGAC_CHECK_GETTEXT],
--- 193,202 
  [
AC_SEARCH_LIBS(bind_textdomain_codeset, intl, [],
   [AC_MSG_ERROR([a gettext implementation is required for NLS])])
+   AC_SEARCH_LIBS(ngettext, intl,
+  [AC_DEFINE(HAVE_NGETTEXT, 1,
+ [Define to 1 if you have the ngettext function.])],
+  [AC_MSG_WARN([NLS broken, plural forms support required for compile .po files])])
AC_CHECK_HEADER([libintl.h], [],
[AC_MSG_ERROR([header file libintl.h is required for NLS])])
AC_CHECK_PROGS(MSGFMT, msgfmt)
*** a/configure
--- b/configure
***
*** 26022,26027  echo $as_me: error: a gettext implementation is required for NLS 2;}
--- 26022,26117 
 { (exit 1); exit 1; }; }
  fi
  
+   { echo $as_me:$LINENO: checking for library containing ngettext 5
+ echo $ECHO_N checking for library containing ngettext... $ECHO_C 6; }
+ if test ${ac_cv_search_ngettext+set} = set; then
+   echo $ECHO_N (cached) $ECHO_C 6
+ else
+   ac_func_search_save_LIBS=$LIBS
+ cat conftest.$ac_ext _ACEOF
+ /* confdefs.h.  */
+ _ACEOF
+ cat confdefs.h conftest.$ac_ext
+ cat conftest.$ac_ext _ACEOF
+ /* end confdefs.h.  */
+ 
+ /* Override any GCC internal prototype to avoid an error.
+Use char because int might match the return type of a GCC
+builtin and then its argument prototype would still apply.  */
+ #ifdef __cplusplus
+ extern C
+ #endif
+ char ngettext ();
+ int
+ main ()
+ {
+ return ngettext ();
+   ;
+   return 0;
+ }
+ _ACEOF
+ for ac_lib in '' intl; do
+   if test -z $ac_lib; then
+ ac_res=none required
+   else
+ ac_res=-l$ac_lib
+ LIBS=-l$ac_lib  $ac_func_search_save_LIBS
+   fi
+   rm -f conftest.$ac_objext conftest$ac_exeext
+ if { (ac_try=$ac_link
+ case (($ac_try in
+   *\* | *\`* | *\\*) ac_try_echo=\$ac_try;;
+   *) ac_try_echo=$ac_try;;
+ esac
+ eval echo \\$as_me:$LINENO: $ac_try_echo\) 5
+   (eval $ac_link) 2conftest.er1
+   ac_status=$?
+   grep -v '^ *+' conftest.er1 conftest.err
+   rm -f conftest.er1
+   cat conftest.err 5
+   echo $as_me:$LINENO: \$? = $ac_status 5
+   (exit $ac_status); }  {
+ 	 test -z $ac_c_werror_flag ||
+ 	 test ! -s conftest.err
+}  test -s conftest$ac_exeext 
+$as_test_x conftest$ac_exeext; then
+   ac_cv_search_ngettext=$ac_res
+ else
+   echo $as_me: failed program was: 5
+ sed 's/^/| /' conftest.$ac_ext 5
+ 
+ 
+ fi
+ 
+ rm -f core conftest.err conftest.$ac_objext conftest_ipa8_conftest.oo \
+   conftest$ac_exeext
+   if test ${ac_cv_search_ngettext+set} = set; then
+   break
+ fi
+ done
+ if test ${ac_cv_search_ngettext+set} = set; then
+   :
+ else
+   ac_cv_search_ngettext=no
+ fi
+ rm conftest.$ac_ext
+ LIBS=$ac_func_search_save_LIBS
+ fi
+ { echo $as_me:$LINENO: result: $ac_cv_search_ngettext 5
+ echo ${ECHO_T}$ac_cv_search_ngettext 6; }
+ ac_res=$ac_cv_search_ngettext
+ if test $ac_res != no; then
+   test $ac_res = none required || LIBS=$ac_res $LIBS
+ 
+ cat confdefs.h \_ACEOF
+ #define HAVE_NGETTEXT 1
+ _ACEOF
+ 
+ else
+   { echo $as_me:$LINENO: WARNING: NLS broken, plural forms support required for compile .po files 5
+ echo $as_me: WARNING: NLS broken, plural forms support required for compile .po files 2;}
+ fi
+ 
if test ${ac_cv_header_libintl_h+set} = set; then
{ echo $as_me:$LINENO: checking for libintl.h 5
  echo $ECHO_N checking for libintl.h... $ECHO_C 6; }
*** a/src/bin/psql/print.c
--- b/src/bin/psql/print.c
***
*** 2348,2357  printQuery(const PGresult *result, const printQueryOpt *opt, FILE *fout, FILE *f
  		char		default_footer[100];
  
  		total_records = opt-topt.prior_records + cont.nrows;
! 		if (total_records == 1)
! 			snprintf(default_footer, 100, _((1 row)));
! 		else
! 			snprintf(default_footer, 100, _((%lu rows)), total_records);
  
  		printTableAddFooter(cont, default_footer);
  	}
--- 2348,2354 
  		char		default_footer[100];
  
  		total_records = opt-topt.prior_records + cont.nrows;
! 		snprintf(default_footer, 100, _P((%lu row), (%lu rows), total_records), total_records);
  
  		printTableAddFooter(cont, default_footer);
  	}
*** a/src/include/c.h
--- b/src/include/c.h
***
*** 91,102 
--- 91,108 
  

Re: [HACKERS] xpath processing brain dead

2009-03-20 Thread James Pye

On Mar 20, 2009, at 8:56 AM, Andrew Dunstan wrote:

Andrew Dunstan wrote:

  A / at the beginning of a path expression is an abbreviation for
  the initial step fn:root(self::node()) treat as document-node()/
  (however, if the / is the entire path expression, the trailing  
/

  is omitted from the expansion.) The effect of this initial step is
  to begin the path at the root node of the tree that contains the
  context node. If the context item is not a node, a type error is
  raised [err:XPTY0020]. At evaluation time, if the root node above
  the context node is not a document node, a dynamic error is raised
  [err:XPDY0050].

The problem is that we certainly do have to provide a context node  
(the standard is clear about that), and unless we want to convert a  
non-document to a node-set as James suggested and then apply the  
xpath expression to each node in the node-set, we have no way of  
sanely specifying the context node.


libxml2 only supports xpath1. Why are you referencing xpath20? And, if  
SQL-XML requires an xpath20 conforming xpath() function, we have  
bigger problems than '/x' + xpath_str. ;)


Although, this is not terribly important to me as:


No-one has come up with an answer to this,


I don't feel fragment()/node-set() is a good idea from a usability  
standpoint alone. I only mentioned it because that's how I've always  
worked with fragments in the xslt1 world.. Mere curiosity drove most  
of the remaining interest I had in it.



so I propose to remove the hackery.


I think this would probably be best for the core xpath() function.

However, it may be wise to relocate the munging functionality into  
another function so users don't have invent their own when they feel  
so inclined to work with fragments.



raise an error?


+1, xpath requires a well-formed document

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] gettext, plural form and translation

2009-03-20 Thread Sergey Burladyan
Sergey Burladyan eshkin...@gmail.com writes:

  gettext-plural-ru-test.patch:
  - correct translation for 1 rows message

hmmm... encoding is broken... i post it again in gzip




gettext-plural-ru-test.patch.gz
Description: gettext-plural-ru-test.patch


-- 
Sergey Burladyan

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets

2009-03-20 Thread Tom Lane
Bryce Cutt pandas...@gmail.com writes:
 Here is the new patch.

Applied with revisions.  I undid some of the optimizations that
cluttered the code in order to save a cycle or two per tuple --- as per
previous discussion, that's not what the performance questions were
about.  Also, I did not like the terminology in-memory/IM; it seemed
confusing since the main hash table is in-memory too.  I revised the
code to consistently refer to the additional hash table as a skew
hashtable and the optimization in general as skew optimization.  Hope
that seems reasonable to you --- we could search-and-replace it to
something else if you'd prefer.

For the moment, I didn't really do anything about teaching the planner
to account for this optimization in its cost estimates.  The initial
estimate of the number of MCVs that will be specially treated seems to
me to be too high (it's only accurate if the inner relation is unique),
but getting a more accurate estimate seems pretty hard, and it's not
clear it's worth the trouble.  Without that, though, you can't tell
what fraction of outer tuples will get the short-circuit treatment.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] mbox-to-html script with stable identifiers

2009-03-20 Thread Robert Haas
Here's an attempt.  Let me know what you think.  I had thought to
maybe just write a script to generate Mediawiki output and then post
it on the wiki where anyone could edit it.  But that's not going to
work if you're going to continue adding to this, or at least not
without more thought about how to manage the update stream.

...Robert


bruce-filter.pl
Description: Perl program

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 8:14 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 Bryce Cutt pandas...@gmail.com writes:
 Here is the new patch.

 Applied with revisions.  I undid some of the optimizations that
 cluttered the code in order to save a cycle or two per tuple --- as per
 previous discussion, that's not what the performance questions were
 about.  Also, I did not like the terminology in-memory/IM; it seemed
 confusing since the main hash table is in-memory too.  I revised the
 code to consistently refer to the additional hash table as a skew
 hashtable and the optimization in general as skew optimization.  Hope
 that seems reasonable to you --- we could search-and-replace it to
 something else if you'd prefer.

 For the moment, I didn't really do anything about teaching the planner
 to account for this optimization in its cost estimates.  The initial
 estimate of the number of MCVs that will be specially treated seems to
 me to be too high (it's only accurate if the inner relation is unique),
 but getting a more accurate estimate seems pretty hard, and it's not
 clear it's worth the trouble.  Without that, though, you can't tell
 what fraction of outer tuples will get the short-circuit treatment.

If the inner relation isn't fairly close to unique you shouldn't be
using this optimization in the first place.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 8:45 PM, Bryce Cutt pandas...@gmail.com wrote:
 On Fri, Mar 20, 2009 at 5:35 PM, Robert Haas robertmh...@gmail.com wrote:
 If the inner relation isn't fairly close to unique you shouldn't be
 using this optimization in the first place.
 Not necessarily true.  Seeing as (when the statistics are correct) we
 know each of these inner tuples will match with the largest amount of
 outer tuples it is just as much of a win per inner tuple as when they
 are unique.  There is just a chance you will have to give up on the
 optimization part way through if too many inner tuples fall into the
 new skew buckets (formerly IM buckets) and dump the tuples back into
 the main buckets.  The potential win is still pretty high though.

 - Bryce Cutt

Maybe I'm remembering wrong, but I thought the estimating functions
assuemd that the inner relation was unique.  So if there turn out to
be 2, 3, 4, or more copies of each value, the chances of blowing out
the skew hash table are almost 100%, I would think...  am I wrong?

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 8:45 PM, Bryce Cutt pandas...@gmail.com wrote:
 On Fri, Mar 20, 2009 at 5:35 PM, Robert Haas robertmh...@gmail.com wrote:
 If the inner relation isn't fairly close to unique you shouldn't be
 using this optimization in the first place.
 Not necessarily true.  Seeing as (when the statistics are correct) we
 know each of these inner tuples will match with the largest amount of
 outer tuples it is just as much of a win per inner tuple as when they
 are unique.  There is just a chance you will have to give up on the
 optimization part way through if too many inner tuples fall into the
 new skew buckets (formerly IM buckets) and dump the tuples back into
 the main buckets.  The potential win is still pretty high though.

 - Bryce Cutt

Maybe I'm remembering wrong, but I thought the estimating functions
assuemd that the inner relation was unique.  So if there turn out to
be 2, 3, 4, or more copies of each value, the chances of blowing out
the skew hash table are almost 100%, I would think...  am I wrong?

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets

2009-03-20 Thread Bryce Cutt
Not necessarily true.  Seeing as (when the statistics are correct) we
know each of these inner tuples will match with the largest amount of
outer tuples it is just as much of a win per inner tuple as when they
are unique.  There is just a chance you will have to give up on the
optimization part way through if too many inner tuples fall into the
new skew buckets (formerly IM buckets) and dump the tuples back into
the main buckets.  The potential win is still pretty high though.

- Bryce Cutt


On Fri, Mar 20, 2009 at 5:35 PM, Robert Haas robertmh...@gmail.com wrote:
 On Fri, Mar 20, 2009 at 8:14 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 Bryce Cutt pandas...@gmail.com writes:
 Here is the new patch.

 Applied with revisions.  I undid some of the optimizations that
 cluttered the code in order to save a cycle or two per tuple --- as per
 previous discussion, that's not what the performance questions were
 about.  Also, I did not like the terminology in-memory/IM; it seemed
 confusing since the main hash table is in-memory too.  I revised the
 code to consistently refer to the additional hash table as a skew
 hashtable and the optimization in general as skew optimization.  Hope
 that seems reasonable to you --- we could search-and-replace it to
 something else if you'd prefer.

 For the moment, I didn't really do anything about teaching the planner
 to account for this optimization in its cost estimates.  The initial
 estimate of the number of MCVs that will be specially treated seems to
 me to be too high (it's only accurate if the inner relation is unique),
 but getting a more accurate estimate seems pretty hard, and it's not
 clear it's worth the trouble.  Without that, though, you can't tell
 what fraction of outer tuples will get the short-circuit treatment.

 If the inner relation isn't fairly close to unique you shouldn't be
 using this optimization in the first place.

 ...Robert


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] contrib function naming, and upgrade issues

2009-03-20 Thread Andrew Gierth
Note that I'm talking here about the names of the C functions, not
the SQL names.

The existing hstore has some very dubious choices of function names
(for non-static functions) in the C code; functions like each(),
delete(), fetchval(), defined(), tconvert(), etc. which all look to me
like prime candidates for name collisions and consequent hilarity.

The patch I'm working on could include fixes for this; but there's an
obvious impact on anyone upgrading from an earlier version... is it
worth it?

-- 
Andrew (irc:RhodiumToad)

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets

2009-03-20 Thread Bryce Cutt
The estimation functions assume the inner relation join column is
unique.  But it freezes (flushes back to the main hash table) one skew
bucket at a time in order of least importance so if 100 inner tuples
can fit in the skew buckets then the skew buckets are only fully blown
out if the best tuple (the single most common value) occurs more than
100 times in the inner relation.  And up until that point you still
have the tuples in memory that are the best per tuple worth of
memory.  But yes, after that point (more than 100 tuples of that best
MCV) the entire effort was wasted.  The skew buckets are dynamically
flushed just like buckets in a dynamic hash join would be.

- Bryce Cutt

On Fri, Mar 20, 2009 at 5:51 PM, Robert Haas robertmh...@gmail.com wrote:
 On Fri, Mar 20, 2009 at 8:45 PM, Bryce Cutt pandas...@gmail.com wrote:
 On Fri, Mar 20, 2009 at 5:35 PM, Robert Haas robertmh...@gmail.com wrote:
 If the inner relation isn't fairly close to unique you shouldn't be
 using this optimization in the first place.
 Not necessarily true.  Seeing as (when the statistics are correct) we
 know each of these inner tuples will match with the largest amount of
 outer tuples it is just as much of a win per inner tuple as when they
 are unique.  There is just a chance you will have to give up on the
 optimization part way through if too many inner tuples fall into the
 new skew buckets (formerly IM buckets) and dump the tuples back into
 the main buckets.  The potential win is still pretty high though.

 - Bryce Cutt

 Maybe I'm remembering wrong, but I thought the estimating functions
 assuemd that the inner relation was unique.  So if there turn out to
 be 2, 3, 4, or more copies of each value, the chances of blowing out
 the skew hash table are almost 100%, I would think...  am I wrong?

 ...Robert


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] contrib function naming, and upgrade issues

2009-03-20 Thread Robert Haas
On Fri, Mar 20, 2009 at 9:57 PM, Andrew Gierth
and...@tao11.riddles.org.uk wrote:
 Note that I'm talking here about the names of the C functions, not
 the SQL names.

 The existing hstore has some very dubious choices of function names
 (for non-static functions) in the C code; functions like each(),
 delete(), fetchval(), defined(), tconvert(), etc. which all look to me
 like prime candidates for name collisions and consequent hilarity.

 The patch I'm working on could include fixes for this; but there's an
 obvious impact on anyone upgrading from an earlier version... is it
 worth it?

Based on that description, +1 from me.  That kind of hilarity can be a
huge time sink when debugging, and it makes it hard to use grep to
find all references to a particular function (or #define, typedef,
etc.).

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Bruce Momjian
Alvaro Herrera wrote:
 Bruce Momjian escribi?:
  Alvaro Herrera wrote:
 
   Note that during the 8.4 timeframe we've stolen a lot of work from
   Bruce.  The TODO list was moved to the wiki, for one; the patch queue
   was also moved to the wiki.  Now the FAQ has moved to wiki (and has
   already seen lots of improvement, so it was clearly a good move).
   Previously this was all handled as email boxes, so while some
   inefficiences in the process still remain, we're a lot better than we
   were in the 8.3 cycle.
  
  Please take as much work from me as you can.  However, looking at the
  TODO commits, I am still the one doing most of the changes:
  
  http://wiki.postgresql.org/index.php?title=Todoaction=history
 
 Yes, but there are commits from others too.  I think the TODO is seen as
 a delicate document and other people is reluctant to edit it.

Agreed.  My only point is that many thought that putting the TODO in a
wiki would significantly reduce my TODO maintenance effort, which it has
not; which is fine, and I expected this outcome, but others did not.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] small but useful patches for text search

2009-03-20 Thread Bruce Momjian
Robert Haas wrote:
 I personally think that the way pgsql-hackers organizes itself using
 email is completely insane.  The only reason that you need to write
 the release notes instead of, say, me, is because the only information
 on what needs to go into them is buried in a thicket of CVS commit
 messages that I am not nearly brave enough to attempt to penetrate.  I
 suggested putting them in CVS yesterday; Tom didn't like that, but
 what about a wiki page or a database?  grep 'release notes'
 /last/six/months/of/email can't possibly be the best way to do this.
 Given any sort of list to work from, even one that is totally
 disorganized and written in broken English, I can't believe this is
 more than an hour or two of work, and I'd be more than happy to take a
 crack at it (I'm probably not the only one, either).

Let me just add, as a way of macro-understanding our approach to
development, that the Postgres community has always been set up to get
the maximum feedback from the community, even if it sometimes increases
the work required by a few core folks to keep things going.

So some of the things we do that seem inefficient are done because many
feel a more structured approach would limit our ability to harness the
strengths of our community.  For example, moving to a bug tracking
system would make some things much easier, but would probably dampen our
momentum.  Handling discussions via web forums instead of email would
probably have the same effect.  Of course, I might be wrong, but that is
what many in the community think.

I think the example of moving the TODO list to a wiki, that was supposed
to relive a lot of the burden I carry to maintain the TODO list, has
really not affected my workload much, which kind of reinforces the
feeling that our existing setup is probably the best we are going to do.
Of course, the commit fest wikis have helped, so I guess there is room
for improvement in some places.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] contrib function naming, and upgrade issues

2009-03-20 Thread Tom Lane
Andrew Gierth and...@tao11.riddles.org.uk writes:
 Note that I'm talking here about the names of the C functions, not
 the SQL names.

 The existing hstore has some very dubious choices of function names
 (for non-static functions) in the C code; functions like each(),
 delete(), fetchval(), defined(), tconvert(), etc. which all look to me
 like prime candidates for name collisions and consequent hilarity.

 The patch I'm working on could include fixes for this; but there's an
 obvious impact on anyone upgrading from an earlier version... is it
 worth it?

I agree that this wasn't an amazingly good choice, but I think there's
no real risk of name collisions because fmgr only searches for such names
within the particular .so.  As you say, renaming *will* break existing
dumps.  I'd be inclined to leave it alone, at least for now.  I hope
that someone will step up and implement a decent module system for us
sometime soon, which might fix the upgrade problem for changes of this
sort.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers