[HACKERS] Is LISTEN/NOTIFY reliable?

[Tatsuo Ishii]

Hi,

Does anybody know PostgreSQL LISTEN/NOTIFY is more reliable than
previous versions?  I vaguely recall that in the previous
implementation, message sent by NOTIFY may not be reached to listner.
Does PostgreSQL 9.0's new implementation guarantee that the message is
received by the listener?
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Extensions, this time with a patch

[Dimitri Fontaine]

Alvaro Herrera  writes:
> Maybe what you should be doing here is that modules should provide
> another definition, say EXTENSION, and they have to explicitely define
> it in their Makefile (maybe require EXTENSION_VERSION too or something
> like that).  I think the idea that modules should continue to work as
> extensions without any modification is doomed.

In fact there's ifndef CONTROL that protects the black magic failing
part, so that we could edit any contrib's Makefile to give the
information we're trying to guess. I just had another try at it that
seems to work much better, based on DATA and DATA_built:

# create extension support
ifndef CONTROL
ifdef DATA_built
EXTENSION = $(basename $(notdir $(firstword $(DATA_built
else ifdef DATA
EXTENSION = $(basename $(notdir $(firstword $(DATA
endif
ifdef EXTENSION
CONTROL = $(EXTENSION).control
endif
endif

Also, I've switched to using echo twice as you recommended, that's much
better too.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Extensions, this time with a patch

[Dimitri Fontaine]

Alvaro Herrera  writes:
> Hmm.  To be honest I don't like the direction that pg_execute_from_file
> has taken.  (Now that I look, it's been like this since inception).  I
> have two problems with it: one is that it is #including half the world
> into genfile.c.  This already smells trouble in itself.  I got worried
> when I saw the CommandDest declaration.  Really, I think having the guts
> of postgres.c into that file is not a good idea from a modularisation
> point of view.

Understood. The thinking back when I worked on that part was to minimize
the diff and remain localized, which is known to be a bad idea... I'll
rework that part as soon as we agree on the other one:

> The other problem is that it's slurping the whole file and executing it
> as a single query.  This is really two problems: one is that you
> shouldn't be trusting that the file is going to be small enough to be
> read that way.  The other one is that I don't think it's a good idea to
> execute it in a fell swoop; seems to be it would be better to split it
> into queries, and rewrite/parse/plan/execute them one by one.
>
> I think a better way to go about this is to have another entry point in
> postgres.c that executes a query the way you want; and somehow read the
> file in small chunks, find where each query ends, and execute them one
> by one.  (To be honest, I have no idea how to find out where each query
> ends.  psql knows how to do it, but I'm not sure how trustworthy it
> is.)

Well, that's the reason why it's done this way now, relying on a multiple
queries portal. The only trustworthy way to split the queries apart in
the SQL install script would be to rely on gram.y, and I didn't find the
API to explicitly loop over each query parsed.

Given some advice, I'll rework that part too. The good news is that it's
well separated from the rest of the extension's work.

We need a way to do the same as \i in psql, but from the backend, and we
won't be returning anything from there, so we don't need to handle more
than one portal definition in a single fe/be communication.

> As far as #include lines go, please keep them in alphabetical order.  As
> a matter of style we always have "postgres.h" alone, then a blank line,
> then system includes, another blank, then the rest of the includes.

Will do.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Trailing Whitespace Tips (was: [HACKERS] Re: starting to review the Extend NOT NULL representation to pg_constraint patch)

[Peter Eisentraut]

On fre, 2010-10-15 at 22:45 +0200, Dimitri Fontaine wrote:
> I suppose people using other editors or tools will come up with other
> tricks and tips.

Here is an alternative recipe that I have been using:

(require 'show-wspace)
(add-hook 'font-lock-mode-hook 'show-ws-highlight-hard-spaces)
(add-hook 'font-lock-mode-hook 'show-ws-highlight-tabs)
(add-hook 'font-lock-mode-hook 'show-ws-highlight-trailing-whitespace)

> Maybe it should go in src/tools/editors/emacs.samples, too?

Yeah, I think we should recommend some way to highlight faulty
whitespace.

The problem is, after you turn it on, it will make you cry as you
realize how sloppy most code and other files are written.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Timeline in the light of Synchronous replication

[fazool mein]

Hello guys,

The concept of time line makes sense to me in the case of asynchronous
replication. But in case of synchronous replication, I am not so sure.

When a standby connects to the primary, it checks if both have the same time
line. If not, it doesn't start.

Now, consider the following scenario. The primary (call it A) fails, the
standby (call it B), via a trigger file, comes out of recovery mode
(increments time line id to say 2), and morphs into a primary. Now, lets say
we start the old primary A as a standby, to connect to the new primary B
(which previously was a standby). As the code is at the moment, the old
primary A will not be allowed to connect to the new primary B because A's
timelineid (1) is not equivalent to that of the new primary B (2). Hence, we
need to create a backup again, and setup the standby from scratch.

In the above scenario, if the system was using asynchronous replication,
time lines would have saved us from doing something wrong. But, if we are
using synchronous replication, we know that both A and B would have been in
sync before the failure. In this case, forcing to create a new standby from
scratch because of time lines might not be very desirable if the database is
huge.

Your comments on the above will be appreciated.

Regards

Re: [HACKERS] Why do we have a database specification in .pgpass?

[Peter Eisentraut]

On ons, 2010-10-13 at 14:32 -0400, Bruce Momjian wrote:
> We have a database specification in .pgpass:
> 
> hostname:port:database:username:password
> 
> What is the purpose of 'database' since username/password combinations
> are global, not per database?  I would like to documents its purpose.

As a side note, the thing at the other end of a connection is not
necessarily a PostgreSQL server.  It could be a connection pool proxy.
I don't know if any implementatation could make use of the database
field at the moment, but it should be kept in mind.

That said, it would probably be good to document that the database field
is currently only useful in certain limited circumstances.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] all keyword for pg_hba.conf host column

[Peter Eisentraut]

As a small addition to the pg_hba.conf host name feature, I figured it
would be useful to allow "all" in the host column, instead of having to
write 0.0.0.0/0 and ::/0.  Patch attached.
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index ab96af8..228cfff 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -257,7 +257,7 @@ hostnossl  database  user
   
 
   
-   You can also write
+   You can also write all to match any IP address,
samehost to match any of the server's own IP
addresses, or samenet to match any address in any
subnet that the server is directly connected to.
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 3f50349..38eaa95 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -879,8 +879,11 @@ parse_hba_line(List *line, int line_num, HbaLine *parsedline)
 		}
 		token = lfirst(line_item);
 
-		/* Is it equal to 'samehost' or 'samenet'? */
-		if (strcmp(token, "samehost\n") == 0)
+		if (strcmp(token, "all\n") == 0)
+		{
+			parsedline->ip_cmp_method = ipCmpAll;
+		}
+		else if (strcmp(token, "samehost\n") == 0)
 		{
 			/* Any IP on this host is allowed to connect */
 			parsedline->ip_cmp_method = ipCmpSameHost;
@@ -1497,6 +1500,8 @@ check_hba(hbaPort *port)
 			continue;
 	}
 	break;
+case ipCmpAll:
+	break;
 case ipCmpSameHost:
 case ipCmpSameNet:
 	if (!check_same_host_or_net(&port->raddr,
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index eb6637f..aa60d8d 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -36,7 +36,8 @@ typedef enum IPCompareMethod
 {
 	ipCmpMask,
 	ipCmpSameHost,
-	ipCmpSameNet
+	ipCmpSameNet,
+	ipCmpAll
 } IPCompareMethod;
 
 typedef enum ConnType

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] knngist - 0.8

[Martijn van Oosterhout]

On Fri, Oct 15, 2010 at 08:45:26PM -0400, Robert Haas wrote:
> But I'm also not sure how far this gets us with KNNGIST, where the
> issue is not the typmods but the auxilliary information about the
> context of the sort and/or whether this is a sort or qual.

ISTM there are two issues here. With Btree you have the issue where the
index is of a particular collation and at planning time you're given a
collation and either it's compatable or it's not.

With Gist you have the situation where index isn't of a specific
collation but it can be used in different ways to get different
collection as output. (I'm here thinking of that Gist can in some
circumstance be asked to return tuples in a certain order.)

Actually, NULLS FIRST/LAST, ASC/DESC are just variations on collations
and my original patch many years back for each locale made *four*
collation IDs, one for each of the combinations of the above. Not
terribly scalable, but I never did get the planning code to work. :)

What you're going for here I think is a sort of cross-collation
operators, or something, that tells the planner how to get the
particular collation out of the given index. So you have something
like:

Input: en_US, NULLS FIRST, ASC
Output: en_US, NULLS_LAST, DESC
-> Reverse index scan

Input: en_US, NULLS FIRST, ASC
Output: en_US, NULLS LAST, ASC
-> Index scan, add x IS NOT NULL test
-> Append output of x IS NULL.

Input: Gist
Output: KnnGist
-> Index scan with scan type 1

Input: Gist
Output: Gist
-> Index scan with scan type 0

Obviously this doesn't solve the problem of being able to represent the
required collation in the first place, and if this is at all compatable
with what the SQL standard calls a collation. I just don't think you
can make hard and fast rules about how to make this work and that
perhaps we should be looking for a way to push that to the index
implementation code, with the default rule being: same collection yes,
different no.

Just some thoughts,

Have a nice day,
-- 
Martijn van Oosterhout  http://svana.org/kleptog/
> Patriotism is when love of your own people comes first; nationalism,
> when hate for people other than your own comes first. 
>   - Charles de Gaulle


signature.asc
Description: Digital signature

Re: [HACKERS] Timeout and wait-forever in sync rep

[Robert Haas]

On Fri, Oct 15, 2010 at 8:41 AM, Fujii Masao  wrote:
> Hi,
>
> As the result of the discussion, I think that we need the following two
> parameters for the case where the standby goes down.
>
> * replication_timeout
>  This is the maximum time to wait for the ACK from the standby. If this
>  timeout expires, the master closes the replication connection and
>  disconnects the standby. This parameter is just used for the master
>  to detect the standby crash or the network outage.
>
>  We already have keepalive parameters for that purpose. But they cannot
>  detect the disconnection in some cases. So replication_timeout needs
>  to be introduced for sync rep.

Good design, +1.

> * allow_standalone_master
>  This specifies whether we allow the master to process transactions
>  alone when there is no connected and sync'd standby.
>
>  If this is false, all the transactions on the master are blocked until
>  sync'd standby has appeared. Of course, this happen not only when
>  replication_timeout expires but also when we start the master alone
>  at the initial setup, when the master detects the disconnection by
>  using keepalive parameters, and when the standby is shut down normally.
>  People who want 'wait-forever' will disable this parameter to reduce
>  the risk of data loss.
>
>  OTOH, if this is true, the absence of sync'd standby doesn't prevent
>  the master from processing transactions alone. People who want high
>  availability even though the risk of data loss increases will enable
>  this parameter.

I'm not wild about the name, but otherwise this seems well-designed.

> The timeout doesn't oppose to 'wait-forever'. Even if you choose 'wait
> -forever' (i.e., you set allow_standalone_master to false), the master
> should detect the standby crash as soon as possible by using the
> timeout. For example, imagine that max_wal_senders is set to one and
> the master cannot detect the standby crash because of absence of the
> timeout. In this case, even if you start new standby, it will not be
> able to connect to the master since there is no free walsender slot.
> As the result, the master actually waits forever.

Good point.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Is LISTEN/NOTIFY reliable?

[Robert Haas]

On Sat, Oct 16, 2010 at 3:27 AM, Tatsuo Ishii  wrote:
> Does anybody know PostgreSQL LISTEN/NOTIFY is more reliable than
> previous versions?  I vaguely recall that in the previous
> implementation, message sent by NOTIFY may not be reached to listner.
> Does PostgreSQL 9.0's new implementation guarantee that the message is
> received by the listener?

I think it was always intended to be reliable (otherwise it's not much
good).  I think I remember a bug where notifications were being lost
on Windows under heavy load, but I thought we fixed that...

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Is LISTEN/NOTIFY reliable?

[Tom Lane]

Tatsuo Ishii  writes:
> Does anybody know PostgreSQL LISTEN/NOTIFY is more reliable than
> previous versions?  I vaguely recall that in the previous
> implementation, message sent by NOTIFY may not be reached to listner.
> Does PostgreSQL 9.0's new implementation guarantee that the message is
> received by the listener?

No more or less than the old one did.

The old design intentionally dropped duplicate notifications, and the
new one does too (though with a narrower definition of "duplicate").
Modulo that well-documented behavior, I know of no reason to describe
either the old or new code as unreliable.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Extensions, this time with a patch

[Tom Lane]

Dimitri Fontaine  writes:
> Alvaro Herrera  writes:
>> The other problem is that it's slurping the whole file and executing it
>> as a single query.

> Given some advice, I'll rework that part too. The good news is that it's
> well separated from the rest of the extension's work.

I think that's something that could be left for later, if not never.
I find it hard to imagine extension modules with more than a few
thousand commands (the largest one in contrib is isn, with about 500).
No modern machine is going to have the slightest difficulty with that.
If it ever does get to be a problem in practice, we could rework the
implementation at that time.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Is LISTEN/NOTIFY reliable?

[Andrew Dunstan]

On 10/16/2010 09:04 AM, Robert Haas wrote:

On Sat, Oct 16, 2010 at 3:27 AM, Tatsuo Ishii  wrote:

Does anybody know PostgreSQL LISTEN/NOTIFY is more reliable than
previous versions?  I vaguely recall that in the previous
implementation, message sent by NOTIFY may not be reached to listner.
Does PostgreSQL 9.0's new implementation guarantee that the message is
received by the listener?

I think it was always intended to be reliable (otherwise it's not much
good).  I think I remember a bug where notifications were being lost
on Windows under heavy load, but I thought we fixed that...


Here's what the docs say:

   If the same channel name is signaled multiple times from the same
   transaction with identical payload strings, the database server can
   decide to deliver a single notification only. On the other hand,
   notifications with distinct payload strings will always be delivered
   as distinct notifications. Similarly, notifications from different
   transactions will never get folded into one notification. Except for
   dropping later instances of duplicate notifications, NOTIFY
   guarantees that notifications from the same transaction get
   delivered in the order they were sent.


cheers

andrew

Re: [HACKERS] knngist plans

[Marios Vodas]

Oleg as I said I do believe some things could get better in docs.
Here are 4 main drawbacks in my opinion:
1. The docs lack example(s). There are some references to gistproc.c etc but
this is not part of the docs and if I am someone that has just started this
is hard and confusing (I shouldn't have to read the full technical solution
to another problem in order to solve mine).
2. The 7 methods are poorly explained, at least theoretically. For example
when I started I couldn't understand where each function was used. I had to
read papers to understand the logic.
3. Describe situations that gist can be used. I am aware of spatial indexing
because I worked on it, but I have the impression that there are more (e.g.
computational biology).
4. Why not add some images/figures? (1 image = 1000 words)

GiST is a very important and useful part of postgresql but it's hard to get
started with it. Moreover, I am sure it has capabilities, other than the
profound, that are not known to many people (including me).
Plus knngist will give gist a boost, and I suggest that it shouldn't be
delayed any more.

We have some documentation already, Teodor will post soon.

I would really like to see what Teodor has done.

I could also prepare a document with the additions I would like to see in
docs. Shall I do it? Of course it might take a while to finish it.

On Fri, Oct 15, 2010 at 2:37 PM, Oleg Bartunov  wrote:

> On Fri, 15 Oct 2010, Marios Vodas wrote:
>
>  Recently I worked a lot with gist and read about knngist. I have spotted
>> weaknesses in docs. So I would be happy to help if you tell me how.
>>
>
> We have some documentation already, Teodor will post soon. Marios, I'd
> happy to help in writing docs. Do you have some plans ?
>
>
>
>> On Fri, Oct 15, 2010 at 5:57 AM, Robert Haas 
>> wrote:
>>
>>  On Wed, Oct 13, 2010 at 7:07 PM, Marios Vodas  wrote:
>>>
 I would like to ask in which future version of postgresql knngist is

>>> planned
>>>
 to be included. Is there any possibility to be included in 9.1?

>>>
>>> There's a possibility, but I think the patch still needs some more
>>> work.  One thing that would help is if someone felt motivated to write
>>> some documentation.
>>>
>>> --
>>> Robert Haas
>>> EnterpriseDB: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>>
>>
>Regards,
>Oleg
> _
> Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
> Sternberg Astronomical Institute, Moscow University, Russia
> Internet: o...@sai.msu.su, 
> http://www.sai.msu.su/~megera/
> phone: +007(495)939-16-83, +007(495)939-23-83
>

Re: [HACKERS] Keywords in pg_hba.conf should be field-specific

[Brendan Jurd]

On 13 October 2010 00:28, Fujii Masao  wrote:
> On Sat, Oct 9, 2010 at 11:13 AM, Brendan Jurd  wrote:
>> I understand that 'replication' is a keyword as far as the database
>> name is concerned, but I was surprised to find that it was treated as
>> a keyword in the username field also.  I had a look in
>> src/backend/libpq/hba.c, and next_token() appears to be completely
>> naive about this.  'replication' (along with 'all', 'sameuser',
>> 'samegroup' and 'samerole')  is treated as a keyword wherever it
>> appears, not just in the field where it is relevant.  next_token()
>> appends a newline to the end of the 'replication' username token, and
>> that's why the entry doesn't match my connection attempt.
>>
>> I suspect this is going to trip a lot of people up.  We could just
>> document it and tell people that if they want to use 'replication' as
>> a username, they'd better quote it in pg_hba.conf.  But I'd prefer to
>> actually solve the problem.
>
> Agreed. We should address that.
>

Hi folks,

Per the above discussion, I've prepared a patch to make keywords in
pg_hba.conf field-specific.

This patch takes a least-resistance approach to solving the problem.
next_token() continues to blithely append a newline character to any
token which *might* be a keyword, but the patch teaches
hba_parse_line() to be a bit more savvy about whether to take
next_token's word for it.

In the Database field, hba_parse_line() acknowledges 'all',
'sameuser', 'samegroup', 'samerole' and 'replication' as legitimate
keywords.  Anything else is considered not a keyword, and if there is
a trailing newline it is stripped from the token (by just reassigning
it to NUL).

Likewise, in the Role field, only 'all' is considered legitimate.

In the Host field, we allow 'samehost' and 'samenet'.

The bottom line is that this makes 'replication' safe to use as a
username, no quoting required.

It seemed a little bit inelegant to allow next_token() to mark up a
keyword, only to ignore that marking later on, but as the number of
fields in each line is not decided until we parse it, there was no
sensible way to teach next_token() which field it was evaluating.

Added to the November CF.

Cheers,
BJ

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Keywords in pg_hba.conf should be field-specific

[Brendan Jurd]

On 17 October 2010 01:52, Brendan Jurd  wrote:
> Per the above discussion, I've prepared a patch to make keywords in
> pg_hba.conf field-specific.
>

Try New and Improved This Message (tm), now with attachment!

Cheers,
BJ
*** a/src/backend/libpq/hba.c
--- b/src/backend/libpq/hba.c
***
*** 855,861  parse_hba_line(List *line, int line_num, HbaLine *parsedline)
line_num, 
HbaFileName)));
return false;
}
!   parsedline->database = pstrdup(lfirst(line_item));
  
/* Get the role. */
line_item = lnext(line_item);
--- 855,873 
line_num, 
HbaFileName)));
return false;
}
!   token = pstrdup(lfirst(line_item));
! 
!   /* Filter out inapplicable keywords for the database field. */
!   if (token[strlen(token) - 1] == '\n'
!   && strcmp(token, "all\n") != 0
!   && strcmp(token, "sameuser\n") != 0
!   && strcmp(token, "samerole\n") != 0
!   && strcmp(token, "samegroup\n") != 0
!   && strcmp(token, "replication\n") != 0)
!   {
!   token[strlen(token) - 1] = '\0';
!   }
!   parsedline->database = token;
  
/* Get the role. */
line_item = lnext(line_item);
***
*** 868,874  parse_hba_line(List *line, int line_num, HbaLine *parsedline)
line_num, 
HbaFileName)));
return false;
}
!   parsedline->role = pstrdup(lfirst(line_item));
  
if (parsedline->conntype != ctLocal)
{
--- 880,894 
line_num, 
HbaFileName)));
return false;
}
!   token = pstrdup(lfirst(line_item));
! 
!   /* Filter out inapplicable keywords for the role field. */
!   if (token[strlen(token) - 1] == '\n'
!   && strcmp(token, "all\n") != 0)
!   {
!   token[strlen(token) - 1] = '\0';
!   }
!   parsedline->role = token;
  
if (parsedline->conntype != ctLocal)
{
***
*** 904,909  parse_hba_line(List *line, int line_num, HbaLine *parsedline)
--- 924,937 
/* need a modifiable copy of token */
token = pstrdup(token);
  
+   /*
+* Filter out any remaining keywords, as the only valid 
keywords
+* for this context ('samehost' and 'samenet') have 
already been
+* handled above.
+*/
+   if (token[strlen(token) - 1] = '\n')
+   token[strlen(token) - 1] = '\0';
+ 
/* Check if it has a CIDR suffix and if so isolate it */
cidr_slash = strchr(token, '/');
if (cidr_slash)

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Extensions, this time with a patch

[Dimitri Fontaine]

Tom Lane  writes:
> I think that's something that could be left for later, if not never.

That's very great news. I'm left with moving the bulk of the code away
from genfile.c and into postgres.c, and have the former be a user
callable shell around the later, I suppose. Right?

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] all keyword for pg_hba.conf host column

[Robert Haas]

On Oct 16, 2010, at 6:56 AM, Peter Eisentraut  wrote:
> As a small addition to the pg_hba.conf host name feature, I figured it
> would be useful to allow "all" in the host column, instead of having to
> write 0.0.0.0/0 and ::/0.  Patch attached.

+1. Looks sane on a quick read.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] all keyword for pg_hba.conf host column

[Brendan Jurd]

On 16 October 2010 21:56, Peter Eisentraut  wrote:
> As a small addition to the pg_hba.conf host name feature, I figured it
> would be useful to allow "all" in the host column, instead of having to
> write 0.0.0.0/0 and ::/0.  Patch attached.
>

Cool.  And, for what it's worth, this doesn't conflict at all with the
field-specific keywords patch I just submitted.

Cheers,
BJ

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] knngist plans

[Alvaro Herrera]

Excerpts from Marios Vodas's message of sáb oct 16 11:11:52 -0300 2010:

> 4. Why not add some images/figures? (1 image = 1000 words)

I can answer this: the reason is that we haven't settled on a
technology for producing figures.  There's a thread about it, but no
consensus.

-- 
Álvaro Herrera 
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Keywords in pg_hba.conf should be field-specific

[Alvaro Herrera]

Excerpts from Brendan Jurd's message of sáb oct 16 11:53:31 -0300 2010:
> On 17 October 2010 01:52, Brendan Jurd  wrote:
> > Per the above discussion, I've prepared a patch to make keywords in
> > pg_hba.conf field-specific.
> 
> Try New and Improved This Message (tm), now with attachment!

Hmm.  Would it be possible to list keywords _applicable_ to each field,
and have these passed down to next_token by the caller instead?  This
seems backwards, but I'm not sure if the other way is really workable.

-- 
Álvaro Herrera 
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Keywords in pg_hba.conf should be field-specific

[Brendan Jurd]

On 17 October 2010 02:27, Alvaro Herrera  wrote:
> Hmm.  Would it be possible to list keywords _applicable_ to each field,
> and have these passed down to next_token by the caller instead?  This
> seems backwards, but I'm not sure if the other way is really workable.
>

Short answer: I don't think it is workable, or I would have done it
that way in the first place.

Full answer: The problem is that pg_hba.conf doesn't have a fixed
structure.  Each line can be 4, 5 or 6 fields (not including the final
'options' field) long, and which of these structures apply to any
given line isn't decided until parse_hba_line goes to work on it.

At the time that next_token gets called, we have no way of knowing
which field is currently being tokenised, at least not without doing
some serious rearrangement of hba.c, so that it tokenises and then
parses one token at a time, instead of tokenising the whole file and
then parsing all the lines.

Cheers,
BJ

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] knngist - 0.8

[Paul Ramsey]

>>   (And, if we are going to break everything
> in sight, now would be a good time to think about changing typmod to
> something more flexible than one int32.)

As someone who is jamming geometry type, spatial reference number and 
dimensionality into said 32bit typmod, let me say emphatically ... Amen!

P

Re: Trailing Whitespace Tips (was: [HACKERS] Re: starting to review the Extend NOT NULL representation to pg_constraint patch)

[Bernd Helmle]

--On 16. Oktober 2010 12:35:06 +0300 Peter Eisentraut  
wrote:



Maybe it should go in src/tools/editors/emacs.samples, too?


Yeah, I think we should recommend some way to highlight faulty
whitespace.

The problem is, after you turn it on, it will make you cry as you
realize how sloppy most code and other files are written.


That's exactly why it is mostly off in my case. But you always can put it 
in a special editing mode, which i currently experimenting with. Thanks for 
your tips.


--
Thanks

Bernd

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

[Terry Laurenzo]

Hi all -
I independently started some work on a similar capability as was contributed
back in August by Joey Adams for a json datatype.  Before starting, I did a
quick search but for some reason didn't turn this existing thread up.

What I've been working on is out on github for now:
http://github.com/tlaurenzo/pgjson

When I started, I was actually aiming for something else, and got caught up
going down this rabbit hole.  I took a different design approach, making the
internal form be an extended BSON stream and implementing event-driven
parsing and serializing to the different formats.  There was some discussion
in the original thread around storing plain text vs a custom format.  I have
to admit I've been back and forth a couple of times on this and have come to
like a BSON-like format for the data at rest.

Pros:
   - It is directly iterable without parsing and/or constructing an AST
   - It is its own representation.  If iterating and you want to tear-off a
value to be returned or used elsewhere, its a simple buffer copy plus some
bit twiddling.
   - It is conceivable that clients already know how to deal with BSON,
allowing them to work with the internal form directly (ala MongoDB)
   - It stores a wider range of primitive types than JSON-text.  The most
important are Date and binary.

Cons:
   - The format appears to have been "grown".  Some of the critical
decisions were made late in the game (ie. why would your integral type codes
be last)
   - Natively, the format falls victim to the artificial document vs element
distinction, which I never understood.  I've worked around this with an
escape mechanism for representing root values, but its not great.
   - The processor is not resilient in the face of unknown element types

I'm leaning towards thinking that the determination comes down to the
following:
   - If you just want a "checkbox" item that the database has a json
datatype and some support functions, storing as text may make sense.  It can
be much simpler; however, it becomes increasingly hard to do anything real
without adding a parse to AST, manipulate, dump to text cycle to every
function.
   - If you want a json datatype that is highly integrated and manipulable,
you want a binary datastructure and in the absence of any other contender in
this area, BSON is ok (not great, but ok).
   - The addition of a JavaScript procedural language probably does not
bring its own format for data at rest.  All of the engines I know of (I
haven't looked at what Guile is doing) do not have a static representation
for internal data structures.  They are heap objects with liberal use of
internal and external pointers.  Most do have a mechanism, however, for
injecting foreign objects into the runtime without resorting to making a
dumb copy.  As such, the integration approach would probably be to determine
the best format for JSON data at rest and provide adapters to the chosen
JavaScript runtime to manipulate this at-rest format directly (potentially
using a copy on write approach).  If the at-rest format is Text, then you
would need to do a parse-to-AST step for each JavaScript function
invocation.

Here's a few notes on my current implementation:
   - Excessive use of lex/yacc: This was quick and easy but the grammars are
simple enough that I'd probably hand-code a parser for any final solution.
   - When the choice between following the json.org spec to the letter and
implementing lenient parsing for valid JavaScript constructs arose, I chose
lenient.
   - Too much buffer copying: When I started, I was just doodling with
writing C code to manipulate JSON/BSON and not working with postgres in
particular.  As such, it all uses straight malloc/free and too many copies
are made to get things in and out of VARDATA structures.  This would all be
eliminated in any real version.
   - UTF-8 is supported but not fully working completely.  The support
functions that Joey wrote do a better job at this.
   - My json path evaluation is crippled.  Given the integration with the PG
type system, I thought I just wanted a simple property traversal mechanism,
punting higher level manipulation to native PG functions.  Seeing real
JSONPath work, though, I'm not so sure.  I like the simplicity of what I've
done but the features of the full bit are nice too.
   - This is first-pass prototype code with the goal of seeing it all
working together.

While I had an end in mind, I did a lot of this for the fun of it and to
just scratch an itch, so I'm not really advocating for anything at this
point.  I'm curious as to what others think the state of JSON and Postgres
should be.  I've worked with JavaScript engines a good deal and would be
happy to help get us there, either using some of the work/approaches here or
going in a different direction.

Terry

Re: [HACKERS] knngist - 0.8

[Peter Eisentraut]

On lör, 2010-10-16 at 09:23 -0700, Paul Ramsey wrote:
> >>   (And, if we are going to break everything
> > in sight, now would be a good time to think about changing typmod to
> > something more flexible than one int32.)
> 
> As someone who is jamming geometry type, spatial reference number and
> dimensionality into said 32bit typmod, let me say emphatically ...
> Amen!

So what kind of data structure would you like for a typmod?



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: extensible enums

[Dean Rasheed]

> On 10/15/2010 04:33 AM, Dean Rasheed wrote:
>>
>> I started looking at this last night, but ran out of time. I'll
>> continue this evening / over the weekend.

Continuing my review of this patch...

Usability review


What the patch does:

This patch adds syntax to allow additional enum values to be added to
an enum type, at any position in the list. The syntax has already been
discussed and a broad consensus reached. To me the new syntax seems
very logical and easy to use/remember.

Do we want that?

Yes, I think so. I can think of a few past projects where I could have
used this. A possible future extension would be the ability to remove
enum values, but that is likely to have additional complications, and
I think the number of use cases for it is smaller. I see no reason to
insist that it be part of this patch.

Do we already have it?

No.

Does it follow SQL spec, or the community-agreed behavior?

Not in the SQL spec, but it does follow agreed behaviour.

Does it include pg_dump support (if applicable)?

Yes.

Are there dangers?

None that I can think of.

Have all the bases been covered?

I just noticed that a couple of columns have been added to pg_type, so
there's another bit documentation that needs updating.

Feature test


Does the feature work as advertised?

Yes.

Are there corner cases the author has failed to consider?

None that I could find.

Are there any assertion failures or crashes?

Yes, I'm afraid I managed to provoke a crash by running the regression
tests with -DCATCACHE_FORCE_RELEASE, after spotting a suspicious bit
of code (see below).

Performance review
--

Does the patch slow down simple tests?

There is a documented performance penalty when working with enum
values that are not in OID order. To attempt to measure this I created
2 tables, foo and bar, each with 800,000 rows. foo had an enum column
using the planets enum from the regression test, with values not in
OID order. bar had a similar enum column, but with values in the
default OID order.

Performance differences for comparison-heavy operations are most noticable:

SELECT MAX(p) FROM foo;
   max
-
 neptune
(1 row)

Time: 132.305 ms

SELECT MAX(p) FROM bar;
   max
-
 neptune
(1 row)

Time: 93.313 ms


SELECT p FROM foo ORDER BY p OFFSET 50 LIMIT 1;
   p

 saturn
(1 row)

Time: 1516.725 ms

SELECT p FROM bar ORDER BY p OFFSET 50 LIMIT 1;
   p

 saturn
(1 row)

Time: 1043.010 ms

(optimised build, asserts off)

That's a bigger performance hit than a I would have expected, even
though it is documented.

enum_ccmp() is using bsearch(), so there's a fair amount of function
call overhead there, and perhaps for small enums, a simple linear scan
would be faster.  I'm not sure to what extent this is worth worrying
about.

Code review
---

I'm not familar enough with the pg_upgrade code to really comment on it.

Looking at AddEnumLabel() I found it a bit hard to follow, but near
the end of the block used when BEFORE/AFTER is specified, it does
this:

ReleaseCatCacheList(list);

/* are the labels sorted by OID? */
if (result && newelemorder > 1)
result = newOid > HeapTupleGetOid(existing[newelemorder-2]);
if (result && newelemorder < nelems + 1)
result = newOid < HeapTupleGetOid(existing[newelemorder-1]);

It looks to me as though 'existing[...]' is a pointer into a member of
the list that's just been freed, so it risks reading freed memory.
That seems to be confirmed by running the tests with
-DCATCACHE_FORCE_RELEASE. Doing so causes a number of the tests to
fail/crash, but I didn't dig any deeper to confirm that this was the
cause.

For the most part, this patch looks good, but I think there is still a
bit of tidying up to do.

Regards,
Dean

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] knngist plans

[Oleg Bartunov]

Marios,

you're right. There are several reasons for poor documentation, but of course,
no excuse, we do need good docs any way ! It's very nice you're willing to 
write one, since it's always better seen from outside of development.

I think it'd be better to use wiki, so other people can join.
Pictures would be nice, but I don't know if there is consensus about using 
pictures in documentation.




Oleg

btw, we're thinking in the background about new indexing infrastructure - 
Spatial GiST, but it's rather big project, so we need sponsors.


On Sat, 16 Oct 2010, Marios Vodas wrote:


Oleg as I said I do believe some things could get better in docs.
Here are 4 main drawbacks in my opinion:
1. The docs lack example(s). There are some references to gistproc.c etc but
this is not part of the docs and if I am someone that has just started this
is hard and confusing (I shouldn't have to read the full technical solution
to another problem in order to solve mine).
2. The 7 methods are poorly explained, at least theoretically. For example
when I started I couldn't understand where each function was used. I had to
read papers to understand the logic.
3. Describe situations that gist can be used. I am aware of spatial indexing
because I worked on it, but I have the impression that there are more (e.g.
computational biology).
4. Why not add some images/figures? (1 image = 1000 words)

GiST is a very important and useful part of postgresql but it's hard to get
started with it. Moreover, I am sure it has capabilities, other than the
profound, that are not known to many people (including me).
Plus knngist will give gist a boost, and I suggest that it shouldn't be
delayed any more.

We have some documentation already, Teodor will post soon.

I would really like to see what Teodor has done.

I could also prepare a document with the additions I would like to see in
docs. Shall I do it? Of course it might take a while to finish it.

On Fri, Oct 15, 2010 at 2:37 PM, Oleg Bartunov  wrote:


On Fri, 15 Oct 2010, Marios Vodas wrote:

 Recently I worked a lot with gist and read about knngist. I have spotted

weaknesses in docs. So I would be happy to help if you tell me how.



We have some documentation already, Teodor will post soon. Marios, I'd
happy to help in writing docs. Do you have some plans ?




On Fri, Oct 15, 2010 at 5:57 AM, Robert Haas 
wrote:

 On Wed, Oct 13, 2010 at 7:07 PM, Marios Vodas  wrote:



I would like to ask in which future version of postgresql knngist is


planned


to be included. Is there any possibility to be included in 9.1?



There's a possibility, but I think the patch still needs some more
work.  One thing that would help is if someone felt motivated to write
some documentation.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company





   Regards,
   Oleg
_
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: o...@sai.msu.su, 
http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83





Regards,
Oleg
_
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: o...@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Extensions, this time with a patch

[Dimitri Fontaine]

Dimitri Fontaine  writes:
> That's very great news. I'm left with moving the bulk of the code away
> from genfile.c and into postgres.c, and have the former be a user
> callable shell around the later, I suppose. Right?

Here it is, looks much better this way.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

*** a/doc/src/sgml/func.sgml
--- b/doc/src/sgml/func.sgml
***
*** 13895,13900  postgres=# SELECT * FROM pg_xlogfile_name_offset(pg_stop_backup());
--- 13895,13907 
 record
 Return information about a file

+   
+
+ pg_execute_from_file(filename text)
+
+void
+Executes the SQL commands contained in a file
+   
   
  
 
***
*** 13933,13938  SELECT (pg_stat_file('filename')).modification;
--- 13940,13954 
  
 
  
+
+ pg_execute_from_file
+
+
+ pg_execute_from_file makes the server
+ execute SQL commands to be found in a file. This function is
+ reserved to superusers.
+
+ 
 
  The functions shown in  manage
  advisory locks.  For details about proper use of these functions, see
***
*** 13955,13960  SELECT (pg_stat_file('filename')).modification;
--- 13971,13977 
 void
 Obtain exclusive advisory lock

+ 

 
  pg_advisory_lock(key1 int, key2 int)
*** a/src/backend/tcop/postgres.c
--- b/src/backend/tcop/postgres.c
***
*** 73,79 
  #include "utils/snapmgr.h"
  #include "mb/pg_wchar.h"
  
- 
  extern char *optarg;
  extern int	optind;
  
--- 73,78 
***
*** 1139,1144  exec_simple_query(const char *query_string)
--- 1138,1277 
  }
  
  /*
+  * exec_multiple_queries
+  *
+  * Execute all queries found in the given query_string. Main use is
+  * pg_execute_from_file() function in backend/utils/adt/genfile.c
+  *
+  * Main differences from exec_simple_query are:
+  * - don't override unnamed portal
+  * - don't start nor finish a transaction
+  * - don't set stats or tracing markers
+  */
+ void
+ exec_multiple_queries(const char *portal_name, const char *query_string)
+ {
+ 	CommandDest dest = DestNone;
+ 	MemoryContext oldcontext;
+ 	List	   *parsetree_list;
+ 	ListCell   *parsetree_item;
+ 	bool		save_log_statement_stats = log_statement_stats;
+ 	bool		was_logged = false;
+ 	bool		isTopLevel;
+ 	char		msec_str[32];
+ 
+ 	oldcontext = MemoryContextSwitchTo(MessageContext);
+ 	parsetree_list = pg_parse_query(query_string);
+ 	MemoryContextSwitchTo(oldcontext);
+ 
+ 	isTopLevel = false;
+ 
+ 	foreach(parsetree_item, parsetree_list)
+ 	{
+ 		Node	   *parsetree = (Node *) lfirst(parsetree_item);
+ 		bool		snapshot_set = false;
+ 		const char *commandTag;
+ 		char		completionTag[COMPLETION_TAG_BUFSIZE];
+ 		List	   *querytree_list,
+    *plantree_list;
+ 		Portal		portal;
+ 		DestReceiver *receiver;
+ 		int16		format = 0; /* TEXT */
+ 
+ 		commandTag = CreateCommandTag(parsetree);
+ 
+ 		/* If we got a cancel signal in parsing or prior command, quit */
+ 		CHECK_FOR_INTERRUPTS();
+ 
+ 		/*
+ 		 * Set up a snapshot if parse analysis/planning will need one.
+ 		 */
+ 		if (analyze_requires_snapshot(parsetree))
+ 		{
+ 			PushActiveSnapshot(GetTransactionSnapshot());
+ 			snapshot_set = true;
+ 		}
+ 
+ 		/*
+ 		 * OK to analyze, rewrite, and plan this query.
+ 		 *
+ 		 * Switch to appropriate context for constructing querytrees (again,
+ 		 * these must outlive the execution context).
+ 		 */
+ 		oldcontext = MemoryContextSwitchTo(MessageContext);
+ 
+ 		querytree_list = pg_analyze_and_rewrite(parsetree, query_string,
+ NULL, 0);
+ 
+ 		plantree_list = pg_plan_queries(querytree_list, 0, NULL);
+ 
+ 		/* Done with the snapshot used for parsing/planning */
+ 		if (snapshot_set)
+ 			PopActiveSnapshot();
+ 
+ 		/* If we got a cancel signal in analysis or planning, quit */
+ 		CHECK_FOR_INTERRUPTS();
+ 
+ 		/*
+ 		 * Create a portal to run the query or queries in. If there already
+ 		 * is one with given portal_name, silently drop it.
+ 		 */
+ 		portal = CreatePortal(portal_name, true, true);
+ 		/* Don't display the portal in pg_cursors */
+ 		portal->visible = false;
+ 
+ 		/*
+ 		 * We don't have to copy anything into the portal, because everything
+ 		 * we are passing here is in MessageContext, which will outlive the
+ 		 * portal anyway.
+ 		 */
+ 		PortalDefineQuery(portal,
+ 		  NULL,
+ 		  query_string,
+ 		  commandTag,
+ 		  plantree_list,
+ 		  NULL);
+ 
+ 		/*
+ 		 * Start the portal.  No parameters here.
+ 		 */
+ 		PortalStart(portal, NULL, InvalidSnapshot);
+ 		PortalSetResultFormat(portal, 1, &format);
+ 
+ 		/*
+ 		 * Now we can create the destination receiver object.
+ 		 */
+ 		receiver = CreateDestReceiver(dest);
+ 		if (dest == DestRemote)
+ 			SetRemoteDestReceiverParams(receiver, portal);
+ 
+ 		/*
+ 		 * Switch back to transaction co

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

[Hitoshi Harada]

2010/10/17 Terry Laurenzo :
> Hi all -
> I independently started some work on a similar capability as was contributed
> back in August by Joey Adams for a json datatype.  Before starting, I did a
> quick search but for some reason didn't turn this existing thread up.
> What I've been working on is out on github for
> now: http://github.com/tlaurenzo/pgjson
> When I started, I was actually aiming for something else, and got caught up
> going down this rabbit hole.  I took a different design approach, making the
> internal form be an extended BSON stream and implementing event-driven
> parsing and serializing to the different formats.  There was some discussion
> in the original thread around storing plain text vs a custom format.  I have
> to admit I've been back and forth a couple of times on this and have come to
> like a BSON-like format for the data at rest.

Reading your proposal, I'm now +1 for BSON-like style. Especially JS
engine's capabilities to map external data to the language
representation are good news. I agree the mapping is engine's task,
not data format task. I'm not sure if your BSON-like format is more
efficient in terms of space and time than plain text, though.

I like as simple design as we can accept. ISTM format, I/O interface,
simple get/set, mapping tuple from/to object, and indexing are minimum
requirement. Something like JSONPath, aggregates, hstore conversion
and whatsoever sound too much.

Regards,

-- 
Hitoshi Harada

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] knngist - 0.8

[Paul Ramsey]

On Sat, Oct 16, 2010 at 10:17 AM, Peter Eisentraut  wrote:
> On lör, 2010-10-16 at 09:23 -0700, Paul Ramsey wrote:
>> >>   (And, if we are going to break everything
>> > in sight, now would be a good time to think about changing typmod to
>> > something more flexible than one int32.)
>>
>> As someone who is jamming geometry type, spatial reference number and
>> dimensionality into said 32bit typmod, let me say emphatically ...
>> Amen!
>
> So what kind of data structure would you like for a typmod?

I'm a primitive enough beast that just having 64-bits would make me
happy. As a general matter though, a bytea?

P

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Extensions, this time with a patch

[Tom Lane]

Dimitri Fontaine  writes:
> Tom Lane  writes:
>> I think that's something that could be left for later, if not never.

> That's very great news. I'm left with moving the bulk of the code away
> from genfile.c and into postgres.c, and have the former be a user
> callable shell around the later, I suppose. Right?

Umm ... I fail to see why an extensions patch should be touching
postgres.c at all, let alone injecting a large amount of code there.
Whatever you're doing there probably requires some rethinking.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Keywords in pg_hba.conf should be field-specific

[Tom Lane]

Alvaro Herrera  writes:
> Excerpts from Brendan Jurd's message of sáb oct 16 11:53:31 -0300 2010:
>> Try New and Improved This Message (tm), now with attachment!

> Hmm.  Would it be possible to list keywords _applicable_ to each field,
> and have these passed down to next_token by the caller instead?

+1.  That newline thing is a crude hack and always has been.  If people
are dissatisfied with it, it's time to start over rather than piling
additional crockery on top.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Keywords in pg_hba.conf should be field-specific

[Tom Lane]

Brendan Jurd  writes:
> Full answer: The problem is that pg_hba.conf doesn't have a fixed
> structure.  Each line can be 4, 5 or 6 fields (not including the final
> 'options' field) long, and which of these structures apply to any
> given line isn't decided until parse_hba_line goes to work on it.

> At the time that next_token gets called, we have no way of knowing
> which field is currently being tokenised, at least not without doing
> some serious rearrangement of hba.c, so that it tokenises and then
> parses one token at a time, instead of tokenising the whole file and
> then parsing all the lines.

Good point.  Maybe the correct fix is to remember whether each token was
quoted or not, so that keyword detection can be done safely after the
initial lexing.  I still think that the current method is impossibly
ugly ...

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Is LISTEN/NOTIFY reliable?

[Tatsuo Ishii]

> On Sat, Oct 16, 2010 at 3:27 AM, Tatsuo Ishii  wrote:
>> Does anybody know PostgreSQL LISTEN/NOTIFY is more reliable than
>> previous versions? ,A (BI vaguely recall that in the previous
>> implementation, message sent by NOTIFY may not be reached to listner.
>> Does PostgreSQL 9.0's new implementation guarantee that the message is
>> received by the listener?
> 
> I think it was always intended to be reliable (otherwise it's not much
> good).  I think I remember a bug where notifications were being lost
> on Windows under heavy load, but I thought we fixed that...

Thanks. Maybe I was confused by the bug.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] knngist - 0.8

[Robert Haas]

On Sat, Oct 16, 2010 at 6:13 PM, Paul Ramsey  wrote:
> On Sat, Oct 16, 2010 at 10:17 AM, Peter Eisentraut  wrote:
>> On lör, 2010-10-16 at 09:23 -0700, Paul Ramsey wrote:
>>> >>   (And, if we are going to break everything
>>> > in sight, now would be a good time to think about changing typmod to
>>> > something more flexible than one int32.)
>>>
>>> As someone who is jamming geometry type, spatial reference number and
>>> dimensionality into said 32bit typmod, let me say emphatically ...
>>> Amen!
>>
>> So what kind of data structure would you like for a typmod?
>
> I'm a primitive enough beast that just having 64-bits would make me
> happy. As a general matter though, a bytea?

Yeah.  It strikes me that there are three main kinds of things people
might want to represent:

1. An integer.  e.g. for a numeric, precision or scale; for a varchar,
length; for an array, number of dimensions.
2. An OID.  e.g. for varchar or text, a collation OID.
3. Recursive structure.  So you might have an array (which is
one-dimensional) containing strings (which are limited to 80
characters and collated in Klingon).  You want to hold onto all of
those details somehow.

There might be use cases for even crazier things - like packing all
the field names and types for a record object in there...  but maybe
that's too crazy to be workable.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] How to reliably detect if it's a promoting standby

[Tatsuo Ishii]

> On Thu, Oct 14, 2010 at 7:35 PM, Tatsuo Ishii  wrote:
>>> What new public interfaces do you think are needed for 9.1 in this
>>> regard?
>>
>> At this point I'm thinking of modifying existing pg_is_in_recovery(),
>> thus 0 new public interface.
> 
> pg_is_in_recovery() returns a bool, are you proposing to change that?

No. I just thought about adding more condition when it returns true.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] knngist - 0.8

[Robert Haas]

On Fri, Oct 15, 2010 at 9:39 PM, Robert Haas  wrote:
> On Fri, Oct 15, 2010 at 8:45 PM, Robert Haas  wrote:
>> On Fri, Oct 15, 2010 at 8:10 PM, Tom Lane  wrote:
 Perhaps we should think of pg_amop not so much
 as a way to tell the AM what to do, but just a way to tell it what
 operator is logically involved without relying on the name or OID.
>>>
>>> I already think of it that way ...
>>
>> OK.
>
> Thinking about it that way, perhaps we could add an integer column
> amop_whats_it_good_for that gets used as a bit field.  That wouldn't
> require changing the index structure, although it might break some
> other things.

I gave this a shot (though I called it amoppurpose rather than
amop_whats_it_good_for) and I think it's a reasonable way to proceed.
Proof-of-concept patch attached.  This just adds the column (using the
existing padding space), defines AMOP_SEARCH and AMOP_ORDER, and makes
just about everything ignore anything not marked AMOP_SEARCH,
attached.  This would obviously need some more hacking to pay
attention to AMOP_ORDER where relevant, etc. and to create some actual
syntax around it.  Currently CREATE OPERATOR CLASS / ALTER OPERATOR
FAMILY have this bit:

OPERATOR strategy_number ( op_type [ , op_type ] )

knngist-0.9 implements this:

[ORDER] OPERATOR strategy_number ( op_type [, op_type ] )

...but with the design proposed above that's not quite what we'd want,
because amoppurpose is a bit field, so you could have one or both of
the two possible purposes.  Perhaps:

OPERATOR strategy_number ( op_type [ , op_type ] ) [ FOR { SEARCH |
ORDER } [, ...] ]

With the default being FOR SEARCH.

Comments?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Extensions, this time with a patch

[Alvaro Herrera]

Excerpts from Tom Lane's message of sáb oct 16 19:52:27 -0300 2010:
> Dimitri Fontaine  writes:
> > Tom Lane  writes:
> >> I think that's something that could be left for later, if not never.
> 
> > That's very great news. I'm left with moving the bulk of the code away
> > from genfile.c and into postgres.c, and have the former be a user
> > callable shell around the later, I suppose. Right?
> 
> Umm ... I fail to see why an extensions patch should be touching
> postgres.c at all, let alone injecting a large amount of code there.
> Whatever you're doing there probably requires some rethinking.

Hm, it was me that led him in that direction.  The original patch was
just copying a bunch of code from postgres.c into genfile.c, which
struck me as a worse proposition.

The intent here is to execute some code from the file directly inside
the server.

Eh, I realize now that the right way to go about this is to use SPI.

-- 
Álvaro Herrera 
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] How to determine failed connection attempt due to invalid authorization (libpq)?

[Robert Haas]

On Thu, Oct 14, 2010 at 2:59 PM, Dmitriy Igrishin  wrote:
> Is there way to determine failed connection attempt due to invalid
> authorization (libpq)?

I think this question would be more appropriate on pgsql-general.  I
suppose you would have to look at PQerrorMessage().

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Extensions, this time with a patch

[Tom Lane]

Alvaro Herrera  writes:
> The intent here is to execute some code from the file directly inside
> the server.

> Eh, I realize now that the right way to go about this is to use SPI.

Yeah, that would be one way to go about it.  But IMO postgres.c should
be solely concerned with interactions with the client.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] knngist - 0.8

[Robert Haas]

On Fri, Oct 15, 2010 at 7:10 PM, Tom Lane  wrote:
> Robert Haas  writes:
>> I still feel vaguely uneasy about the fact that the proposed patch
>> can't handle ASC/DESC or NULLS FIRST/LAST, and that unease grew a bit
>> more last night when I read Peter's patch to add collation support.
>
> Good point.
>
>> We could possibly cram ASC/DESC and NULLS FIRST/LAST in by defining
>> four new categories of operator strategies rather than one, but
>> there's no way that's going to work for collations.  Is there some
>> other way to approach this problem?  Can we leave pg_amop as it is,
>> and pass the context (sort vs. qual, ASC/DESC, NULLS FIRST/LAST,
>> collation, whatever...) to the index via some sort of side channel?
>
> Well, we cannot avoid changing pg_amop, or at least changing its
> interpretation, because the current scheme simply can't represent
> indexable operators that are used for anything except simple boolean
> WHERE tests.  I agree though that we do *not* want pg_amop involved
> in the details of exactly what sort ordering is referenced by a sortable
> operator.  Somehow that needs to be passed in a side channel.

I spent some time tonight looking at how this is handled in
builtin_knngist_core-0.9.  It looks like the requested sort, if there
is one, just gets tossed into the indexquals list and eventually
transformed into a scankey with a new flag SK_ORDER set.  I think what
we should do instead is add an additional IndexPath field which can
point to a node indicating the desired ordering properties.  This node
might look similar to a scan key but having it be a separate node type
will allow us to add in whatever, ahem, miscellaneous crap we want to
pass: currently ASC/DESC and NULLS FIRST/LAST, and eventually
collation and so forth.  That can then get propagated into the
IndexScan and passed to index_beginscan, where it can be passed to the
AM's ambeginscan function (i.e. we'll add an additional argument to
the signatures of those functions).

There's a second problem, too.  If we assume that we want to treat
this problem with some degree of generality - that is, we really do
care about things like ASC/DESC and NULLS FIRST/LAST and eventually
collation - then the proposed amcanorderbyop flag isn't really
sufficient.  The AM will need to be able to indicate whether a given
combination of parameters is one that it can handle.  I'm thinking
perhaps in lieu of a boolean, we can add another indexam method which,
if not InvalidOid, gets called when we're wondering about whether a
given clause is something that the index can order by.  Although
knngist focuses on a the ORDER BY col OP constant case, which
certainly seems like the most useful one, there's no theoretical
reason why an AM couldn't allow ordering by some more complex clause.
I don't know whether anyone will ever feel like writing code to do
something like that, but if we set the API up this way then it should
be at least theoretically possible.

Comments?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Extensions, this time with a patch

[Alvaro Herrera]

Excerpts from Tom Lane's message of sáb oct 16 23:32:49 -0300 2010:
> Alvaro Herrera  writes:
> > The intent here is to execute some code from the file directly inside
> > the server.
> 
> > Eh, I realize now that the right way to go about this is to use SPI.
> 
> Yeah, that would be one way to go about it.  But IMO postgres.c should
> be solely concerned with interactions with the client.

Duly noted.

-- 
Álvaro Herrera 
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Review: Fix snapshot taking inconsistencies

[Alvaro Herrera]

Excerpts from Tom Lane's message of mar oct 12 20:49:28 -0300 2010:
> Marko Tiikkaja  writes:
> > On 2010-10-13 2:10 AM +0300, Tom Lane wrote:
> >> BTW, this patch seems to be also the time to remove the AtStart_Cache()
> >> call in CommandCounterIncrement, as foreseen in the comment there.
> 
> > Frankly, I have no idea what to do about this.
> 
> Just delete the call.  The only reason I didn't remove it in 2007 is
> I was afraid to risk changing things in late beta; but that's not the
> situation now.

I just applied just this change and ran the regression tests; it works
fine.  I didn't do anything else though, like the cache-clobber-always
flag, etc.  If no one objects I will push this patch to see what the
buildfarm has to say about it.

diff --git a/src/backend/access/transam/xact.c 
b/src/backend/access/transam/xact.c
index b02db9e..d2e2e11 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -729,17 +729,6 @@ CommandCounterIncrement(void)
 */
AtCCI_LocalCache();
}
-
-   /*
-* Make any other backends' catalog changes visible to me.
-*
-* XXX this is probably in the wrong place: CommandCounterIncrement should
-* be purely a local operation, most likely.  However fooling with this
-* will affect asynchronous cross-backend interactions, which doesn't seem
-* like a wise thing to do in late beta, so save improving this for
-* another day - tgl 2007-11-30
-*/
-   AtStart_Cache();
 }
 
 /*


-- 
Álvaro Herrera 
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Keywords in pg_hba.conf should be field-specific

[Brendan Jurd]

On 17 October 2010 09:59, Tom Lane  wrote:
> Brendan Jurd  writes:
>> At the time that next_token gets called, we have no way of knowing
>> which field is currently being tokenised, at least not without doing
>> some serious rearrangement of hba.c, so that it tokenises and then
>> parses one token at a time, instead of tokenising the whole file and
>> then parsing all the lines.
>
> Good point.  Maybe the correct fix is to remember whether each token was
> quoted or not, so that keyword detection can be done safely after the
> initial lexing.  I still think that the current method is impossibly
> ugly ...
>

I'm happy to revise the patch on that basis.  Any suggestions about
how to communicate the 'quotedness' of each token?  We could make each
token a struct consisting of the token itself, plus a boolean flag to
indicate whether it had been quoted.  Does that work for you?

Cheers,
BJ

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] How to determine failed connection attempt due to invalid authorization (libpq)?

[Dmitriy Igrishin]

Hey Robert,

I've asked pgsql-general.
Unfortunately it seems that there is no better way to do it except
parsing PQerrorMessage(). Sadly.


2010/10/17 Robert Haas 

> On Thu, Oct 14, 2010 at 2:59 PM, Dmitriy Igrishin 
> wrote:
> > Is there way to determine failed connection attempt due to invalid
> > authorization (libpq)?
>
> I think this question would be more appropriate on pgsql-general.  I
> suppose you would have to look at PQerrorMessage().
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>



-- 
// Dmitriy.