Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[Hannu Krosing]

On Fri, 2012-05-04 at 16:12 -0400, Tom Lane wrote:
> Robert Haas  writes:
> > On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing  wrote:
> >> Can we at least have the xxx_to_json() functions try cast to json first
> >> and fall back to text if the cast fails.
> 
> > I think the idea that you can involve the casting machinery in this is
> > misguided.

> It is possible that that can be made to work, but it's a research
> project, not something to be crammed into 9.2 at the last possible
> minute.  In any case, I really dislike the idea that array_to_json
> and row_to_json would contain two entirely different behaviors.
> Leave the extensibility ideas for a future to_json() function.

I did not mean that array_to_json and row_to_json would be different
than the generic to_json, just that they would be thin wrappers around
the to_json function which check that the arguments are of the correct
types for casting to "JSON text". 

They need to recurse to each other and generic to_json anyway.

>   regards, tom lane

-- 
---
Hannu Krosing
PostgreSQL Unlimited Scalability and Performance Consultant
2ndQuadrant Nordic
PG Admin Book: http://www.2ndQuadrant.com/books/


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[Hannu Krosing]

On Fri, 2012-05-04 at 15:59 -0400, Robert Haas wrote:
> On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing  wrote:
> > On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote:
> >> Hannu Krosing  writes:
> >> > On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
> >> >> So given that do we do anything about this now, or wait till 9.3?
> >>
> >> > I'd like the json support in 9.2 updated as follows
> >>
> >> I think it's too late to be entertaining proposals for such changes in
> >> 9.2.  If we had concluded that the existing functions were actively
> >> wrong or a bad idea,
> >
> > I think that hard-coding "postgresql text" representation as our json
> > representation without a possibility for the user tio easily fix it
> > without rewriting foll xx_to_json() functions is borderline "actively
> > wrong".
> >
> > Can we at least have the xxx_to_json() functions try cast to json first
> > and fall back to text if the cast fails.
> 
> I think the idea that you can involve the casting machinery in this is
> misguided.  sometextval::json has got to mean that sometextval is
> expected to be in the form of a syntactically correct JSON value - and
> NOT that we wrap it in a JSON string.  

PostgreSQL CAST is different from casting a C pointer, they actually
perform a conversion when defined WITH FUNCTION or WITH INOUT. 
And they pass value unchanged when defined WITHOUT FUNCTION.

Casts _do_not_ perform syntax checks, they assume the source to be of
the type castted from and they return result of the target type.

I think you are confusing input/output formatting with CAST here.

This confusion about cast-as-syntax-check probably qualifies as
something being "actively wrong" with current implementation, though it
does not seem to be something actively defined (there are no casts
defined for json type) but rather an artifact of how postgresql input
works.

CAST is something that should convert one type to another, in this case
a textual type to its "json value" representation and back.

'sometext'::text::json --> '"sometext"'

and 

'"sometext"'::json::text --> 'sometext'


the suggested "syntax check only" should only be done by the type i/o
functions json_in(cstring) and json_recv(internal) and not casts.

Casts should do "casting", that in PostgreSQL means type conversion


> We can have constructors for
> JSON, but they've got to be separate from the casting machinery.

Currently we do have constructors - json_in(csting) and
json_recv(internal)

These are the ones that should and do  check for correct syntax.


Some more confusiong examples for pondering on cast vs i/o functions


Some of the difficulties of understanding and explaining what a json
type should be are exemplified in the following


hannu=# create table jtest(plaintext text, jsontext json); 
CREATE TABLE

hannu=# insert into jtest values('"A"','"A"');
hannu=# insert into jtest values('true','true');
hannu=# insert into jtest values('null','null');
hannu=# insert into jtest values(null,null);

hannu=# select row_to_json(jtest) from jtest;
 row_to_json  
--
 {"plaintext":"\"A\"","jsontext":"A"}
 {"plaintext":"true","jsontext":true}
 {"plaintext":"null","jsontext":null}
 {"plaintext":null,"jsontext":null}
(4 rows)

hannu=# insert into jtest values('a','a');
ERROR:  invalid input syntax for type json
LINE 1: insert into jtest values('a','a');
 ^
DETAIL:  line 1: Token "a" is invalid.



-- 
---
Hannu Krosing
PostgreSQL Unlimited Scalability and Performance Consultant
2ndQuadrant Nordic
PG Admin Book: http://www.2ndQuadrant.com/books/


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Documentation for temp_file_limit

[Bruce Momjian]

The new 9.2 GUC parameter temp_file_limit says it restricts temporary
file usage per session, but it doesn't say what happens if a session
needs to exceed that value --- it throws an error.  Shouldn't we mention
that?

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] remove dead ports?

[Bruce Momjian]

On Fri, May 04, 2012 at 06:25:24PM -0400, Tom Lane wrote:
> Bruce Momjian  writes:
> > On Fri, May 04, 2012 at 08:45:10PM +0300, Peter Eisentraut wrote:
> >> I'm not so much opposed to removing the port.  I am more concerned about
> >> the manner in which it was done.  The other ports I removed were known
> >> to not work anyway, for years, and there were at least several days of
> >> discussion.  The bsdi case was removing a working port with less than 24
> >> hours notice.
> 
> What's the grounds for asserting they were known not to work?  Not
> actual testing, I assume.
> 
> > Basically, we have beta next week so I wanted to do it before then, and
> > I have my head down doing the release notes, so I wanted to do it before
> > I started that too.  I kind of knew the bsdi answer before I even asked.
> 
> > If you are objecting to me short-circuiting this, I will revert the
> > patch.  If we can't short-circuiting thinks when we already know the
> > answer, everyone's work will take more time.
> 
> Leave it as-is.  I agree with the upthread comment that we can revert
> the patch during beta (or even later than that), if anyone complains.
> Furthermore, I would want to insist that a complainer provide a
> buildfarm member as the price of us continuing to support an old
> uncommon platform.  Otherwise the apparent support is hollow.  The BSDI
> port was viable for us to support as long as Bruce was using it daily,
> but with that gone, we need somebody else to be testing it.

Yes, it was an odd port that probably would have been removed five years
ago if I had not been using it, which I am no longer.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] remove dead ports?

[Tom Lane]

Bruce Momjian  writes:
> On Fri, May 04, 2012 at 08:45:10PM +0300, Peter Eisentraut wrote:
>> I'm not so much opposed to removing the port.  I am more concerned about
>> the manner in which it was done.  The other ports I removed were known
>> to not work anyway, for years, and there were at least several days of
>> discussion.  The bsdi case was removing a working port with less than 24
>> hours notice.

What's the grounds for asserting they were known not to work?  Not
actual testing, I assume.

> Basically, we have beta next week so I wanted to do it before then, and
> I have my head down doing the release notes, so I wanted to do it before
> I started that too.  I kind of knew the bsdi answer before I even asked.

> If you are objecting to me short-circuiting this, I will revert the
> patch.  If we can't short-circuiting thinks when we already know the
> answer, everyone's work will take more time.

Leave it as-is.  I agree with the upthread comment that we can revert
the patch during beta (or even later than that), if anyone complains.
Furthermore, I would want to insist that a complainer provide a
buildfarm member as the price of us continuing to support an old
uncommon platform.  Otherwise the apparent support is hollow.  The BSDI
port was viable for us to support as long as Bruce was using it daily,
but with that gone, we need somebody else to be testing it.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: xReader, double-effort (was: Temporary tables under hot standby)

[Jim Nasby]

On 5/3/12 2:54 PM, Josh Berkus wrote:

(2)  If logical transactions had been implemented as additions to
>  the WAL stream, and Slony was using that, do you think they would
>  still have been usable for this recovery?

Quite possibly not.


The key advantage that I see in londiste/slony replication is that your data 
stream has absolutely nothing to do with anything binary or internal to 
Postgres. That means that the only way corruption will travel from a master to 
a slave is if the corruption is in the actual fields being updated, and even 
that's not a given (ie: UPDATING a field to a completely new value would not 
propagate corruption even if the old value of the field was corrupted).

So, embedding a logical stream into WAL is not inherently bad... what would be bad is if 
that "logical" stream was susceptible to corruption due to something like full 
page writes. Simply embedding the exact same info slony or londiste captures into the WAL 
should be fine (though likely defeats the purpose). Translating binary WAL data into DML 
statements would very likely allow corruption to travel from master to slave.
--
Jim C. Nasby, Database Architect   j...@nasby.net
512.569.9461 (cell) http://jim.nasby.net

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] remove dead ports?

[Bruce Momjian]

On Fri, May 04, 2012 at 08:45:10PM +0300, Peter Eisentraut wrote:
> On tor, 2012-05-03 at 17:39 +0100, Peter Geoghegan wrote:
> > On 3 May 2012 17:21, Bruce Momjian  wrote:
> > > I think I was the only user left;  I have never heard from a BSD/OS user
> > > in the past 5-7 years.
> > 
> > I'm inclined to agree with Bruce. While it's not reasonable to assume
> > that the lack of a BSD/OS user complaining on -general indicates that
> > there are none, it's also not reasonable for them to expect us to
> > support their operating system for 8 years after the original
> > proprietary vendor. Better to not support BSD/OS than to supply a port
> > that no one really has any confidence in. It's not as if we've ceased
> > support in release branches.
> 
> I'm not so much opposed to removing the port.  I am more concerned about
> the manner in which it was done.  The other ports I removed were known
> to not work anyway, for years, and there were at least several days of
> discussion.  The bsdi case was removing a working port with less than 24
> hours notice.

Not sure where you got 24 hours:

  Tues  http://archives.postgresql.org/pgsql-hackers/2012-05/msg00061.php
  Wed   http://archives.postgresql.org/pgsql-general/2012-05/msg00060.php
  Thur  http://archives.postgresql.org/pgsql-committers/2012-05/msg00023.php

Basically, we have beta next week so I wanted to do it before then, and
I have my head down doing the release notes, so I wanted to do it before
I started that too.  I kind of knew the bsdi answer before I even asked.

If you are objecting to me short-circuiting this, I will revert the
patch.  If we can't short-circuiting thinks when we already know the
answer, everyone's work will take more time.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[David Johnston]

> -Original Message-
> From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-
> ow...@postgresql.org] On Behalf Of Hannu Krosing
> Sent: Friday, May 04, 2012 4:40 PM
> To: Robert Haas
> Cc: Tom Lane; Andrew Dunstan; PostgreSQL-development; Merlin Moncure
> Subject: Re: [HACKERS] JSON in 9.2 - Could we have just one to_json()
> function instead of two separate versions ?
> 
> On Fri, 2012-05-04 at 15:59 -0400, Robert Haas wrote:
> > On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing 
> wrote:
> > > On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote:
> > >> Hannu Krosing  writes:
> > >> > On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
> > >> >> So given that do we do anything about this now, or wait till 9.3?
> > >>
> > >> > I'd like the json support in 9.2 updated as follows
> > >>
> > >> I think it's too late to be entertaining proposals for such changes
> > >> in 9.2.  If we had concluded that the existing functions were
> > >> actively wrong or a bad idea,
> > >
> > > I think that hard-coding "postgresql text" representation as our
> > > json representation without a possibility for the user tio easily
> > > fix it without rewriting foll xx_to_json() functions is borderline
> > > "actively wrong".
> > >
> > > Can we at least have the xxx_to_json() functions try cast to json
> > > first and fall back to text if the cast fails.
> >
> > I think the idea that you can involve the casting machinery in this is
> > misguided.  sometextval::json has got to mean that sometextval is
> > expected to be in the form of a syntactically correct JSON value - and
> > NOT that we wrap it in a JSON string.  We can have constructors for
> > JSON, but they've got to be separate from the casting machinery.
> 
> on the contrary - the string representation of textual value a is "a"
> 
> casting should _not_ neam syntax check, casting is by definition a conversion.
> 
> if we cast text to int, we return value of type int , if we cast int to
> numeric(5,2) we return value of type numeric(5,2)
> 
> why should casring to json work differntly ?
> 

What is the distinction between what you are thinking regarding JSON and this 
example?

SELECT '1a'::integer;
SQL Error: ERROR:  invalid input syntax for integer: "1a"
LINE 1: SELECT '1a'::integer

As a user if I cast something to something else I want the result to be of the 
correct type and deterministic; otherwise throw me some kind of "invalid input 
format" exception (or syntax exception).  Casting vs. Constructors is really a 
meaningless distinction to a lay person.  When I cast I do so by constructing a 
new value using my existing value for input.  When I use an explicit CAST I am 
unable to supply additional parameters to configure the casting whereas a 
constructor function gives me that possibility.  But a constructor function 
without any additional parameters is not semantically different than a cast.

I guess the concern to address is something like:

SELECT '{key: value}'::json OR SELECT '[1.25]'::json;  Do you interpret this as 
already being valid JSON and thus output object/array constructs (i.e., JSON 
Text) or do you treat them as string literals and output scalars (i.e., JSON 
Value).  Even if you feel these are artificial constructs the concepts holds 
that there may be ambiguous data that can be interpreted in multiple ways (this 
applies even to function forms, though in the function form you could specify 
which one you want to output using a separate DEFAULTed parameter). 



I can see the primary use-case for JSON Value casting as being queries of the 
following forms (since the record and array forms are going through the 
record/array_to_json function):

SELECT COUNT(*)::json FROM table [WHERE ...];
SELECT single_col::json FROM table WHERE id = ?;

Where the single provided value can be sent directly back to the web-caller 
JavaScript and used as-is because it is valid JSON.  Though, honestly, both 
SELECT to_json(single_col) and SELECT to_json(COUNT(*)) are equally usable so 
any distinction between them is a pure technical issue to me.

Am I correct in assuming the following expected behavior (the forgive the 
blatantly wrong syntax but you should get the point)?

RAISE NOTICE '%', SELECT 'A'::text  =>  A
RAISE NOTICE '%', SELECT 'A'::json => "A"

David J



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] remove dead ports?

[Marko Kreen]

On Fri, May 4, 2012 at 8:45 PM, Peter Eisentraut  wrote:
> On tor, 2012-05-03 at 17:39 +0100, Peter Geoghegan wrote:
>> On 3 May 2012 17:21, Bruce Momjian  wrote:
>> > I think I was the only user left;  I have never heard from a BSD/OS user
>> > in the past 5-7 years.
>>
>> I'm inclined to agree with Bruce. While it's not reasonable to assume
>> that the lack of a BSD/OS user complaining on -general indicates that
>> there are none, it's also not reasonable for them to expect us to
>> support their operating system for 8 years after the original
>> proprietary vendor. Better to not support BSD/OS than to supply a port
>> that no one really has any confidence in. It's not as if we've ceased
>> support in release branches.
>
> I'm not so much opposed to removing the port.  I am more concerned about
> the manner in which it was done.  The other ports I removed were known
> to not work anyway, for years, and there were at least several days of
> discussion.  The bsdi case was removing a working port with less than 24
> hours notice.

There is enough time until 9.2-final for a BSD/OS user raise complaints.

-- 
marko

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[Hannu Krosing]

On Fri, 2012-05-04 at 15:59 -0400, Robert Haas wrote:
> On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing  wrote:
> > On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote:
> >> Hannu Krosing  writes:
> >> > On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
> >> >> So given that do we do anything about this now, or wait till 9.3?
> >>
> >> > I'd like the json support in 9.2 updated as follows
> >>
> >> I think it's too late to be entertaining proposals for such changes in
> >> 9.2.  If we had concluded that the existing functions were actively
> >> wrong or a bad idea,
> >
> > I think that hard-coding "postgresql text" representation as our json
> > representation without a possibility for the user tio easily fix it
> > without rewriting foll xx_to_json() functions is borderline "actively
> > wrong".
> >
> > Can we at least have the xxx_to_json() functions try cast to json first
> > and fall back to text if the cast fails.
> 
> I think the idea that you can involve the casting machinery in this is
> misguided.  sometextval::json has got to mean that sometextval is
> expected to be in the form of a syntactically correct JSON value - and
> NOT that we wrap it in a JSON string.  We can have constructors for
> JSON, but they've got to be separate from the casting machinery.

on the contrary - the string representation of textual value a is "a"

casting should _not_ neam syntax check, casting is by definition a
conversion.

if we cast text to int, we return value of type int , if we cast int to
numeric(5,2) we return value of type numeric(5,2)

why should casring to json work differntly ?

> -- 
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
> 



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[Tom Lane]

Robert Haas  writes:
> On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing  wrote:
>> Can we at least have the xxx_to_json() functions try cast to json first
>> and fall back to text if the cast fails.

> I think the idea that you can involve the casting machinery in this is
> misguided.

It is possible that that can be made to work, but it's a research
project, not something to be crammed into 9.2 at the last possible
minute.  In any case, I really dislike the idea that array_to_json
and row_to_json would contain two entirely different behaviors.
Leave the extensibility ideas for a future to_json() function.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[Robert Haas]

On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing  wrote:
> On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote:
>> Hannu Krosing  writes:
>> > On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
>> >> So given that do we do anything about this now, or wait till 9.3?
>>
>> > I'd like the json support in 9.2 updated as follows
>>
>> I think it's too late to be entertaining proposals for such changes in
>> 9.2.  If we had concluded that the existing functions were actively
>> wrong or a bad idea,
>
> I think that hard-coding "postgresql text" representation as our json
> representation without a possibility for the user tio easily fix it
> without rewriting foll xx_to_json() functions is borderline "actively
> wrong".
>
> Can we at least have the xxx_to_json() functions try cast to json first
> and fall back to text if the cast fails.

I think the idea that you can involve the casting machinery in this is
misguided.  sometextval::json has got to mean that sometextval is
expected to be in the form of a syntactically correct JSON value - and
NOT that we wrap it in a JSON string.  We can have constructors for
JSON, but they've got to be separate from the casting machinery.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[Hannu Krosing]

On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote:
> Hannu Krosing  writes:
> > On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
> >> So given that do we do anything about this now, or wait till 9.3?
> 
> > I'd like the json support in 9.2 updated as follows
> 
> I think it's too late to be entertaining proposals for such changes in
> 9.2.  If we had concluded that the existing functions were actively
> wrong or a bad idea,

I think that hard-coding "postgresql text" representation as our json
representation without a possibility for the user tio easily fix it
without rewriting foll xx_to_json() functions is borderline "actively
wrong".

Can we at least have the xxx_to_json() functions try cast to json first
and fall back to text if the cast fails.

This would address my worst problem, all the rest can be easily defined
in user functions.

>  then of course we'd need to do something; but they
> are not, so we can just as well consider additions in the 9.3 cycle
> rather than now.  I am not convinced that this proposal is fully baked
> yet, anyway; not to mention that right now we need to have our heads
> down on resolving the remaining open issues, not designing,
> implementing, and reviewing a pile of brand new code for json.
> 
> > By allowing developers just to define their own to_json(date) function
> > we give them the power do decide which one to use. And if we honour
> > search_path when looking up the to_json() functions, then they can even
> > choose to have different conventions for different applications.
> 
> This is not going to work anywhere near as nicely as you think.  If
> somebody tries to define multiple to_json() functions that override a
> generic to_json(anyelement) one, he will start getting "function is not
> unique" parse failures.  The parser will only successfully decide which
> function to call when the input data type exactly matches one of the
> specialized functions, which means you might as well not have the
> generic one at all.
> 
>   regards, tom lane
> 



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[Hannu Krosing]

On Fri, 2012-05-04 at 13:43 -0400, Robert Haas wrote:
> On Fri, May 4, 2012 at 12:56 PM, Tom Lane  wrote:
> > Andrew Dunstan  writes:
> >> Yeah, what I've been thinking about in conjunction with similar problems
> >> is some sort of type registry, so that we could code for non-builtin
> >> types in certain cases. Maybe we should add that the the developers'
> >> meeting agenda.
> >
> > Maybe.  I don't want to see a json-specific hack for this, but some sort
> > of generic way to add type knowledge could be useful, if we could figure
> > out what we want.
> 
> For this particular case, I think you just need some place to store a
> pg_type -> pg_proc mapping.  I'm not exactly sure how to make that not
> a JSON-specific hack, since I certainly don't think we'd want to add a
> new catalog just for that.

This was my initial proposal to have casts to ::json for all types.

I backed out from this in favot of generic to_json(datum, indent) in
order to support prettyprinting.

> -- 
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
> 



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PL/Python result set slicing broken in Python 3

[Jan Urbański]

On 03/05/12 11:04, Jan Urbański wrote:

On 02/05/12 20:18, Peter Eisentraut wrote:

This doesn't work anymore with Python 3:

rv = plpy.execute(...)
do_something(rv[0:1])

Apparently, they changed the C API for doing slicing, or rather made one
of the two APIs for it silently do nothing. Details are difficult to
find, but this email message seems to contain something:
.

I'll try to sort this out sometime, but if someone wants to take a shot
at it, go ahead.


Sounds ugly. I'll take a look.


I found some instructions on how to deal with the Python 2/Python 3 
slicing mess:


http://renesd.blogspot.com/2009/07/python3-c-api-simple-slicing-sqslice.html

Apparently you need that egregious hack in order to avoid code 
duplication. I'll try to produce a patch over the weekend.


Cheers,
Jan

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Uppercase tab completion keywords in psql?

[Peter Eisentraut]

On tor, 2012-05-03 at 15:47 -0400, Bruce Momjian wrote:
> Peter, where are we on this?

I hadn't received any clear feedback, but if no one objects, I can
commit it.

> ---
> 
> On Fri, Mar 30, 2012 at 08:16:59PM +0300, Peter Eisentraut wrote:
> > On fre, 2012-03-23 at 07:52 -0700, David Fetter wrote:
> > > On Thu, Mar 22, 2012 at 06:05:30PM -0400, Andrew Dunstan wrote:
> > > > On 03/22/2012 05:49 PM, Bruce Momjian wrote:
> > > > >Robert Haas and I are disappointed by this change.  I liked the
> > > > >fact that I could post nice-looking SQL queries without having to
> > > > >use my capslock key (which I use as a second control key).  Any
> > > > >chance of reverting this change?
> > > > >
> > > > 
> > > > Should it be governed by a setting?
> > > 
> > > Something like (upper|lower|preserve) ?
> > 
> > How about this patch then?  (There are actually four possible settings,
> > see patch.)



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] remove dead ports?

[Peter Eisentraut]

On tor, 2012-05-03 at 17:39 +0100, Peter Geoghegan wrote:
> On 3 May 2012 17:21, Bruce Momjian  wrote:
> > I think I was the only user left;  I have never heard from a BSD/OS user
> > in the past 5-7 years.
> 
> I'm inclined to agree with Bruce. While it's not reasonable to assume
> that the lack of a BSD/OS user complaining on -general indicates that
> there are none, it's also not reasonable for them to expect us to
> support their operating system for 8 years after the original
> proprietary vendor. Better to not support BSD/OS than to supply a port
> that no one really has any confidence in. It's not as if we've ceased
> support in release branches.

I'm not so much opposed to removing the port.  I am more concerned about
the manner in which it was done.  The other ports I removed were known
to not work anyway, for years, and there were at least several days of
discussion.  The bsdi case was removing a working port with less than 24
hours notice.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[Robert Haas]

On Fri, May 4, 2012 at 12:56 PM, Tom Lane  wrote:
> Andrew Dunstan  writes:
>> Yeah, what I've been thinking about in conjunction with similar problems
>> is some sort of type registry, so that we could code for non-builtin
>> types in certain cases. Maybe we should add that the the developers'
>> meeting agenda.
>
> Maybe.  I don't want to see a json-specific hack for this, but some sort
> of generic way to add type knowledge could be useful, if we could figure
> out what we want.

For this particular case, I think you just need some place to store a
pg_type -> pg_proc mapping.  I'm not exactly sure how to make that not
a JSON-specific hack, since I certainly don't think we'd want to add a
new catalog just for that.

In general, I think it would be very useful to have some way of
identifying particular types - and versions of types - independently
of a particular installation - e.g. by assigning each type a UUID that
never changes and a version number that we bump when we change
something about that type.  That seems like it would be very useful
for schema comparison tools, or for logical replication, where you
want to know whether two types are "the same type" even though they
are in different clusters.  pg_upgrade has had past needs in this area
as well.  However, I'm not sure that'd help solve this particular
problem.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Future In-Core Replication

[Robert Haas]

On Fri, May 4, 2012 at 12:59 PM, Andres Freund  wrote:
> In my understanding - as the person doing quite a bit of the coding atm - the
> point is to provide a very minimal *early* prototype to have a sensible basis
> for design decisions/discussions. On one side thats useful to get a feeling
> for the problems involved. On the other side doing design discussions without
> an underlaying basic patch & design on -hackers tends to often go into
> directions of feature creep and bikeshedding. It also helps against "this is
> impossible" claims.
> Parts of this thread and related ones are a somewhat good example of this.
>
> The plan is to show the early prototype around pgcon and send design documents
> and split-up patches (of that prototype) a holiday and some cleanup later to -
> hackers. I/We aim to have individual, independently usable, parts of the patch
> submitted to the first 9.3 commitfest.
>
> I definitely do not wish to screw anyone over doing this or such. And I am
> sure thats the same with others working on this. Even if sometimes emotions
> get in the way/into play.

Thanks.  I think we are basically all on the same page, here.  As I
said upthread, I am not trying to accuse anything of doing anything
wrong, and as you and Simon and Greg have all said, you're not trying
to do anything wrong.  So, no problem.  :-)

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Future In-Core Replication

[Andres Freund]

Hi Robert, Hi all,

On Friday, May 04, 2012 06:29:33 PM Robert Haas wrote:
> On Fri, May 4, 2012 at 11:06 AM, Greg Smith  wrote:
> > The straw man argument here would require 100% transparency on everything
> > you do in regards to PostgreSQL and related software.  Before doing any
> > development on any code, first post here to ask for design review.  And
> > if someone asks you to work on a program that isn't open source from day
> > one, refuse unless you can operate that transparently.
> 
> Well, look.  At the end of the day, I don't really care whether you
> post your designs before writing code or not - unless it turns out
> that we get to the end of the development cycle, a gigantic patch
> shows up at the last minute, it gets rejected because people aren't
> satisfied with the design, and then massive bitching ensues because
> the author(s) put a lot of work into that patch.  Then I care, because
> now the fact that no design consensus was sought at the outset has
> been transformed into a defect in the community process, which does in
> fact have defects, but that isn't one of them.  We all know that
> design review is going to have to happen at some point, and if there's
> not an adequate opportunity to do that before the code is written then
> it will happen after the code is written.  If that means the code has
> to be thrown out, then that's the risk you take by writing the code
> first.  As long as everybody understands that, do it in whatever order
> you like.
In my understanding - as the person doing quite a bit of the coding atm - the 
point is to provide a very minimal *early* prototype to have a sensible basis 
for design decisions/discussions. On one side thats useful to get a feeling 
for the problems involved. On the other side doing design discussions without 
an underlaying basic patch & design on -hackers tends to often go into 
directions of feature creep and bikeshedding. It also helps against "this is 
impossible" claims.
Parts of this thread and related ones are a somewhat good example of this.

The plan is to show the early prototype around pgcon and send design documents 
and split-up patches (of that prototype) a holiday and some cleanup later to -
hackers. I/We aim to have individual, independently usable, parts of the patch 
submitted to the first 9.3 commitfest.

I definitely do not wish to screw anyone over doing this or such. And I am 
sure thats the same with others working on this. Even if sometimes emotions 
get in the way/into play.

Greetings,

Andres
-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Beta time?

[Tom Lane]

Josh Berkus  writes:
>> How are we handling the Monday release with everyone at PGCon?  Was that
>> resolved?

> I have yet to see a confirmed date, guys.  If we expect any support from
> the packagers and/or the advocacy volunteers, then people need at least
> a week's notice, probably more.

I haven't seen anybody positively say we can't do a wrap next Thursday
and release Monday, so I've been assuming that's what will happen.
If there are reasons to think it won't work, let's hear 'em now.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[Tom Lane]

Andrew Dunstan  writes:
> Yeah, what I've been thinking about in conjunction with similar problems 
> is some sort of type registry, so that we could code for non-builtin 
> types in certain cases. Maybe we should add that the the developers' 
> meeting agenda.

Maybe.  I don't want to see a json-specific hack for this, but some sort
of generic way to add type knowledge could be useful, if we could figure
out what we want.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] c-function variants running time

[Armando]

Hi everybody.

First of all I have to thank you for your wonderful job! PostgreSQL rocks!

I am writing you because I am interested in understanding some specifics related
to PostgreSQL internals. More precisely, I am investigating the running time
of the different function implementation approaches, which is part of my BSc
thesis.

Here is the thing: I have implemented as a proof of concept three functions,
which are
a) text[] arraypoc(text, int); this returns an array of the form 'text,int'
   where the text is the copy of the text passed as parameter and int is a
   simple counter. The number of tuples is specified by the integer parameter.
   I have estimated the running time in this manner:
   SELECT *
 FROM unnest(arraypoc('abcdefghilmnopqrstuvz',100));
   The estimated running time is after 10 executions is:
   (791.571 + 797.163 + 677.331 + 686.674 + 686.691 + 686.438 +
797.910 + 795.955 + 793.459 + 794.110)/10 = 750.7302

b) TABLE(text,int) srfpoc(text, int); is similar as the previous one but this
   is a set returning function, which returns a table of a similar shape as in
   the previous case. Instead of a string, I return a text and an integer. Again
   text is just the copy of the parameter and int is a counter.
   I have estimated the running time in this manner:
   SELECT *
 FROM srfpoc('abcdefghilmnopqrstuvz',100);
   The estimated running time is after 10 executions is: 
   (665.016 + 778.100 + 640.605 + 787.102 + 785.501 + 791.307 +
784.780 + 793.222 + 794.624 + 790.357)/10 = 761.0614

c) TABLE(text,int) srfmatpoc(text, int); this does the same as the previous one,
   but in this case I wrote a SRF_Materialized using the SPI interface.  I have
   estimated the running time in this manner:
   SELECT *
 FROM srfmatpoc('abcdefghilmnopqrstuvz',100);
   The estimated running time is after 10 executions is: 
   (747.095 + 703.894 + 762.310 + 763.299 + 764.582 + 760.991 + 763.427 +
764.033 + 731.292 + 770.895)/10   = 753.1818

I have executed all the tests on the same server. The functions are compiled
using the -O3 compilation parameter. I am using PostgreSQL 9.1.3.

I would have expected the version a) to be slower than b) and c) but it turns
out that it is actually the fastest (the difference is not so big anyway). What
am I doing wrong? What can I do to improve the functions? Have I misunderstood
something?

Attached you find the code of all three functions.

Thanks a lot!
Armando
#include "postgres.h"
#include 

#include "utils/array.h"
#include "utils/builtins.h"	/* text_to_cstring */
#include "catalog/pg_type.h"
#include "utils/lsyscache.h"
#include "fmgr.h"

#ifdef PG_MODULE_MAGIC
PG_MODULE_MAGIC;
#endif

PG_FUNCTION_INFO_V1(arraypoc);

Datum arraypoc(PG_FUNCTION_ARGS) {
  ArrayType   *result;
  Datum   *result_data = NULL;
  int16o_typlen;
  bool o_typbyval;
  char o_typalign;
  text*t,
  *cpyt;
  uint32   c;
  int i, t_len;

  t = (text *)PG_GETARG_TEXT_PP(0);
  c = (uint32)PG_GETARG_INT32(1);

  if(c>0) result_data = (Datum *)palloc(sizeof(Datum) * c);

  for(i=0; i#include "postgres.h"

#include /* snprintf */
#include /* malloc, free */
#include /* strncpy, strlen */
#include   /* for uint32 and size_t*/

#include "catalog/pg_type.h"
#include "utils/builtins.h"  /* text_to_cstring */
#include "funcapi.h"
#include "fmgr.h"

#ifdef PG_MODULE_MAGIC
PG_MODULE_MAGIC;
#endif

PG_FUNCTION_INFO_V1(srfpoc);

Datum srfpoc(PG_FUNCTION_ARGS) {
  FuncCallContext *funcctx;
  TupleDesctupdesc;
  text*t;
  int  max_calls,
   call_cntr;

  t = (text *)PG_GETARG_TEXT_PP(0);
  
  if (SRF_IS_FIRSTCALL()) {
MemoryContextoldcontext;

funcctx = SRF_FIRSTCALL_INIT();
oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);

funcctx->max_calls = (uint32)PG_GETARG_INT32(1);

tupdesc = CreateTemplateTupleDesc(2, false);
TupleDescInitEntry(tupdesc, (AttrNumber) 1, "t", TEXTOID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 2, "count", INT4OID, -1, 0);

funcctx->user_fctx = BlessTupleDesc(tupdesc);

MemoryContextSwitchTo(oldcontext);
  }

  funcctx = SRF_PERCALL_SETUP();
  max_calls = funcctx->max_calls;
  call_cntr = funcctx->call_cntr;
  tupdesc = funcctx->user_fctx;

  if(call_cntr < max_calls && t != NULL) {
bool nulls[2];
Datumvalues[2];
HeapTupletuple;
text*cpyt;

cpyt = (text *)palloc(VARHDRSZ + VARSIZE_ANY_EXHDR(t));
SET_VARSIZE(cpyt, VARHDRSZ + VARSIZE_ANY_EXHDR(t));
memcpy((void *)VARDATA(cpyt), (void *)VARDATA(t), VARSIZE_ANY_EXHDR(t));

values[0] = PointerGetDatum(cpyt);
values[1] = Int32GetDatum(call_cntr);
nulls[0] = false;
nulls[1] = false;

tuple = heap_form_tuple(tupdesc, values, nulls);
SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum

Re: [HACKERS] Beta time?

[Josh Berkus]

>> Next week, I thought.
> 
> How are we handling the Monday release with everyone at PGCon?  Was that
> resolved?

I have yet to see a confirmed date, guys.  If we expect any support from
the packagers and/or the advocacy volunteers, then people need at least
a week's notice, probably more.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[Andrew Dunstan]

On 05/04/2012 09:52 AM, Tom Lane wrote:

Hannu Krosing  writes:

On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:

So given that do we do anything about this now, or wait till 9.3?

I'd like the json support in 9.2 updated as follows

I think it's too late to be entertaining proposals for such changes in
9.2.  If we had concluded that the existing functions were actively
wrong or a bad idea, then of course we'd need to do something; but they
are not, so we can just as well consider additions in the 9.3 cycle
rather than now.  I am not convinced that this proposal is fully baked
yet, anyway; not to mention that right now we need to have our heads
down on resolving the remaining open issues, not designing,
implementing, and reviewing a pile of brand new code for json.



Yeah, that was my feeling. We usually take a release or two to get 
things right, fill in what's missing, etc. and I don't think this will 
be ant different.






By allowing developers just to define their own to_json(date) function
we give them the power do decide which one to use. And if we honour
search_path when looking up the to_json() functions, then they can even
choose to have different conventions for different applications.

This is not going to work anywhere near as nicely as you think.  If
somebody tries to define multiple to_json() functions that override a
generic to_json(anyelement) one, he will start getting "function is not
unique" parse failures.  The parser will only successfully decide which
function to call when the input data type exactly matches one of the
specialized functions, which means you might as well not have the
generic one at all.




Yeah, what I've been thinking about in conjunction with similar problems 
is some sort of type registry, so that we could code for non-builtin 
types in certain cases. Maybe we should add that the the developers' 
meeting agenda.


cheers

andrew



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Future In-Core Replication

[Robert Haas]

On Fri, May 4, 2012 at 11:06 AM, Greg Smith  wrote:
> The straw man argument here would require 100% transparency on everything
> you do in regards to PostgreSQL and related software.  Before doing any
> development on any code, first post here to ask for design review.  And if
> someone asks you to work on a program that isn't open source from day one,
> refuse unless you can operate that transparently.

Well, look.  At the end of the day, I don't really care whether you
post your designs before writing code or not - unless it turns out
that we get to the end of the development cycle, a gigantic patch
shows up at the last minute, it gets rejected because people aren't
satisfied with the design, and then massive bitching ensues because
the author(s) put a lot of work into that patch.  Then I care, because
now the fact that no design consensus was sought at the outset has
been transformed into a defect in the community process, which does in
fact have defects, but that isn't one of them.  We all know that
design review is going to have to happen at some point, and if there's
not an adequate opportunity to do that before the code is written then
it will happen after the code is written.  If that means the code has
to be thrown out, then that's the risk you take by writing the code
first.  As long as everybody understands that, do it in whatever order
you like.

I think the real straw man here is the idea that it will somehow save
time to skip the design phase and start writing code.  I have never
worked on a project, open source or otherwise, where that was true,
and I believe that any textbook on software engineering you pick up is
likely to tell you that in fact exactly the opposite is the case.

Obviously, there are times when you need to write some throw-away code
just to see how things shake out, and I do that all the time, and it
makes complete sense, and I'm not knocking it.  But if any of that
code makes it into the committed patch, I count that as unusually
lucky.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Future In-Core Replication

[Greg Smith]

On 05/04/2012 09:03 AM, Robert Haas wrote:

I try pretty hard not to go off and do large amounts of work in a
vacuum.  If something is more than a couple days work, I post the
design on hackers and wait for feedback before writing a line of code.


That is an excellent luxury to have.  You've worked very hard to earn 
it.  Not everyone is in the position where that's possible though.



Are you requesting more transparency in general, asking for my
thoughts on logical replication specifically, or something else?


The straw man argument here would require 100% transparency on 
everything you do in regards to PostgreSQL and related software.  Before 
doing any development on any code, first post here to ask for design 
review.  And if someone asks you to work on a program that isn't open 
source from day one, refuse unless you can operate that transparently.


That standard is nice if you can pull it off.  But I don't give you a 
hard time if you have to make some compromises from that ideal to keep 
yourself gainfully employed.  You do a ton of good work for the 
PostgreSQL community in a transparent way, so I assume that you're doing 
the best you can.  I would like to see that assumption presumed on our 
side, too.


Here are the individual straw men in this area I'd like to see put out 
of their misery:


"You're developing things in secret":  if that's the case, we're pretty 
bad at it, given the history I outlined at 
http://archives.postgresql.org/message-id/4f9b1b6c.5010...@2ndquadrant.com


"That discussion didn't happen in the right place":  it's not our fault 
that the cluster-hackers list exists.  Go joust at getting that list 
shut down and their meeting during PGCon canceled if you think it's 
unproductive for discussions to happen there.  I've been trying to 
bridge that gap for over two years now; note how many times I appear in 
the edit history at 
http://wiki.postgresql.org/index.php?title=ClusterFeatures&action=history


"You might do too much development in the wrong direction and not build 
the right thing":  and?  Yes, there are people who develop into a corner 
and end up doing unproductive work as a result.  And there are others 
who submit things and give up when faced with feedback on them.  Last 
time I checked, there wasn't anyone who flat-out rejects on-list 
feedback working for 2ndQuadrant.   Instead, I see features that go 
through extensive and numerous review cycles based on what we hear back.


"Designs should be presented on-list before doing any development": 
this is not always practical for those of us who are doing feature 
development.  Some feature sponsors are still getting used to open 
development.  If we have a private development milestone date to hit *in 
order to get more funding for public PostgreSQL work*, which is often 
the case here, we try not to miss it.  We'd be bad community members to 
do so.  And sometimes that involves building a proof of concept or 
prototype here first, then submitting it to the community once it's 
moved onto being a proven concept.  Since the community has a clear set 
of guidelines for how and when to submit new features, we make sure the 
development plans line up with them.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Advisory locks seem rather broken

[Robert Haas]

On Fri, May 4, 2012 at 10:21 AM, Tom Lane  wrote:
> Robert Haas  writes:
>> Originally, I thought that the patch should include some kind of
>> accounting mechanism to prevent that from happening, where we'd keep
>> track of the number of fast-path locks that were outstanding and make
>> sure to keep that many slots free in the main lock table, but Noah
>> talked me out of it, on theory that (1) it was very unlikely to occur
>> in practice and (2) if it did occur, then you probably need to bump up
>> max_locks_per_transaction anyway and (3) it amounted to forcing
>> failures in cases where that might not be strictly necessary, which is
>> usually not a great thing to do.
>
> I agree with that, as long as we can be sure that the system behaves
> sanely (doesn't leave the data structures in a corrupt state) when an
> out-of-memory condition does occur.

OK.  I believe that commit 53c5b869b464d567c3b8f617201b49a395f437ab
robustified this code path quite a bit; but it is certainly possible
that there are remaining oversights, and I would certainly appreciate
any further review you have time to do.  Basically, it seems like the
likely failure modes, if there are further bugs, would be either (1)
failing to track the strong lock counts properly, leading to
performance degradation if the counters become permanently stuck at a
value other than zero even after all the locks are gone or (2) somehow
muffing the migration of a lock from the fast-path mechanism to the
regular mechanism.

When taking a strong lock, the idea is that the strong locker first
bumps the strong lock count.  That bump must be unwound if we fail to
acquire the lock, which means it has to be cleaned up in the error
path and any case where we give up (e.g. conditional acquire of a
contended lock).  Next, we iterate through all the backend slots and
transfer fast path locks for each one individually.  If we fail midway
through, the strong locker must simply make sure to unwind the strong
lock count.  The weak lockers whose locks got transferred are fine:
they need to know how to cope with releasing a transferred lock
anyway; whether the backend that did the transfer subsequently blew up
is not something they have any need to care about.  Once all the
transfers are complete, the strong locker queues for the lock using
the main mechanism, which now includes all possible conflicting locks.
 Again, if we blow up while waiting for the lock, the only extra thing
we need to do is unwind the strong lock count acquisition.

Of course, I may be missing some other kind of bug altogether...

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Advisory locks seem rather broken

[Tom Lane]

Robert Haas  writes:
> Originally, I thought that the patch should include some kind of
> accounting mechanism to prevent that from happening, where we'd keep
> track of the number of fast-path locks that were outstanding and make
> sure to keep that many slots free in the main lock table, but Noah
> talked me out of it, on theory that (1) it was very unlikely to occur
> in practice and (2) if it did occur, then you probably need to bump up
> max_locks_per_transaction anyway and (3) it amounted to forcing
> failures in cases where that might not be strictly necessary, which is
> usually not a great thing to do.

I agree with that, as long as we can be sure that the system behaves
sanely (doesn't leave the data structures in a corrupt state) when an
out-of-memory condition does occur.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Advisory locks seem rather broken

[Robert Haas]

On Fri, May 4, 2012 at 9:25 AM, Tom Lane  wrote:
> Robert Haas  writes:
>> In 9.1, we just did this:
>
>>                 if (locallock->proclock == NULL || locallock->lock == NULL)
>>                 {
>>                         /*
>>                          * We must've run out of shared memory while
>> trying to set up this
>>                          * lock.  Just forget the local entry.
>>                          */
>>                         Assert(locallock->nLocks == 0);
>>                         RemoveLocalLock(locallock);
>>                         continue;
>>                 }
>
>> ...and I just shoved the new logic into that stanza without thinking
>> hard enough about what order to do things in.
>
> Right.  The other thing that was bothering me about that was that it's
> not clear now how to tell a broken locallock entry (which is what this
> logic originally intended to clean up) from a fastpath one.  Perhaps
> it'd be a good idea to add a "valid" flag?

Well, I think nLocks == 0 should serve that purpose adequately.

> And while I'm wondering
> about such things, what happens when it's necessary to convert a
> fastpath lock to a regular one, but there's no room in shared memory
> for more lock objects?

Then you error out.  Of course, if the fast path mechanism didn't
exist at all, you would have started erroring out much sooner.  Now,
there is some rub here, because the mechanism isn't "fair": strong
lockers will error out instead of weak lockers, and in the worst case
where the lock table remains perpetually on the edge of overflowing,
strong lock requests could be fail repeatedly, essentially a DOS.
Originally, I thought that the patch should include some kind of
accounting mechanism to prevent that from happening, where we'd keep
track of the number of fast-path locks that were outstanding and make
sure to keep that many slots free in the main lock table, but Noah
talked me out of it, on theory that (1) it was very unlikely to occur
in practice and (2) if it did occur, then you probably need to bump up
max_locks_per_transaction anyway and (3) it amounted to forcing
failures in cases where that might not be strictly necessary, which is
usually not a great thing to do.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CLOG extension

[Robert Haas]

On Fri, May 4, 2012 at 9:11 AM, Simon Riggs  wrote:
> On 4 May 2012 13:59, Robert Haas  wrote:
>> On Fri, May 4, 2012 at 3:35 AM, Simon Riggs  wrote:
>>> On Thu, May 3, 2012 at 9:56 PM, Robert Haas  wrote:
 On Thu, May 3, 2012 at 3:20 PM, Simon Riggs  wrote:
> Your two paragraphs have roughly opposite arguments...
>
> Doing it every 32 pages would give you 30 seconds to complete the
> fsync, if you kicked it off when half way through the previous file -
> at current maximum rates. So there is utility in doing it in larger
> chunks.

 Maybe, but I'd like to try changing one thing at a time.  If we change
 too much at once, it's likely to be hard to figure out where the
 improvement is coming from.  Moving the task to a background process
 is one improvement; doing it in larger chunks is another.  Those
 deserve independent testing.
>>>
>>> You gave a good argument why background pre-allocation wouldn't work
>>> very well if we do it a page at a time. I believe you.
>>
>> Your confidence is sort of gratifying, but in this case I believe it's
>> misplaced.  On more careful analysis, it seems that ExtendCLOG() does
>> just two things: (1) evict a CLOG buffer and replace it with a zero'd
>> page representing the new page and (2)  write an XLOG record for the
>> change.  Apparently, "extending" CLOG doesn't actually involve
>> extending anything on disk at all.  We rely on the future buffer
>> eviction to do that, which is surprisingly different from the way
>> relation extension is handled.
>>
>> So CLOG extension is normally fast, but occasionally something goes
>> wrong.
>
> I don't agree its normally fast.
>
> WALInsert contention is high, so there is usually a long queue. As
> we've discussed this can be done offline and and so (2) can completely
> avoided in the main line. Considering that all new xids wait for this
> action, any wait at all is bad and takes time to drain once it clears.
>
> Evicting a clog has cost because the tail is almost always dirty when
> we switch pages.
>
> Doing both of those will ensure switch to new page requires zero wait time.
>
> So you have the solution. Not sure what else you're looking for.

Nothing, really.  I was just mooting some ideas before I went and
started coding, to see what people thought.  I've got your opinion and
Tom's, and of course my own, so now I'm off to test some different
approaches.  At the moment I'm running a battery of tests on
background-writing CLOG, which I will post about when they are
complete, and I intend to play around with some of the ideas from this
thread as well.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[Tom Lane]

Hannu Krosing  writes:
> On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
>> So given that do we do anything about this now, or wait till 9.3?

> I'd like the json support in 9.2 updated as follows

I think it's too late to be entertaining proposals for such changes in
9.2.  If we had concluded that the existing functions were actively
wrong or a bad idea, then of course we'd need to do something; but they
are not, so we can just as well consider additions in the 9.3 cycle
rather than now.  I am not convinced that this proposal is fully baked
yet, anyway; not to mention that right now we need to have our heads
down on resolving the remaining open issues, not designing,
implementing, and reviewing a pile of brand new code for json.

> By allowing developers just to define their own to_json(date) function
> we give them the power do decide which one to use. And if we honour
> search_path when looking up the to_json() functions, then they can even
> choose to have different conventions for different applications.

This is not going to work anywhere near as nicely as you think.  If
somebody tries to define multiple to_json() functions that override a
generic to_json(anyelement) one, he will start getting "function is not
unique" parse failures.  The parser will only successfully decide which
function to call when the input data type exactly matches one of the
specialized functions, which means you might as well not have the
generic one at all.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Advisory locks seem rather broken

[Tom Lane]

Robert Haas  writes:
> In 9.1, we just did this:

> if (locallock->proclock == NULL || locallock->lock == NULL)
> {
> /*
>  * We must've run out of shared memory while
> trying to set up this
>  * lock.  Just forget the local entry.
>  */
> Assert(locallock->nLocks == 0);
> RemoveLocalLock(locallock);
> continue;
> }

> ...and I just shoved the new logic into that stanza without thinking
> hard enough about what order to do things in.

Right.  The other thing that was bothering me about that was that it's
not clear now how to tell a broken locallock entry (which is what this
logic originally intended to clean up) from a fastpath one.  Perhaps
it'd be a good idea to add a "valid" flag?  And while I'm wondering
about such things, what happens when it's necessary to convert a
fastpath lock to a regular one, but there's no room in shared memory
for more lock objects?

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Advisory locks seem rather broken

[Robert Haas]

On Thu, May 3, 2012 at 5:18 PM, Tom Lane  wrote:
> ... btw, it appears to me that the "fast path" patch has broken things
> rather badly in LockReleaseAll.  AFAICS it's not honoring either the
> lockmethodid restriction nor the allLocks restriction with respect to
> fastpath locks.  Perhaps user locks and session locks are never taken
> fast path, but still it would be better to be making those checks
> further up, no?

User locks are never taken fast path, but session locks can be, so I
think you're right that there is a bug here.  I think what we should
probably do is put the nLocks == 0 test before the lockmethodid and
allLocks checks, and then the fast path stuff after those two checks.

In 9.1, we just did this:

if (locallock->proclock == NULL || locallock->lock == NULL)
{
/*
 * We must've run out of shared memory while
trying to set up this
 * lock.  Just forget the local entry.
 */
Assert(locallock->nLocks == 0);
RemoveLocalLock(locallock);
continue;
}

...and I just shoved the new logic into that stanza without thinking
hard enough about what order to do things in.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Future In-Core Replication

[Simon Riggs]

On 4 May 2012 14:01, Robert Haas  wrote:
> On Fri, May 4, 2012 at 8:32 AM, Hannu Krosing  wrote:
>> For logical we don't really need to uniquely identify such rows - it
>> should sufficient if we just update exactly one of the matching rows.
>>
>> The way to do this is to put all fields of the OLD.* tuple in the WHERE
>> clause and then update just one matching row.
>>
>> IIRC updating (or deleting) CURRENT OF a cursor is currently supported
>> only in pl/pgsql so this needs to be done using a plpgsql cursor.
>>
>> If the table has no indexes or index lookup returns lots of rows, then
>> this is bound to be slow, but in this case it was probably slow on
>> master too :)
>
> I was about to write a reply saying exactly this, but you said it
> better than I would have been able to manage.
>
> I think this is all exactly right.

Yes, but its not a high priority for inclusion. Many things like this
will need to wait behind the really critical additional features.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CLOG extension

[Simon Riggs]

On 4 May 2012 13:59, Robert Haas  wrote:
> On Fri, May 4, 2012 at 3:35 AM, Simon Riggs  wrote:
>> On Thu, May 3, 2012 at 9:56 PM, Robert Haas  wrote:
>>> On Thu, May 3, 2012 at 3:20 PM, Simon Riggs  wrote:
 Your two paragraphs have roughly opposite arguments...

 Doing it every 32 pages would give you 30 seconds to complete the
 fsync, if you kicked it off when half way through the previous file -
 at current maximum rates. So there is utility in doing it in larger
 chunks.
>>>
>>> Maybe, but I'd like to try changing one thing at a time.  If we change
>>> too much at once, it's likely to be hard to figure out where the
>>> improvement is coming from.  Moving the task to a background process
>>> is one improvement; doing it in larger chunks is another.  Those
>>> deserve independent testing.
>>
>> You gave a good argument why background pre-allocation wouldn't work
>> very well if we do it a page at a time. I believe you.
>
> Your confidence is sort of gratifying, but in this case I believe it's
> misplaced.  On more careful analysis, it seems that ExtendCLOG() does
> just two things: (1) evict a CLOG buffer and replace it with a zero'd
> page representing the new page and (2)  write an XLOG record for the
> change.  Apparently, "extending" CLOG doesn't actually involve
> extending anything on disk at all.  We rely on the future buffer
> eviction to do that, which is surprisingly different from the way
> relation extension is handled.
>
> So CLOG extension is normally fast, but occasionally something goes
> wrong.

I don't agree its normally fast.

WALInsert contention is high, so there is usually a long queue. As
we've discussed this can be done offline and and so (2) can completely
avoided in the main line. Considering that all new xids wait for this
action, any wait at all is bad and takes time to drain once it clears.

Evicting a clog has cost because the tail is almost always dirty when
we switch pages.

Doing both of those will ensure switch to new page requires zero wait time.

So you have the solution. Not sure what else you're looking for.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Future In-Core Replication

[Robert Haas]

On Thu, May 3, 2012 at 8:22 PM, Greg Smith  wrote:
> On 05/01/2012 09:09 AM, Robert Haas wrote:
>>
>> I think we ought to be sharing and debugging designs in
>> public, not internally within 2ndQuadrant - or any other company, or
>> any other mailing list other than this one.
>
> OK.  You go first.

Are you requesting more transparency in general, asking for my
thoughts on logical replication specifically, or something else?

I try pretty hard not to go off and do large amounts of work in a
vacuum.  If something is more than a couple days work, I post the
design on hackers and wait for feedback before writing a line of code.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Future In-Core Replication

[Robert Haas]

On Fri, May 4, 2012 at 8:32 AM, Hannu Krosing  wrote:
> For logical we don't really need to uniquely identify such rows - it
> should sufficient if we just update exactly one of the matching rows.
>
> The way to do this is to put all fields of the OLD.* tuple in the WHERE
> clause and then update just one matching row.
>
> IIRC updating (or deleting) CURRENT OF a cursor is currently supported
> only in pl/pgsql so this needs to be done using a plpgsql cursor.
>
> If the table has no indexes or index lookup returns lots of rows, then
> this is bound to be slow, but in this case it was probably slow on
> master too :)

I was about to write a reply saying exactly this, but you said it
better than I would have been able to manage.

I think this is all exactly right.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CLOG extension

[Robert Haas]

On Fri, May 4, 2012 at 3:35 AM, Simon Riggs  wrote:
> On Thu, May 3, 2012 at 9:56 PM, Robert Haas  wrote:
>> On Thu, May 3, 2012 at 3:20 PM, Simon Riggs  wrote:
>>> Your two paragraphs have roughly opposite arguments...
>>>
>>> Doing it every 32 pages would give you 30 seconds to complete the
>>> fsync, if you kicked it off when half way through the previous file -
>>> at current maximum rates. So there is utility in doing it in larger
>>> chunks.
>>
>> Maybe, but I'd like to try changing one thing at a time.  If we change
>> too much at once, it's likely to be hard to figure out where the
>> improvement is coming from.  Moving the task to a background process
>> is one improvement; doing it in larger chunks is another.  Those
>> deserve independent testing.
>
> You gave a good argument why background pre-allocation wouldn't work
> very well if we do it a page at a time. I believe you.

Your confidence is sort of gratifying, but in this case I believe it's
misplaced.  On more careful analysis, it seems that ExtendCLOG() does
just two things: (1) evict a CLOG buffer and replace it with a zero'd
page representing the new page and (2)  write an XLOG record for the
change.  Apparently, "extending" CLOG doesn't actually involve
extending anything on disk at all.  We rely on the future buffer
eviction to do that, which is surprisingly different from the way
relation extension is handled.

So CLOG extension is normally fast, but occasionally something goes
wrong.  So far I see two ways that can happen: (1) the WAL insertion
stalls because wal_buffers are full, and we're forced to wait for WAL
to be written (and perhaps fsync'd, since both are covered by the same
lock) or (2) the page we choose to evict happens to be dirty, and we
have to write+fsync it before repurposing it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Future In-Core Replication

[Hannu Krosing]

On Thu, 2012-05-03 at 00:58 -0500, Jim Nasby wrote:
> On 4/29/12 6:03 AM, Simon Riggs wrote:
> >> The DML-WITH-LIMIT-1 is required to do single logical updates on tables
> >> >  with non-unique rows.
> >> >  And as for any logical updates we will have huge performance problem
> >> >  when doing UPDATE or DELETE on large table with no indexes, but
> >> >  fortunately this problem is on slave, not master;)
> > While that is possible, I would favour the do-nothing approach. By
> > making the default replication mode = none, we then require a PK to be
> > assigned before allowing replication mode = on for a table. Trying to
> > replicate tables without PKs is a problem that can wait basically.
> >
> 
> Something that a in-core method might be able to do that an external one 
> can't would be to support a method of uniquely identifying rows in tables
> with no PK's. A gross example (that undoubtedly wouldn't work in the real 
> world) would be using TID's. 
> A real-world implementation might be based on a hidden serial column.

For logical we don't really need to uniquely identify such rows - it
should sufficient if we just update exactly one of the matching rows.

The way to do this is to put all fields of the OLD.* tuple in the WHERE
clause and then update just one matching row.

IIRC updating (or deleting) CURRENT OF a cursor is currently supported
only in pl/pgsql so this needs to be done using a plpgsql cursor.

If the table has no indexes or index lookup returns lots of rows, then
this is bound to be slow, but in this case it was probably slow on
master too :)

> -- 
> Jim C. Nasby, Database Architect   j...@nasby.net
> 512.569.9461 (cell) http://jim.nasby.net
> 



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

[Hannu Krosing]

On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
> 
> 
> On Wed, May 2, 2012 at 2:29 AM, Hannu Krosing 
> wrote:
> 
> 
> 
> I don't object to row_to_json() and array_to_json() functions
> being
> there as a convenience and as the two "official" functions
> guaranteed to
> return "JSON text".
> 
> 
> So given that do we do anything about this now, or wait till 9.3?

Sorry for missing this mail, followed this only on list
I hope it is ok to CC this back to list



I'd like the json support in 9.2 updated as follows


Generic to_json(...) returning a "JSON value"
=

we should have a generic to_json(...) both for eas and use and for easy
extensibility, as explained below.

to_json(...) should work for all types, returning a "json value" similar
to what current json_in does, but for all types, not just cstring.

We could keep row_to_json() and array_to_json() as official json-text
returning functions 


Configurable and extensible to_json()
==

When working on structured types, always the first try for getting an
element-as-json should be running to_json(element) and only if this
fails fall back to current "use text representation" code.

this enables two important things

1) configurable json-coding of values not explicitly supported by
standard

You can read about an attempt to standardise json-date formart here

http://weblogs.asp.net/bleroy/archive/2008/01/18/dates-and-json.aspx .

By allowing developers just to define their own to_json(date) function
we give them the power do decide which one to use. And if we honour
search_path when looking up the to_json() functions, then they can even
choose to have different conventions for different applications.

2) flexibility in adding support for extension types, like representing
hstore as object/dict by just providing the to_json(hstore, ...)
functions in hstore extension

Pretty-printing
===

If we were to support prettyprinting of anything more complex than
single level structs (record or array), then we need to pass "ident"
into the to_json() function

my recommendation would be to have the signature 

to_json(datum any, ident int)

with ident = NULL meaning no prettyprint , ident =0 meaninf top level,
or "starting at left margin" and anything else meaning the amount of
spaces needed to be added to the beginning of all rows exept the first
one, for example the query 

hannu=# select array_to_json(array(select test from test limit 2),true);
array_to_json
-
 [{"id":9,"data":"testdata","tstamp":"2012-05-01 09:44:50.175189"}, +
  {"id":10,"data":"testdata","tstamp":"2012-05-01 09:45:50.260276"}]
(1 row)

could return this:

[{'data': 'testdata',
  'id': 9,
  'tstamp': '2012-05-01 09:44:50.175189'},
 {'data': 'testdata',
  'id': 10,
  'tstamp': '2012-05-01 09:45:50.260276'}]

if it would call to_json(row, 1) for getting each row prettyprinted with
ident 1


Getting a record _from_ json()
==

JSON support would be much more useful if we supported the function of
converting the other way as well, that is from json to record

The best model easy to support seems to be what Merlin suggested, that
is the populate_record(record,hstore) function and corresponding #=
operator (see: http://www.postgresql.org/docs/9.1/static/hstore.html)



The complete plan for 9.2 once more
===

What is needed to nicely round up a  simple and powerful json type is

1. the "json" TYPE

   an agreement tha json type represents a "JSON value" (this is what 
   json_in currently does)


2. json output

2.1 doing the right thing with internal values

   to_json() function for converting to this "JSON value" for any type.
   default implementation for 'non-json' types returns their 
   postgresql textual representation in double quotes 
(date -> "2012-05-01 09:45:50.260276"

   structured types use to_json() for getting values internally, 
   so that by defining to_json(hstore) you can automatically get 
   hstore represented in javascript object or dictionary representation

   hannu=# select row_to_json(r) 
   from (select 1::int id, '"foo"=>1, "bar"=>"baz"'::hstore)r;

   should not return this:

row_to_json  
   --
{"id":1,"hstore":"\"bar\"=>\"baz\", \"foo\"=>\"1\""}
   (1 row)

   but this
row_to_json  
   --
{"id":1,"hstore":{"bar": "baz", "foo":"1"}}
   (1 row)

2.1 getting the pretty-printing right for structured types 

   to_json(any, indent) functions for getting recursive prettyprinting.
   we might also need another argument

Re: [HACKERS] index-only scans vs. Hot Standby, round two

[Simon Riggs]

On 2 May 2012 13:41, Robert Haas  wrote:

> So on further reflection I'm thinking it may be best just to stick
> with a hard conflict for now and see what feedback we get from beta
> testers.

Which is what I was expecting y'all to conclude once you'd looked at
the task in more detail.

And I'm happy with the concept of beta being a period where we listen
to feedback, not just bugs, and decide whether further refinements are
required.

What I think is possible is to alter the conflict as it hits the
backend. If a backend has enable_indexonly = off then it wouldn't be
affected by those conflicts anyhow. So if we simply record whether we
are using an index only plan then we'll know whether to ignore it or
abort. I think we can handle that by marking the snapshot at executor
startup time. Needs a few other pushups but not that hard.

The likelihood is that SQL that uses index only won't run for long
enough to be cancelled anyway.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CLOG extension

[Simon Riggs]

On Thu, May 3, 2012 at 9:56 PM, Robert Haas  wrote:
> On Thu, May 3, 2012 at 3:20 PM, Simon Riggs  wrote:
>> Your two paragraphs have roughly opposite arguments...
>>
>> Doing it every 32 pages would give you 30 seconds to complete the
>> fsync, if you kicked it off when half way through the previous file -
>> at current maximum rates. So there is utility in doing it in larger
>> chunks.
>
> Maybe, but I'd like to try changing one thing at a time.  If we change
> too much at once, it's likely to be hard to figure out where the
> improvement is coming from.  Moving the task to a background process
> is one improvement; doing it in larger chunks is another.  Those
> deserve independent testing.

You gave a good argument why background pre-allocation wouldn't work
very well if we do it a page at a time. I believe you.

If we do it a file at a time, we can just write the file without
calling it in page by page into the SLRU, as long as we write the WAL
file for it first then we don't need to fsync either of them.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers