date:20141212

On Fri, Dec 12, 2014 at 08:57:55PM -0500, Tom Lane wrote:
> Peter Eisentraut  writes:
> > On 12/12/14 10:16 AM, Tom Lane wrote:
> >> I think pg_upgrade should continue to have SQL scripts that create and
> >> delete the SQL function definitions for these.
> 
> > That won't actually work very easily.  LANGUAGE internal functions need
> > to be in fmgr_builtins, and the only way to get them there is by listing
> > them in pg_proc.h.  (We could drop the functions in initdb, but seems
> > kind of silly.)
> 
> Oh, good point.
> 
> > The functions do already check themselves that they are called in binary
> > upgrade mode, so exposing them in pg_proc doesn't seem risky.
> 
> Fair enough ... binary upgrade mode is not readily accessible, right?

Well, the postmaster allows anyone to use the flag, while the backends
have:

case 'b':
/* Undocumented flag used for binary upgrades */
if (secure)
IsBinaryUpgrade = true;
break;

which means it can only be passed in from the postmaster.  I think only
the super-user can set postmaster options.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + Everyone has their own god. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] On partitioning

El 12/12/2014 23:09, "Alvaro Herrera"  escribió:
>
> Claudio Freire wrote:
>
> > Fair enough, but that's not the same as not requiring easy proofs. The
> > planner might not the one doing the proofs, but you still need proofs.
> >
> > Even if the proving method is hardcoded into the partitioning method,
> > as in the case of list or range partitioning, it's still a proof. With
> > arbitrary functions (which is what prompted me to mention proofs) you
> > can't do that. A function works very well for inserting, but not for
> > selecting.
> >
> > I could be wrong though. Maybe there's a way to turn SQL functions
> > into analyzable things? But it would still be very easy to shoot
> > yourself in the foot by writing one that is too complex.
>
> Arbitrary SQL expressions (including functions) are not the thing to use
> for partitioning -- at least that's how I understand this whole
> discussion.  I don't think you want to do "proofs" as such -- they are
> expensive.
>
> To make this discussion a bit clearer, there are two things to
> distinguish: one is routing tuples, when an INSERT or COPY command
> references the partitioned table, into the individual partitions
> (ingress); the other is deciding which partitions to read when a SELECT
> query wants to read tuples from the partitioned table (egress).
>
> On ingress, what you want is something like being able to do something
> on the tuple that tells you which partition it belongs into.  Ideally
> this is something much lighter than running an expression; if you can
> just apply an operator to the partitioning column values, that should be
> plenty fast.  This requires no proof.
>
> On egress you need some direct way to compare the scan quals with the
> partitioning values.  I would imagine this to be similar to how scan
> quals are compared to the values stored in a BRIN index: each scan qual
> has a corresponding operator strategy and a scan key, and you can say
> "aye" or "nay" based on a small set of operations that can be run
> cheaply, again without any proof or running arbitrary expressions.

Interesting that you mention BRIN. It does seem that it could be made to
work with BRIN's operator classes.

In fact, a partition-wide brin tuple could be stored per partition and that
in itself could be the definition for the partition.

Either preinitialized or dynamically updated. Would work even for arbitrary
routing functions, especially if the operator class to use is customizable.

I stand corrected.

Re: [HACKERS] moving from contrib to bin

On 12/12/14 11:14 AM, Robert Haas wrote:
> contrib is a nice middle-ground.

I respect that view.  But if we consider contrib a place where things
can try to mature, then there must be a way to move things out of there
once they have matured.  If it turns out that moving things out of
contrib causes major problems, then we must reconsider that system and
find a different process for maturing things (e.g., a makefile flag,
just a random idea).  Therefore, I think it's worth trying this.  We can
do it piece by piece.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On 12/12/14 11:45 AM, Andres Freund wrote:
> On 2014-12-12 11:42:57 -0500, Robert Haas wrote:
>>> I think the amount of effort a simple renamed directory which wholly
>>> contains a binary creates is acceptable. Just use patch -p4 instead of
>>> patch -p1...
>>
>> That is fine if you are manually applying a patch that touches only
>> that directory, but if the patch also touches other stuff then it's
>> not as simple.
> 
> I think backpatchable commits that touch individual binaries and other
> code at the same time are (and ought to be!) pretty rare.
> 
>> And I don't know how well git cherry-pick will follow
>> the moves.
> 
> Not well if the patch is done in master first.

It does work.  I have tried it with the changes I'm working on.

There might very well be limits to the cleverness of these tools, but if
you know of a fundamental reason why this would be a problem, let's
please hear it.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] On partitioning

Claudio Freire wrote:

> Fair enough, but that's not the same as not requiring easy proofs. The
> planner might not the one doing the proofs, but you still need proofs.
> 
> Even if the proving method is hardcoded into the partitioning method,
> as in the case of list or range partitioning, it's still a proof. With
> arbitrary functions (which is what prompted me to mention proofs) you
> can't do that. A function works very well for inserting, but not for
> selecting.
> 
> I could be wrong though. Maybe there's a way to turn SQL functions
> into analyzable things? But it would still be very easy to shoot
> yourself in the foot by writing one that is too complex.

Arbitrary SQL expressions (including functions) are not the thing to use
for partitioning -- at least that's how I understand this whole
discussion.  I don't think you want to do "proofs" as such -- they are
expensive.

To make this discussion a bit clearer, there are two things to
distinguish: one is routing tuples, when an INSERT or COPY command
references the partitioned table, into the individual partitions
(ingress); the other is deciding which partitions to read when a SELECT
query wants to read tuples from the partitioned table (egress).

On ingress, what you want is something like being able to do something
on the tuple that tells you which partition it belongs into.  Ideally
this is something much lighter than running an expression; if you can
just apply an operator to the partitioning column values, that should be
plenty fast.  This requires no proof.

On egress you need some direct way to compare the scan quals with the
partitioning values.  I would imagine this to be similar to how scan
quals are compared to the values stored in a BRIN index: each scan qual
has a corresponding operator strategy and a scan key, and you can say
"aye" or "nay" based on a small set of operations that can be run
cheaply, again without any proof or running arbitrary expressions.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

Peter Eisentraut  writes:
> On 12/12/14 10:16 AM, Tom Lane wrote:
>> I think pg_upgrade should continue to have SQL scripts that create and
>> delete the SQL function definitions for these.

> That won't actually work very easily.  LANGUAGE internal functions need
> to be in fmgr_builtins, and the only way to get them there is by listing
> them in pg_proc.h.  (We could drop the functions in initdb, but seems
> kind of silly.)

Oh, good point.

> The functions do already check themselves that they are called in binary
> upgrade mode, so exposing them in pg_proc doesn't seem risky.

Fair enough ... binary upgrade mode is not readily accessible, right?

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On 12/12/14 3:20 PM, Tom Lane wrote:
> Pardon me for not knowing much about Debian packages, but how would
> that work exactly?  Is it possible to do make install-client, then
> package the installed files, then rm -rf the install tree, then
> repeat for install-server and install-contrib?  In the RPM world
> this would never work because the build/install step happens in
> toto before the packaging step.

I don't know exactly what Christoph had in mind, but the short answer to
your question is: Yes, that is possible.  Almost anything is possible.
There are many tools available at various levels of abstraction that
facility common patterns, but you can also do everything by hand, as
long as you produce the right *.deb files at the right location at the
end.  The only thing stopping you is your sanity.

However, other packaging systems clearly don't work that way, and so
designing something that would only work for one packaging system
doesn't seem worthwhile.  And we don't even know whether and how it
would work anyway.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On 12/12/14 10:16 AM, Tom Lane wrote:
> I don't particularly object to having the C code built into the backend;
> there's not that much of it, and if we could static-ize some of the global
> variables that are involved presently, it'd be a Good Thing IMO.  However,
> the current arrangement makes sure that the function are not accessible
> except during pg_upgrade, and that seems like a Good Thing as well.  So
> I think pg_upgrade should continue to have SQL scripts that create and
> delete the SQL function definitions for these.

That won't actually work very easily.  LANGUAGE internal functions need
to be in fmgr_builtins, and the only way to get them there is by listing
them in pg_proc.h.  (We could drop the functions in initdb, but seems
kind of silly.)

The functions do already check themselves that they are called in binary
upgrade mode, so exposing them in pg_proc doesn't seem risky.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] operator does not exist: character varying[] <> character[]

Jim Nasby  writes:
> I'd say that array_eq (and probably _cmp) just needs to be taught to fall 
> back to what oper() does, but this part of the commit message gives me pause:

> "Change the operator search algorithms to look for appropriate btree or hash 
> index opclasses, instead of assuming operators named '<' or '=' have the 
> right semantics."

As it should.  array_cmp is the basis for a btree opclass, therefore
it must *not* use operators that are not themselves btree operators.

Quite aside from that, we need to move even further away from having
internal system operations depend on operator-name-based lookups;
see for instance the recent complaints over stuff like IS DISTINCT FROM
failing for types whose operators aren't in the search path.

It's arguable that the typcache code should be taught to look for
binary-compatible opclasses if it can't find one directly for the
specified type.  I'm not sure offhand what rules we'd need to make
to ensure such a search would yield deterministic results, though.

Another possibility is that we might be able to extend the "text_ops"
btree operator family to include an opclass entry for varchar, rather than
relying on binary compatibility to find the text opclass.  But that would
also require some careful thought to understand what the relaxed
invariants should be for the opfamily structure as a whole.  We don't want
to add more actual operators, for fear of creating ambiguous-operator
lookup failures.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] operator does not exist: character varying[] <> character[]

On 12/9/14, 5:06 PM, Jim Nasby wrote:

On 12/9/14, 4:30 PM, Tom Lane wrote:

Jim Nasby writes:

On 12/9/14, 4:19 PM, Jim Nasby wrote:

Is there any particular reason we don't allow comparing char and varchar
arrays? If not I'll submit a patch.

We're also missing operators on text and varchar arrays.

Adding operators would be an incorrect fix.

Right, I'm assuming this is a problem somewhere else (haven't looked into it
yet).

I just wanted confirmation that this is unexpected before I try and fix it.
I'll take your silence on that point as confirmation that this is a bug. :)

I've tracked down what's going on here; array_eq is lazy about finding an
equality operator. It asks lookup_type_cache for TYPECACHE_EQ_OPR_FINFO, which
means it looks first for a Btree Opclass, then a Hash Opclass. If neither is
found then we fail.

OTOH, the path taken in transformAExprOp is very different. It ends up at
oper(), which looks for an exact operator match; if that fails we look for
binary operators we can coerce to. That's the path that allows this to work in
the non-array case.

The question is why. :)

array_eq's call to lookup_type_cache was created in 2003 [1] and hasn't been
touched since. Previously it called equality_oper, which called
compatible_oper, which called oper (same as transforAExprOp does).

I'd say that array_eq (and probably _cmp) just needs to be taught to fall back
to what oper() does, but this part of the commit message gives me pause:

"Change the operator search algorithms to look for appropriate btree or hash index
opclasses, instead of assuming operators named '<' or '=' have the right semantics."

I can see where there are many places where we don't want to just assume than an oprname
of = actually means =, but does that apply to arrays? If the user says "array =
array", isn't it safe to assume that that's the same thing as if tried to compare
two values of the respective typelem's? Wouldn't the same be true for row comparison as
well?

[1]
https://github.com/postgres/postgres/blame/master/src/backend/utils/adt/arrayfuncs.c#L3231
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] parallel mode and parallel contexts

Attached is a patch that adds two new concepts: parallel mode, and
parallel contexts.  The idea of this code is to provide a framework
for specific parallel things that you want to do, such as parallel
sequential scan or parallel sort.  When you're in parallel mode,
certain operations - like DDL, and anything that would update the
command counter - are prohibited.  But you gain the ability to create
a parallel context, which in turn can be used to fire up parallel
workers.  And if you do that, then your snapshot, combo CID hash, and
GUC values will be copied to the worker, which is handy.

This patch is very much half-baked.  Among the things that aren't right yet:

- There's no handling of heavyweight locking, so I'm quite sure it'll
be possible to cause undetected deadlocks if you work at it.  There
are some existing threads on this topic and perhaps we can incorporate
one of those concepts into this patch, but this version does not.
- There's no provision for copying the parent's XID and sub-XIDs, if
any, to the background workers, which means that if you use this and
your transaction has written data, you will get wrong answers, because
TransactionIdIsCurrentTransactionId() will do the wrong thing.
- There's no really deep integration with the transaction system yet.
Previous discussions seem to point toward the need to do various types
of coordinated cleanup when the parallel phase is done, or when an
error happens.  In particular, you probably don't want the abort
record to get written while there are still possibly backends that are
part of that transaction doing work; and you certainly don't want
files created by the current transaction to get removed while some
other backend is still writing them.  The right way to work all of
this out needs some deep thought; agreeing on what the design should
be is probably harder than implement it.

Despite the above, I think this does a fairly good job laying out how
I believe parallelism can be made to work in PostgreSQL: copy a bunch
of state from the user backend to the parallel workers, compute for a
while, and then shut everything down.  Meanwhile, while parallelism is
running, forbid changes to state that's already been synchronized, so
that things don't get out of step.  I think the patch it shows how the
act of synchronizing state from the master to the workers can be made
quite modular and painless, even though it doesn't synchronize
everything relevant.  I'd really appreciate any design thoughts anyone
may have on how to fix the problems mentioned above, how to fix any
other problems you foresee, or even just a list of reasons why you
think this will blow up.

What I think is that we're really pretty close to do real parallelism,
and that this is probably the last major piece of infrastructure that
we need in order to support parallel execution in a reasonable way.
That's a pretty bold statement, but I believe it to be true: despite
the limitations of the current version of this patch, I think we're
very close to being able to sit down and code up a parallel algorithm
in PostgreSQL and have that not be all that hard.  Once we get the
first one, I expect a whole bunch more to come together far more
quickly than the first one did.

I would be remiss if I failed to mention that this patch includes work
by my colleagues Amit Kapila, Rushabh Lathia, and Jeevan Chalke, as
well as my former colleague Noah Misch; and that it would not have
been possible without the patient support of EnterpriseDB management.

Thanks,

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
diff --git a/contrib/parallel_dummy/Makefile b/contrib/parallel_dummy/Makefile
new file mode 100644
index 000..de00f50
--- /dev/null
+++ b/contrib/parallel_dummy/Makefile
@@ -0,0 +1,19 @@
+MODULE_big = parallel_dummy
+OBJS = parallel_dummy.o $(WIN32RES)
+PGFILEDESC = "parallel_dummy - dummy use of parallel infrastructure"
+
+EXTENSION = parallel_dummy
+DATA = parallel_dummy--1.0.sql
+
+REGRESS = parallel_dummy
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/parallel_dummy
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/parallel_dummy/parallel_dummy--1.0.sql b/contrib/parallel_dummy/parallel_dummy--1.0.sql
new file mode 100644
index 000..2a7251c
--- /dev/null
+++ b/contrib/parallel_dummy/parallel_dummy--1.0.sql
@@ -0,0 +1,7 @@
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION parallel_dummy" to load this file. \quit
+
+CREATE FUNCTION parallel_dummy(sleep_time pg_catalog.int4,
+			  nworkers pg_catalog.int4)
+RETURNS pg_catalog.void STRICT
+	AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/contrib/parallel_dummy/parallel_dummy.c b/contrib/parallel_dummy/parallel_dummy.c
new file mode 100644
index 000..99b830f
--- /dev/n

Re: [HACKERS] On partitioning

On Fri, Dec 12, 2014 at 7:40 PM, Josh Berkus  wrote:
> On 12/12/2014 02:10 PM, Tom Lane wrote:
>> Actually, I'm not sure that's what we want.  I thought what we really
>> wanted here was to postpone partition-routing decisions to runtime,
>> so that the behavior would be efficient whether or not the decision
>> could be predetermined at plan time.
>>
>> This still leads to the same point Robert is making: the routing
>> decisions have to be cheap and fast.  But it's wrong to think of it
>> in terms of planner proofs.
>
> The other reason I'd really like to have the new partitioning taken out
> of the planner: expressions.
>
> Currently, if you have partitions with constraints on, day,
> "event_date", the following WHERE clause will NOT use CE and will scan
> all partitions:
>
> WHERE event_date BETWEEN ( '2014-12-11' - interval '1 month' ) and
> '2014-12-11'.
>
> This is despite the fact that the expression above gets rewritten to a
> constant by the time the query is executed; by then it's too late.  To
> say nothing of functions like to_timestamp(), now(), etc.
>
> As long as partitions need to be chosen at plan time, I don't see a good
> way to fix the expression problem.

Fair enough, but that's not the same as not requiring easy proofs. The
planner might not the one doing the proofs, but you still need proofs.

Even if the proving method is hardcoded into the partitioning method,
as in the case of list or range partitioning, it's still a proof. With
arbitrary functions (which is what prompted me to mention proofs) you
can't do that. A function works very well for inserting, but not for
selecting.

I could be wrong though. Maybe there's a way to turn SQL functions
into analyzable things? But it would still be very easy to shoot
yourself in the foot by writing one that is too complex.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] On partitioning

On 12/12/2014 02:10 PM, Tom Lane wrote:
> Actually, I'm not sure that's what we want.  I thought what we really
> wanted here was to postpone partition-routing decisions to runtime,
> so that the behavior would be efficient whether or not the decision
> could be predetermined at plan time.
> 
> This still leads to the same point Robert is making: the routing
> decisions have to be cheap and fast.  But it's wrong to think of it
> in terms of planner proofs.

The other reason I'd really like to have the new partitioning taken out
of the planner: expressions.

Currently, if you have partitions with constraints on, day,
"event_date", the following WHERE clause will NOT use CE and will scan
all partitions:

WHERE event_date BETWEEN ( '2014-12-11' - interval '1 month' ) and
'2014-12-11'.

This is despite the fact that the expression above gets rewritten to a
constant by the time the query is executed; by then it's too late.  To
say nothing of functions like to_timestamp(), now(), etc.

As long as partitions need to be chosen at plan time, I don't see a good
way to fix the expression problem.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Fri, Dec 12, 2014 at 7:25 PM, Michael Paquier
 wrote:
> On Sat, Dec 13, 2014 at 1:08 AM, Robert Haas  wrote:
>> On Fri, Dec 12, 2014 at 10:04 AM, Andres Freund  wrote:
 Note that autovacuum and fsync are off.
 =# select phase, user_diff, system_diff,
 pg_size_pretty(pre_update - pre_insert),
 pg_size_pretty(post_update - pre_update) from results;
phase| user_diff | system_diff | pg_size_pretty |
 pg_size_pretty
 +---+-++
  Compression FPW| 42.990799 |0.868179 | 429 MB | 567 MB
  No compression | 25.688731 |1.236551 | 429 MB | 727 MB
  Compression record | 56.376750 |0.769603 | 429 MB | 566 MB
 (3 rows)
 If we do record-level compression, we'll need to be very careful in
 defining a lower-bound to not eat unnecessary CPU resources, perhaps
 something that should be controlled with a GUC. I presume that this stands
 true as well for the upper bound.
>>>
>>> Record level compression pretty obviously would need a lower boundary
>>> for when to use compression. It won't be useful for small heapam/btree
>>> records, but it'll be rather useful for large multi_insert, clean or
>>> similar records...
>>
>> Unless I'm missing something, this test is showing that FPW
>> compression saves 298MB of WAL for 17.3 seconds of CPU time, as
>> against master.  And compressing the whole record saves a further 1MB
>> of WAL for a further 13.39 seconds of CPU time.  That makes
>> compressing the whole record sound like a pretty terrible idea - even
>> if you get more benefit by reducing the lower boundary, you're still
>> burning a ton of extra CPU time for almost no gain on the larger
>> records.  Ouch!
>>
>> (Of course, I'm assuming that Michael's patch is reasonably efficient,
>> which might not be true.)
> Note that I was curious about the worst-case ever, aka how much CPU
> pg_lzcompress would use if everything is compressed, even the smallest
> records. So we'll surely need a lower-bound. I think that doing some
> tests with a lower bound set as a multiple of SizeOfXLogRecord would
> be fine, but in this case what we'll see is a result similar to what
> FPW compression does.


In general, lz4 (and pg_lz is similar to lz4) compresses very poorly
anything below about 128b in length. Of course there are outliers,
with some very compressible stuff, but with regular text or JSON data,
it's quite unlikely to compress at all with smaller input. Compression
is modest up to about 1k when it starts to really pay off.

That's at least my experience with lots JSON-ish, text-ish and CSV
data sets, compressible but not so much in small bits.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PATCH: hashjoin - gracefully increasing NTUP_PER_BUCKET instead of batching

2014-12-12 Thread Peter Geoghegan

On Fri, Dec 12, 2014 at 5:19 AM, Robert Haas  wrote:
> Well, this is sort of one of the problems with work_mem.  When we
> switch to a tape sort, or a tape-based materialize, we're probably far
> from out of memory.  But trying to set work_mem to the amount of
> memory we have can easily result in a memory overrun if a load spike
> causes lots of people to do it all at the same time.  So we have to
> set work_mem conservatively, but then the costing doesn't really come
> out right.  We could add some more costing parameters to try to model
> this, but it's not obvious how to get it right.

I've heard of using "set work_mem = *" with advisory locks plenty of
times. There might be a better way to set it dynamically than a full
admission control implementation.

-- 
Peter Geoghegan


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Sat, Dec 13, 2014 at 1:08 AM, Robert Haas  wrote:
> On Fri, Dec 12, 2014 at 10:04 AM, Andres Freund  wrote:
>>> Note that autovacuum and fsync are off.
>>> =# select phase, user_diff, system_diff,
>>> pg_size_pretty(pre_update - pre_insert),
>>> pg_size_pretty(post_update - pre_update) from results;
>>>phase| user_diff | system_diff | pg_size_pretty |
>>> pg_size_pretty
>>> +---+-++
>>>  Compression FPW| 42.990799 |0.868179 | 429 MB | 567 MB
>>>  No compression | 25.688731 |1.236551 | 429 MB | 727 MB
>>>  Compression record | 56.376750 |0.769603 | 429 MB | 566 MB
>>> (3 rows)
>>> If we do record-level compression, we'll need to be very careful in
>>> defining a lower-bound to not eat unnecessary CPU resources, perhaps
>>> something that should be controlled with a GUC. I presume that this stands
>>> true as well for the upper bound.
>>
>> Record level compression pretty obviously would need a lower boundary
>> for when to use compression. It won't be useful for small heapam/btree
>> records, but it'll be rather useful for large multi_insert, clean or
>> similar records...
>
> Unless I'm missing something, this test is showing that FPW
> compression saves 298MB of WAL for 17.3 seconds of CPU time, as
> against master.  And compressing the whole record saves a further 1MB
> of WAL for a further 13.39 seconds of CPU time.  That makes
> compressing the whole record sound like a pretty terrible idea - even
> if you get more benefit by reducing the lower boundary, you're still
> burning a ton of extra CPU time for almost no gain on the larger
> records.  Ouch!
>
> (Of course, I'm assuming that Michael's patch is reasonably efficient,
> which might not be true.)
Note that I was curious about the worst-case ever, aka how much CPU
pg_lzcompress would use if everything is compressed, even the smallest
records. So we'll surely need a lower-bound. I think that doing some
tests with a lower bound set as a multiple of SizeOfXLogRecord would
be fine, but in this case what we'll see is a result similar to what
FPW compression does.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On Sat, Dec 13, 2014 at 1:00 AM, Alvaro Herrera
 wrote:
> Tom Lane wrote:
>> Heikki Linnakangas  writes:
>> > On 12/12/2014 03:07 PM, Peter Eisentraut wrote:
>> >> On 12/9/14 4:10 PM, Alvaro Herrera wrote:
>> >>> Maybe it makes sense to have a distinction between client programs and
>> >>> server programs.  Can we have src/sbin/ and move stuff that involves the
>> >>> server side in there?  I think that'd be pg_xlogdump, pg_archivecleanup,
>> >>> pg_upgrade, pg_test_timing, pg_test_fsync.  (If we were feeling bold we
>> >>> could also move pg_resetxlog, pg_controldata and initdb there.)
>>
>> >> I was thinking about that.  What do others think?
>>
>> > Sounds good. We already separate server and client programs in the docs,
>> > and packagers put them in different packages too. This should make
>> > packagers' life a little bit easier in the long run.
>>
>> I'm pretty much -1 on relocating anything that's under src/bin already.
>
> So let's put the whole bunch under src/bin/ and be done with it.

When doing so let's be careful with the MSVC specs, this will need a
set of definitions with AddSimpleFrontend. This should ensure as well
that versioning work that has been done on Windows for this release
cycle is not broken for all the things moved. I don't mind
double-checking that..
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

On Sat, Dec 13, 2014 at 5:30 AM, Magnus Hagander  wrote:
> On Fri, Dec 12, 2014 at 9:05 PM, Josh Berkus  wrote:
>> On 12/12/2014 11:52 AM, Magnus Hagander wrote:
>>> On Fri, Dec 12, 2014 at 8:43 PM, Tomas Vondra  wrote:
 On 11.12.2014 16:06, Bruce Momjian wrote:
> On Wed, Dec 10, 2014 at 11:00:21PM -0800, Josh Berkus wrote:
 3) manual processing that could be automated

I think the CF site was a huge step forward, but maybe we could
improve it, to automate some of the CFM tasks? For example
integrating it a bit more tightly with the mailinglist (which would
make the life easier even for patch authors and reviewers)?
>>>
>>> Just as a note abot this one part along (I'll read the rest later). I
>>> do have the new version of the CF app more or less ready to deploy,
>>> but I got bogged down by thinking "I'll do it between two commitfests
>>> to not be disruptive". But there has been no "between two
>>> commitfests". Hopefully I can get around to doing it during the
>>> holidays. It does integrate much tighter with the archives, that's
>>> probably the core feature of it.
>>
>> It also automates a bunch of the emailing no?
>
> Yes.
That's pretty cool.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems



On 12/12/2014 01:22 PM, Jim Nasby wrote:


On 12/12/14, 1:44 PM, Joshua D. Drake wrote:

1. We don't need a full-time engineer to manage a commitfest. We need
a manager or PM.


I don't think that's actually true. The major points on this thread are
that 1) we don't have enough capacity for doing reviews and 2) the CFM
has no authority to enforce anything.

I see no way that #2 can be addressed by a mere manager/PM. If three
*very* senior community members (David, Josh and Robert) couldn't get
this done there's no way a PM could. (Well, I suppose of Tom was
standing behind them with a flaming sword it might work...)


#2 is solved by my previous comments about giving the CFM/C the 
authority. -Core could do that, they are in charge of release.


#1 is a more difficult problem for sure.

JD
--
Command Prompt, Inc. - http://www.commandprompt.com/  503-667-4564
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, @cmdpromptinc
"If we send our children to Caesar for their education, we should
 not be surprised when they come back as Romans."


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] On partitioning

On Fri, Dec 12, 2014 at 7:10 PM, Tom Lane  wrote:
> Claudio Freire  writes:
>> On Fri, Dec 12, 2014 at 6:48 PM, Robert Haas  wrote:
>>> I have very little idea what the API you're imagining would actually
>>> look like from this description, but it sounds like a terrible idea.
>>> We don't want to make this infinitely general.  We need a *fast* way
>>> to go from a value (or list of values, one per partitioning column) to
>>> a partition OID, and the way to get there is not to call arbitrary
>>> user code.
>
>> I think this was mentioned upthread, but I'll repeat it anyway since
>> it seems to need repeating.
>
>> More than fast, you want it analyzable (by the planner). Ie: it has to
>> be easy to prove partition exclusion against a where clause.
>
> Actually, I'm not sure that's what we want.  I thought what we really
> wanted here was to postpone partition-routing decisions to runtime,
> so that the behavior would be efficient whether or not the decision
> could be predetermined at plan time.
>
> This still leads to the same point Robert is making: the routing
> decisions have to be cheap and fast.  But it's wrong to think of it
> in terms of planner proofs.

You'll need proofs whether at the planner or at the execution engine.

A sequential scan over a partition with a query like

select * from foo where date between X and Y

Would be ripe for that but at some point you need to prove that the
where clause excludes whole partitions. Be it at runtime (while
executing the sequential scan node) or planning time.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] On partitioning

Claudio Freire  writes:
> On Fri, Dec 12, 2014 at 6:48 PM, Robert Haas  wrote:
>> I have very little idea what the API you're imagining would actually
>> look like from this description, but it sounds like a terrible idea.
>> We don't want to make this infinitely general.  We need a *fast* way
>> to go from a value (or list of values, one per partitioning column) to
>> a partition OID, and the way to get there is not to call arbitrary
>> user code.

> I think this was mentioned upthread, but I'll repeat it anyway since
> it seems to need repeating.

> More than fast, you want it analyzable (by the planner). Ie: it has to
> be easy to prove partition exclusion against a where clause.

Actually, I'm not sure that's what we want.  I thought what we really
wanted here was to postpone partition-routing decisions to runtime,
so that the behavior would be efficient whether or not the decision
could be predetermined at plan time.

This still leads to the same point Robert is making: the routing
decisions have to be cheap and fast.  But it's wrong to think of it
in terms of planner proofs.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PATCH: hashjoin - gracefully increasing NTUP_PER_BUCKET instead of batching

On Fri, Dec 12, 2014 at 4:54 PM, Tomas Vondra  wrote:
 Well, this is sort of one of the problems with work_mem.  When we
 switch to a tape sort, or a tape-based materialize, we're probably far
 from out of memory.  But trying to set work_mem to the amount of
 memory we have can easily result in a memory overrun if a load spike
 causes lots of people to do it all at the same time.  So we have to
 set work_mem conservatively, but then the costing doesn't really come
 out right.  We could add some more costing parameters to try to model
 this, but it's not obvious how to get it right.
>>>
>>> Ummm, I don't think that's what I proposed. What I had in mind was a
>>> flag "the batches are likely to stay in page cache". Because when it is
>>> likely, batching is probably faster (compared to increased load factor).
>>
>> How will you know whether to set the flag?
>
> I don't know. I just wanted to make it clear that I'm not suggesting
> messing with work_mem (increasing it or whatewer). Or maybe I got your
> comments about memory overrun etc. wrong - now that I read it again,
> maybe it's meant just as an example of how difficult problem it is?

More or less, yeah.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] On partitioning

On Fri, Dec 12, 2014 at 6:48 PM, Robert Haas  wrote:
> On Fri, Dec 12, 2014 at 4:28 PM, Jim Nasby  wrote:
>>> Sure.  Mind you, I'm not proposing that the syntax I just mooted is
>>> actually for the best.  What I'm saying is that we need to talk about
>>> it.
>>
>> Frankly, if we're going to require users to explicitly define each partition
>> then I think the most appropriate API would be a function. Users will be
>> writing code to create new partitions as needed, and it's generally easier
>> to write code that calls a function as opposed to glomming a text string
>> together and passing that to EXECUTE.
>
> I have very little idea what the API you're imagining would actually
> look like from this description, but it sounds like a terrible idea.
> We don't want to make this infinitely general.  We need a *fast* way
> to go from a value (or list of values, one per partitioning column) to
> a partition OID, and the way to get there is not to call arbitrary
> user code.

I think this was mentioned upthread, but I'll repeat it anyway since
it seems to need repeating.

More than fast, you want it analyzable (by the planner). Ie: it has to
be easy to prove partition exclusion against a where clause.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PATCH: hashjoin - gracefully increasing NTUP_PER_BUCKET instead of batching

On 12.12.2014 22:13, Robert Haas wrote:
> On Fri, Dec 12, 2014 at 11:50 AM, Tomas Vondra  wrote:
>> On 12.12.2014 14:19, Robert Haas wrote:
>>> On Thu, Dec 11, 2014 at 5:46 PM, Tomas Vondra  wrote:
>>>
 Regarding the "sufficiently small" - considering today's hardware, we're
 probably talking about gigabytes. On machines with significant memory
 pressure (forcing the temporary files to disk), it might be much lower,
 of course. Of course, it also depends on kernel settings (e.g.
 dirty_bytes/dirty_background_bytes).
>>>
>>> Well, this is sort of one of the problems with work_mem.  When we
>>> switch to a tape sort, or a tape-based materialize, we're probably far
>>> from out of memory.  But trying to set work_mem to the amount of
>>> memory we have can easily result in a memory overrun if a load spike
>>> causes lots of people to do it all at the same time.  So we have to
>>> set work_mem conservatively, but then the costing doesn't really come
>>> out right.  We could add some more costing parameters to try to model
>>> this, but it's not obvious how to get it right.
>>
>> Ummm, I don't think that's what I proposed. What I had in mind was a
>> flag "the batches are likely to stay in page cache". Because when it is
>> likely, batching is probably faster (compared to increased load factor).
> 
> How will you know whether to set the flag?

I don't know. I just wanted to make it clear that I'm not suggesting
messing with work_mem (increasing it or whatewer). Or maybe I got your
comments about memory overrun etc. wrong - now that I read it again,
maybe it's meant just as an example of how difficult problem it is?

regards
Tomas


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] On partitioning

On Fri, Dec 12, 2014 at 4:28 PM, Jim Nasby  wrote:
>> Sure.  Mind you, I'm not proposing that the syntax I just mooted is
>> actually for the best.  What I'm saying is that we need to talk about
>> it.
>
> Frankly, if we're going to require users to explicitly define each partition
> then I think the most appropriate API would be a function. Users will be
> writing code to create new partitions as needed, and it's generally easier
> to write code that calls a function as opposed to glomming a text string
> together and passing that to EXECUTE.

I have very little idea what the API you're imagining would actually
look like from this description, but it sounds like a terrible idea.
We don't want to make this infinitely general.  We need a *fast* way
to go from a value (or list of values, one per partitioning column) to
a partition OID, and the way to get there is not to call arbitrary
user code.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On Fri, Dec 12, 2014 at 12:19 PM, Alvaro Herrera
 wrote:
>> If we decide that executables can no longer live in contrib,
>
> Nobody is saying that.

The reason I though that might be on the table is that the first post
on this thread, at least as I understood it, proposed to move every
executable out of contrib, which would seem to also imply that we
wouldn't put any new ones there.  I agree that course of action
doesn't seem to be garnering many votes, but I think it does seem to
have been proposed, unless I am confused.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Fri, Dec 12, 2014 at 1:51 PM, Simon Riggs  wrote:
> What I don't understand is why we aren't working on double buffering,
> since that cost would be paid in a background process and would be
> evenly spread out across a checkpoint. Plus we'd be able to remove
> FPWs altogether, which is like 100% compression.

The previous patch to implement that - by somebody at vmware - was an
epic fail.  I'm not opposed to seeing somebody try again, but it's a
tricky problem.  When the double buffer fills up, then you've got to
finish flushing the pages whose images are stored in the buffer to
disk before you can overwrite it, which acts like a kind of
mini-checkpoint.  That problem might be solvable, but let's use this
thread to discuss this patch, not some other patch that someone might
have chosen to write but didn't.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Final Patch for GROUPING SETS

Andrew Gierth  writes:
> "Tom" == Tom Lane  writes:
>  Tom> That seems pretty messy, especially given your further comments
>  Tom> that these plan nodes are interconnected and know about each
>  Tom> other (though you failed to say exactly how).

> I'd already explained in more detail way back when we posted the
> patch. But to reiterate: the ChainAggregate nodes pass through their
> input data unchanged, but on group boundaries they write aggregated
> result rows to a tuplestore shared by the whole chain. The top node
> returns the data from the tuplestore after its own output is
> completed.

That seems pretty grotty from a performance+memory consumption standpoint.
At peak memory usage, each one of the Sort nodes will contain every input
row, and the shared tuplestore will contain every output row.  That will
lead to either a lot of memory eaten, or a lot of temp-file I/O, depending
on how big work_mem is.

In principle, with the CTE+UNION approach I was suggesting, the peak
memory consumption would be one copy of the input rows in the CTE's
tuplestore plus one copy in the active branch's Sort node.  I think a
bit of effort would be needed to get there (ie, shut down one branch's
Sort node before starting the next, something I'm pretty sure doesn't
happen today).  But it's doable whereas I don't see how we dodge the
multiple-active-sorts problem with the chained implementation.

>  Tom> ISTM that maybe what we should do is take a cue from the SQL
>  Tom> spec, which defines these things in terms of UNION ALL of
>  Tom> plain-GROUP-BY operations reading from a common CTE.

> I looked at that, in fact that was our original plan, but it became
> clear quite quickly that it was not going to be easy.

> I tried two different approaches. First was to actually re-plan the
> input (i.e. running query_planner more than once) for different sort
> orders; that crashed and burned quickly thanks to the extent to which
> the planner assumes that it'll be run once only on any given input.

Well, we'd not want to rescan the input multiple times, so I don't think
that generating independent plan trees for each sort order would be the
thing to do anyway.  I suppose ideally it would be nice to check the costs
of getting the different sort orders, so that the one Sort we elide is the
one that gets the best cost savings.  But the WindowAgg code isn't that
smart either and no one's really complained, so I think this can wait.
(Eventually I'd like to make such cost comparisons possible as part of the
upper-planner Pathification that I keep nattering about.  But it doesn't
seem like a prerequisite for getting GROUPING SETS in.)

> Second was to generate a CTE for the input data. This didn't get very
> far because everything that already exists to handle CTE nodes assumes
> that they are explicit in the planner's input (that they have their
> own Query node, etc.) and I was not able to determine a good solution.

Seems like restructuring that wouldn't be *that* hard.  We probably don't
want it to be completely like a CTE for planning purposes anyway --- that
would foreclose passing down any knowledge of desired sort order, which
we don't want.  But it seems like we could stick a variant of CtePath
atop the chosen result path of the scan/join planning phase.  If you like
I can poke into this a bit.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] On partitioning


On 12/12/14, 8:03 AM, Robert Haas wrote:

On Thu, Dec 11, 2014 at 11:43 PM, Amit Langote
  wrote:

>In case of what we would have called a 'LIST' partition, this could look like
>
>... FOR VALUES (val1, val2, val3, ...)
>
>Assuming we only support partition key to contain only one column in such a 
case.
>
>In case of what we would have called a 'RANGE' partition, this could look like
>
>... FOR VALUES (val1min, val2min, ...) TO (val1max, val2max, ...)
>
>How about BETWEEN ... AND ... ?

Sure.  Mind you, I'm not proposing that the syntax I just mooted is
actually for the best.  What I'm saying is that we need to talk about
it.


Frankly, if we're going to require users to explicitly define each partition 
then I think the most appropriate API would be a function. Users will be 
writing code to create new partitions as needed, and it's generally easier to 
write code that calls a function as opposed to glomming a text string together 
and passing that to EXECUTE.
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems


On 12/12/14, 1:44 PM, Joshua D. Drake wrote:

1. We don't need a full-time engineer to manage a commitfest. We need a manager 
or PM.


I don't think that's actually true. The major points on this thread are that 1) 
we don't have enough capacity for doing reviews and 2) the CFM has no authority 
to enforce anything.

I see no way that #2 can be addressed by a mere manager/PM. If three *very* 
senior community members (David, Josh and Robert) couldn't get this done 
there's no way a PM could. (Well, I suppose of Tom was standing behind them 
with a flaming sword it might work...)

Even so, this still wouldn't address the real problem, which is lack of review 
capacity.

FWIW, I faced the same problem at Enova: good, solid reviews were very 
important for maintaining the quality of the data and database code, yet were a 
constant source of pain and friction. And that was with people being paid to do 
them, as well as a very extensive set of unit tests.

In other words, this isn't an easy problem to solve.

One thing that I think would help is modifying the CF app to allow for multiple 
reviewers. IIRC I reviewed 4 or 5 patches but I didn't mark myself as reviewer 
of any of them because I don't feel I have enough knowledge to fulfill that 
role.
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PATCH: hashjoin - gracefully increasing NTUP_PER_BUCKET instead of batching

On Fri, Dec 12, 2014 at 11:50 AM, Tomas Vondra  wrote:
> On 12.12.2014 14:19, Robert Haas wrote:
>> On Thu, Dec 11, 2014 at 5:46 PM, Tomas Vondra  wrote:
>>
>>> Regarding the "sufficiently small" - considering today's hardware, we're
>>> probably talking about gigabytes. On machines with significant memory
>>> pressure (forcing the temporary files to disk), it might be much lower,
>>> of course. Of course, it also depends on kernel settings (e.g.
>>> dirty_bytes/dirty_background_bytes).
>>
>> Well, this is sort of one of the problems with work_mem.  When we
>> switch to a tape sort, or a tape-based materialize, we're probably far
>> from out of memory.  But trying to set work_mem to the amount of
>> memory we have can easily result in a memory overrun if a load spike
>> causes lots of people to do it all at the same time.  So we have to
>> set work_mem conservatively, but then the costing doesn't really come
>> out right.  We could add some more costing parameters to try to model
>> this, but it's not obvious how to get it right.
>
> Ummm, I don't think that's what I proposed. What I had in mind was a
> flag "the batches are likely to stay in page cache". Because when it is
> likely, batching is probably faster (compared to increased load factor).

How will you know whether to set the flag?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems


On 12/12/14, 2:38 PM, Josh Berkus wrote:



Just as a note abot this one part along (I'll read the rest later). I
do have the new version of the CF app more or less ready to deploy,
but I got bogged down by thinking "I'll do it between two commitfests
to not be disruptive". But there has been no "between two
commitfests". Hopefully I can get around to doing it during the
holidays. It does integrate much tighter with the archives, that's
probably the core feature of it.


It also automates a bunch of the emailing no?


Yes.


I can key in a bunch of the backlog of patches into the new app over the
holidays, but not before then.


FWIW, I suspect a call for help on -general or IRC would find volunteers for 
any necessary data entry work...
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems


>>> Just as a note abot this one part along (I'll read the rest later). I
>>> do have the new version of the CF app more or less ready to deploy,
>>> but I got bogged down by thinking "I'll do it between two commitfests
>>> to not be disruptive". But there has been no "between two
>>> commitfests". Hopefully I can get around to doing it during the
>>> holidays. It does integrate much tighter with the archives, that's
>>> probably the core feature of it.
>>
>> It also automates a bunch of the emailing no?
> 
> Yes.

I can key in a bunch of the backlog of patches into the new app over the
holidays, but not before then.


-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Peter Geoghegan

On Fri, Dec 12, 2014 at 12:30 PM, Magnus Hagander  wrote:
>> It also automates a bunch of the emailing no?
>
> Yes.

Please let me know the details (privately or otherwise). I'd like to
try it out again.


-- 
Peter Geoghegan


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

Tom Lane wrote:
> Christoph Berg  writes:
> > However, for PostgreSQL this means lengthy debian/*.install files
> > (the equivalent of %files in rpm spec speak):
> 
> Right ...
> 
> > If there were separate "install-client", "install-server", and
> > "install-contrib" targets, that would probably shorten those files
> > quite a bit. Especially messy is the part where *.so needs to be
> > sorted into server/contrib, along with an similar large bunch of
> > binaries.
> 
> Pardon me for not knowing much about Debian packages, but how would
> that work exactly?  Is it possible to do make install-client, then
> package the installed files, then rm -rf the install tree, then
> repeat for install-server and install-contrib?  In the RPM world
> this would never work because the build/install step happens in
> toto before the packaging step.

Uh, couldn't you just run "make install-client DESTDIR=.../client" for
client-only files, and so on?  You would end up with separate
directories containing files for each subpackage.

> Even without that, it seems like it'd be hard to make it entirely
> automatic since some files would be installed in multiple cases (and
> directories even more so).

Yeah, you would need to fix that somehow.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Magnus Hagander

On Fri, Dec 12, 2014 at 9:05 PM, Josh Berkus  wrote:
> On 12/12/2014 11:52 AM, Magnus Hagander wrote:
>> On Fri, Dec 12, 2014 at 8:43 PM, Tomas Vondra  wrote:
>>> On 11.12.2014 16:06, Bruce Momjian wrote:
 On Wed, Dec 10, 2014 at 11:00:21PM -0800, Josh Berkus wrote:
>>> 3) manual processing that could be automated
>>>
>>>I think the CF site was a huge step forward, but maybe we could
>>>improve it, to automate some of the CFM tasks? For example
>>>integrating it a bit more tightly with the mailinglist (which would
>>>make the life easier even for patch authors and reviewers)?
>>
>> Just as a note abot this one part along (I'll read the rest later). I
>> do have the new version of the CF app more or less ready to deploy,
>> but I got bogged down by thinking "I'll do it between two commitfests
>> to not be disruptive". But there has been no "between two
>> commitfests". Hopefully I can get around to doing it during the
>> holidays. It does integrate much tighter with the archives, that's
>> probably the core feature of it.
>
> It also automates a bunch of the emailing no?

Yes.


-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

Christoph Berg  writes:
> However, for PostgreSQL this means lengthy debian/*.install files
> (the equivalent of %files in rpm spec speak):

Right ...

> If there were separate "install-client", "install-server", and
> "install-contrib" targets, that would probably shorten those files
> quite a bit. Especially messy is the part where *.so needs to be
> sorted into server/contrib, along with an similar large bunch of
> binaries.

Pardon me for not knowing much about Debian packages, but how would
that work exactly?  Is it possible to do make install-client, then
package the installed files, then rm -rf the install tree, then
repeat for install-server and install-contrib?  In the RPM world
this would never work because the build/install step happens in
toto before the packaging step.  Even without that, it seems like
it'd be hard to make it entirely automatic since some files would
be installed in multiple cases (and directories even more so).

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Christoph Berg

Re: Andres Freund 2014-12-12 <20141212152723.go31...@awork2.anarazel.de>
> On 2014-12-12 10:20:58 -0500, Tom Lane wrote:
> > Peter Eisentraut  writes:
> > > On 12/12/14 8:13 AM, Andres Freund wrote:
> > >> Wouldn't a make install-server/client targets or something similar
> > >> actually achieve the same thing? Seems simpler to maintain to me.

Ack. The default install location would still be .../bin, but invoked
from different targets.

> > > Adding non-standard makefile targets comes with its own set of
> > > maintenance issues.
> > 
> > It would be of zero value to packagers anyway; certainly so for those
> > following the Red Hat tradition, in which you tell the package Makefile
> > to install everything and then what goes into which subpackage is
> > sorted out in a separate, subsequent step.  Possibly Debian or other
> > packaging infrastructures do it differently, but I doubt that.
> 
> Debian has that step as well - you don't really have to use it, but the
> postgres debian packages do so. They already don't adhere to the current
> distinction.

The standard Debian package installs into debian/tmp/ and then picks
files from there into individual packages.

However, for PostgreSQL this means lengthy debian/*.install files
(the equivalent of %files in rpm spec speak):

$ wc -l debian/*.install
   2 debian/libecpg6.install
   1 debian/libecpg-compat3.install
  17 debian/libecpg-dev.install
   1 debian/libpgtypes3.install
   2 debian/libpq5.install
  14 debian/libpq-dev.install
  39 debian/postgresql-9.4.install
  40 debian/postgresql-client-9.4.install
  65 debian/postgresql-contrib-9.4.install
   2 debian/postgresql-doc-9.4.install
   3 debian/postgresql-plperl-9.4.install
   2 debian/postgresql-plpython3-9.4.install
   3 debian/postgresql-plpython-9.4.install
   5 debian/postgresql-pltcl-9.4.install
   3 debian/postgresql-server-dev-9.4.install
 199 total

If there were separate "install-client", "install-server", and
"install-contrib" targets, that would probably shorten those files
quite a bit. Especially messy is the part where *.so needs to be
sorted into server/contrib, along with an similar large bunch of
binaries.

Of course that would only solve part of the problem (I'm not going to
suggest creating 15 targets for the 15 binary packages we are
building), but it would solve the uglier part.

Christoph
-- 
c...@df7cb.de | http://www.df7cb.de/


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

On 12/12/2014 11:52 AM, Magnus Hagander wrote:
> On Fri, Dec 12, 2014 at 8:43 PM, Tomas Vondra  wrote:
>> On 11.12.2014 16:06, Bruce Momjian wrote:
>>> On Wed, Dec 10, 2014 at 11:00:21PM -0800, Josh Berkus wrote:
>> 3) manual processing that could be automated
>>
>>I think the CF site was a huge step forward, but maybe we could
>>improve it, to automate some of the CFM tasks? For example
>>integrating it a bit more tightly with the mailinglist (which would
>>make the life easier even for patch authors and reviewers)?
> 
> Just as a note abot this one part along (I'll read the rest later). I
> do have the new version of the CF app more or less ready to deploy,
> but I got bogged down by thinking "I'll do it between two commitfests
> to not be disruptive". But there has been no "between two
> commitfests". Hopefully I can get around to doing it during the
> holidays. It does integrate much tighter with the archives, that's
> probably the core feature of it.

It also automates a bunch of the emailing no?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Magnus Hagander

On Fri, Dec 12, 2014 at 8:43 PM, Tomas Vondra  wrote:
> On 11.12.2014 16:06, Bruce Momjian wrote:
>> On Wed, Dec 10, 2014 at 11:00:21PM -0800, Josh Berkus wrote:
> 3) manual processing that could be automated
>
>I think the CF site was a huge step forward, but maybe we could
>improve it, to automate some of the CFM tasks? For example
>integrating it a bit more tightly with the mailinglist (which would
>make the life easier even for patch authors and reviewers)?

Just as a note abot this one part along (I'll read the rest later). I
do have the new version of the CF app more or less ready to deploy,
but I got bogged down by thinking "I'll do it between two commitfests
to not be disruptive". But there has been no "between two
commitfests". Hopefully I can get around to doing it during the
holidays. It does integrate much tighter with the archives, that's
probably the core feature of it.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems



On 12/12/2014 11:35 AM, Alvaro Herrera wrote:


We (not CMD, the community) with proper incentive could fund this. It really
wouldn't be that hard. That said, there would have to be a clear
understanding of expectations, results, and authority.


Uh, really?


Yeah I think so. Money can be easy to get, when clear leadership and 
goals are presented.



Last I looked at the numbers from SPI treasurer reports,
they are not impressive enough to hire a full-time engineer, let alone a
senior one.


1. We don't need a full-time engineer to manage a commitfest. We need a 
manager or PM.


2. The original idea came from cross-company (which is part of the 
community)


3. There is more non-profits in this game than just SPI

4. We could do it on a 6 month for 1 year contract

Sincerely,

Joshua D. Drake



--
Command Prompt, Inc. - http://www.commandprompt.com/  503-667-4564
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, @cmdpromptinc
"If we send our children to Caesar for their education, we should
 not be surprised when they come back as Romans."


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

On 12/12/2014 11:35 AM, Alvaro Herrera wrote:
> Uh, really?  Last I looked at the numbers from SPI treasurer reports,
> they are not impressive enough to hire a full-time engineer, let alone a
> senior one.
> 
> The Linux Foundation has managed to pay for Linus Torvalds somehow, so
> it does sound possible.  We have a number of companies making money all
> over the globe, at least.

You're looking at this wrong.  We have that amount of money in the
account based on zero fundraising whatsoever, which we don't do because
we don't spend the money.  We get roughly $20,000 per year just by
putting up a "donate" link, and not even promoting it.

So, what this would take is:

1) a candidate who is currently a known major committer

2) clear goals for what this person would spend their time doing

3) buy-in from the Core Team, the committers, and the general hackers
community (including buy-in to the idea of favorable publicity for
funding supporters)

4) an organizing committee with the time to deal with managing
foundation funds

If we had those four things, the fundraising part would be easy.  I
speak as someone who used to raise $600,000 per year for a non-profit in
individual gifts alone.

However, *I'm* not clear on what problems this non-profit employed
person would be solving for the community.  I doubt anyone else is
either.  Until we have consensus on that, there's no point in talking
about anything else.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

On 11.12.2014 16:06, Bruce Momjian wrote:
> On Wed, Dec 10, 2014 at 11:00:21PM -0800, Josh Berkus wrote:
>>
>> I will add:
>>
>> 4. commitfest managers have burned out and refuse to do it again
> 
> Agreed. The "fun", if it was ever there, has left the commitfest 
> process.

I've never been a CFM, but from my experience as a patch author and
reviewer, I think there are two or three reasons for that (of course,
I'm not saying those are the most important ones or the only ones):

1) unclear definition of what CFM is expected to do

   The current wiki page describing the role of CFM [1] is rather
   obsolete, IMHO. For example it says that CFM assigns patches ro
   reviewers, posts announcements to pgsql-rrreviewers, etc.

   I don't think this was really followed in recent CFs.

   This however results in people filling the gaps with what they
   believe the CFM should do, causing misunderstandings etc. Shall
   we update the description a bit, to reflect the current state
   of affairs?

   Maybe we should also consider which responsibilities should be
   shifted back to the developers and reviewers. E.g. do we really
   expect the CFM to assign patches to reviewers?

2) not really following the rules

   We do have a few rules that we don't follow as much as we should,
   notably:

   * 1:1 for patches:reviews (one review for each submitted patch)
   * no new patches after the CF starts (post it to the next one)
   * CF ends at a specific date

   I believe violating those rules is related to (1) because it may
   lead to perception that CFM makes them up or does not enforce them
   equally for all patches.

3) manual processing that could be automated

   I think the CF site was a huge step forward, but maybe we could
   improve it, to automate some of the CFM tasks? For example
   integrating it a bit more tightly with the mailinglist (which would
   make the life easier even for patch authors and reviewers)?

However as I said before, I never was a CFM - I'd like to hear from the
actual CFMs what's their opinion on this.

kind regards
Tomas

[1] https://wiki.postgresql.org/wiki/Running_a_CommitFest

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

Joshua D. Drake wrote:
> 
> On 12/12/2014 06:30 AM, Robert Haas wrote:
> 
> >Yeah, that would be great, and even better if we could get 2 or 3
> >positions funded so that the success or failure isn't too much tied to
> >a single individual.  But even getting 1 position funded in a
> >stable-enough fashion that someone would be willing to bet on it seems
> >like a challenge.  (Maybe other people here are less risk-averse than
> >I am.)
> 
> We (not CMD, the community) with proper incentive could fund this. It really
> wouldn't be that hard. That said, there would have to be a clear
> understanding of expectations, results, and authority.

Uh, really?  Last I looked at the numbers from SPI treasurer reports,
they are not impressive enough to hire a full-time engineer, let alone a
senior one.

The Linux Foundation has managed to pay for Linus Torvalds somehow, so
it does sound possible.  We have a number of companies making money all
over the globe, at least.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems



On 12/12/2014 10:59 AM, Simon Riggs wrote:


On 12 December 2014 at 15:10, David Fetter  wrote:

On Thu, Dec 11, 2014 at 05:55:56PM -0500, Tom Lane wrote:

Josh Berkus  writes:

How about *you* run the next one, Tom?


I think the limited amount of time I can put into a commitfest is
better spent on reviewing patches than on managing the process.


IIRC Tom was pretty much the only person doing patch review for
probably 5 years during 2003-2008, maybe others. AFAICS he was
managing that process. Thank you, Tom.

I've never seen him moan loudly about this, so I'm surprised to hear
such things from people that have done much less.

Any solution to our current problems will come from working together,
not by fighting.

We just need to do more reviews. Realising this, I have begun to do
more. I encourage others to do this also.



Simon,

Well said but again, I think a lot of people are hand waving about a 
simple problem (within the current structure) and that problem is just 
one of submission.


Those doing the patch review/writing need to submit to the authority of 
the CFM or CFC (commit fest comittee). That happens and a lot of the 
angst around this process goes away.


Sincerely,

Joshua D. Drake


--
Command Prompt, Inc. - http://www.commandprompt.com/  503-667-4564
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, @cmdpromptinc
"If we send our children to Caesar for their education, we should
 not be surprised when they come back as Romans."


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems



On 12/12/2014 06:30 AM, Robert Haas wrote:


Yeah, that would be great, and even better if we could get 2 or 3
positions funded so that the success or failure isn't too much tied to
a single individual.  But even getting 1 position funded in a
stable-enough fashion that someone would be willing to bet on it seems
like a challenge.  (Maybe other people here are less risk-averse than
I am.)


We (not CMD, the community) with proper incentive could fund this. It 
really wouldn't be that hard. That said, there would have to be a clear 
understanding of expectations, results, and authority.


JD






--
Command Prompt, Inc. - http://www.commandprompt.com/  503-667-4564
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, @cmdpromptinc
"If we send our children to Caesar for their education, we should
 not be surprised when they come back as Romans."


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems



On 12/11/2014 02:55 PM, Tom Lane wrote:

Josh Berkus  writes:

How about *you* run the next one, Tom?


I think the limited amount of time I can put into a commitfest is better
spent on reviewing patches than on managing the process.


Agreed but

That means committers/hackers have to suck it up when the manager closes 
the commit fest.


We don't get our cake and eat it too. We either accept that the CFM has 
the authority to do exactly what they are supposed to do, or we don't.

JD


--
Command Prompt, Inc. - http://www.commandprompt.com/  503-667-4564
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, @cmdpromptinc
"If we send our children to Caesar for their education, we should
 not be surprised when they come back as Romans."


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb generator functions

2014-12-12 Thread Andrew Dunstan



On 12/12/2014 01:55 PM, Tom Lane wrote:

Andrew Dunstan  writes:

Also here is a patch factored out which applies the
find_coercion_pathway change to json.c. I'm inclined to say we should
backpatch this to 9.4 (and with a small change 9.3). Thoughts?

Meh.  Maybe I'm just feeling gunshy because I broke something within
the past 24 hours, but at this point (with 9.4.0 wrap only 3 days away)
I'm inclined to avoid any 9.4 code churn that's not clearly necessary.
You argued upthread that this change would not result in any behavioral
changes in which cast method gets selected.  If that's true, then we don't
really need to back-patch; while if it turns out not to be true, we
definitely don't want it in 9.3 and I'd argue it's too late for 9.4 also.

In short, I think it's fine for the 9.4 JSON code to start diverging
from HEAD at this point ...


Ok

cheers

andrew



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Simon Riggs

On 12 December 2014 at 15:10, David Fetter  wrote:
> On Thu, Dec 11, 2014 at 05:55:56PM -0500, Tom Lane wrote:
>> Josh Berkus  writes:
>> > How about *you* run the next one, Tom?
>>
>> I think the limited amount of time I can put into a commitfest is
>> better spent on reviewing patches than on managing the process.

IIRC Tom was pretty much the only person doing patch review for
probably 5 years during 2003-2008, maybe others. AFAICS he was
managing that process. Thank you, Tom.

I've never seen him moan loudly about this, so I'm surprised to hear
such things from people that have done much less.

Any solution to our current problems will come from working together,
not by fighting.

We just need to do more reviews. Realising this, I have begun to do
more. I encourage others to do this also.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

On 12.12.2014 19:07, Bruce Momjian wrote:
> On Fri, Dec 12, 2014 at 10:50:56AM -0500, Tom Lane wrote:
>> Also, one part of the point of the review mechanism is that it's
>> supposed to provide an opportunity for less-senior reviewers to
>> look at parts of the code that they maybe don't know so well, and
>> thereby help grow them into senior people. If we went over to the
>> notion of some one (or a few) senior people doing all the
>> reviewing, it might make the review process more expeditious but it
>> would lose the training aspect. Of course, maybe the training
>> aspect was never worth anything; I'm not in a position to opine on
>> that. But I don't really think that centralizing that 
>> responsibility would be a good thing in the long run.
> 
> That is a very good point --- we have certainly had people doing
> reviews long enough to know if the review process is preparing
> developers for more complex tasks. I don't know the answer myself,
> which might say something.

I can't speak for the others, but for me it certainly is a useful way to
learn new stuff. Maybe not as important as working on my own patches,
but it usually forces me to learn something new, and gives me a
different perspective.

regards
Tomas


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb generator functions

Andrew Dunstan  writes:
>> Also here is a patch factored out which applies the 
>> find_coercion_pathway change to json.c. I'm inclined to say we should 
>> backpatch this to 9.4 (and with a small change 9.3). Thoughts?

Meh.  Maybe I'm just feeling gunshy because I broke something within
the past 24 hours, but at this point (with 9.4.0 wrap only 3 days away)
I'm inclined to avoid any 9.4 code churn that's not clearly necessary.
You argued upthread that this change would not result in any behavioral
changes in which cast method gets selected.  If that's true, then we don't
really need to back-patch; while if it turns out not to be true, we
definitely don't want it in 9.3 and I'd argue it's too late for 9.4 also.

In short, I think it's fine for the 9.4 JSON code to start diverging
from HEAD at this point ...

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Simon Riggs

On 12 December 2014 at 18:04, Bruce Momjian  wrote:

> Well, it seems we need to see some actual cases where compression does
> help before moving forward.  I thought Amit had some amazing numbers for
> WAL compression --- has that changed?

For background processes, like VACUUM, then WAL compression will be
helpful. The numbers show that only applies to FPWs.

I remain concerned about the cost in foreground processes, especially
since the cost will be paid immediately after checkpoint, making our
spikes worse.

What I don't understand is why we aren't working on double buffering,
since that cost would be paid in a background process and would be
evenly spread out across a checkpoint. Plus we'd be able to remove
FPWs altogether, which is like 100% compression.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

On 12/12/2014 06:30 AM, Robert Haas wrote:
> Yeah, that would be great, and even better if we could get 2 or 3
> positions funded so that the success or failure isn't too much tied to
> a single individual.  But even getting 1 position funded in a
> stable-enough fashion that someone would be willing to bet on it seems
> like a challenge.  (Maybe other people here are less risk-averse than
> I am.)

Well, first, who would that person be? Last I checked, all of the senior
committers were spoken for.  I like this idea, but the list of people
who could fill the role is pretty short, and I couldn't possibly start
fundraising unless I had a candidate.

Second, I don't think someone's employment will make a difference in
fixing the commitfest and patch review *process* unless the contributors
agree that it needs fixing and that they are willing to make changes to
their individual workflow to fix it.  Right now there is no consensus
about moving forward in our patch review process; everyone seems to want
the problem to go away without changing anything.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

2014-12-12 Thread David Fetter

On Fri, Dec 12, 2014 at 04:21:43PM +0100, Andres Freund wrote:
> On 2014-12-12 07:10:40 -0800, David Fetter wrote:
> > On Thu, Dec 11, 2014 at 05:55:56PM -0500, Tom Lane wrote:
> > > Josh Berkus  writes:
> > > > How about *you* run the next one, Tom?
> > > 
> > > I think the limited amount of time I can put into a commitfest is
> > > better spent on reviewing patches than on managing the process.
> > 
> > With utmost respect,
> 
> FWIW, the way you frequently use this phrase doesn't come over as
> actually being respectful.

Respect is quantified, and in this case, the most afforded is the most
earned.  In the case of criticizing the work of others without an
offer of help them do it better, respect for that behavior does have
some pretty sharp upper limits, so yes, utmost is in that context.

> > Tom, you seem to carve off an enormous amount of time to follow
> > -bugs and -general.  What say you unsubscribe to those lists for
> > the duration of your tenure as CFM?
> 
> And why on earth would that be a good idea?

Because Tom Lane is not the person whose time is best spent screening
these mailing lists.

> These bugs need to be fixed - we're actually behind on that front.

So you're proposing a bug triage system, which is a separate
discussion.  Let's have that one in a separate thread.

> Are we now really trying to dictate how other developers manage
> their time?

I was merely pointing out that time can be allocated, and that it
appeared it could be allocated from a bucket for which persons less
knowledgeable--perhaps a good bit less knowledgeable--about the entire
code base than Tom are well suited.

> It's one thing to make up rules that say "one review for one commit"
> or something,

And how do you think that would work out.  Are you up for following
it?

> it's something entirely else to try to assign tasks to them.

I was, as I mentioned, merely pointing out that trade-offs are
available.

Cheers,
David.
-- 
David Fetter  http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter  XMPP: david.fet...@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb generator functions

2014-12-12 Thread Andrew Dunstan



On 12/12/2014 01:10 PM, Andrew Dunstan wrote:


On 12/08/2014 01:00 PM, Andrew Dunstan wrote:


On 12/08/2014 04:21 AM, Alvaro Herrera wrote:

Andrew Dunstan wrote:


OK, here is a new patch version that

  * uses find_coercion_path() to find the cast function if any, as
discussed elsewhere
  * removes calls to getTypeOutputInfo() except where required
  * honors a cast to json only for rendering both json and jsonb
  * adds processing for the date type that was previously missing in
datum_to_jsonb

Did this go anywhere?



Not, yet. I hope to get to it this week.





OK, here is a new version.

The major change is that the aggregate final functions now clone the 
transition value rather than modifying it directly, avoiding a similar 
nearby error which Tom fixed recently.


Also here is a patch factored out which applies the 
find_coercion_pathway change to json.c. I'm inclined to say we should 
backpatch this to 9.4 (and with a small change 9.3). Thoughts?




Er this time with patches.

cheers

andrew

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index da138e1..ef69b94 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -10245,9 +10245,10 @@ table2-mapping
 
   
 shows the functions that are
-   available for creating json values.
-   (Currently, there are no equivalent functions for jsonb, but you
-   can cast the result of one of these functions to jsonb.)
+   available for creating json and jsonb values.
+   (There are no equivalent functions for jsonb, of the row_to_json
+   and array_to_json functions. However, the to_jsonb
+   function supplies much the same functionality as these functions would.)
   
 
   
@@ -10268,6 +10269,18 @@ table2-mapping
   
json_object
   
+  
+   to_jsonb
+  
+  
+   jsonb_build_array
+  
+  
+   jsonb_build_object
+  
+  
+   jsonb_object
+  
 
   
 JSON Creation Functions
@@ -10282,17 +10295,18 @@ table2-mapping
  
  
   
+   to_json(anyelement)
+  to_jsonb(anyelement)
+   

- to_json(anyelement)
-   
-   
- Returns the value as JSON.  Arrays and composites are converted
+ Returns the value as json or jsonb.
+ Arrays and composites are converted
  (recursively) to arrays and objects; otherwise, if there is a cast
  from the type to json, the cast function will be used to
- perform the conversion; otherwise, a JSON scalar value is produced.
+ perform the conversion; otherwise, a scalar value is produced.
  For any scalar type other than a number, a Boolean, or a null value,
- the text representation will be used, properly quoted and escaped
- so that it is a valid JSON string.
+ the text representation will be used, in such a fashion that it is a 
+ valid json or jsonb value.

to_json('Fred said "Hi."'::text)
"Fred said \"Hi.\""
@@ -10321,9 +10335,9 @@ table2-mapping
{"f1":1,"f2":"foo"}
   
   
-   
- json_build_array(VARIADIC "any")
-   
+   json_build_array(VARIADIC "any")
+  jsonb_build_array(VARIADIC "any")
+   

  Builds a possibly-heterogeneously-typed JSON array out of a variadic
  argument list.
@@ -10332,9 +10346,9 @@ table2-mapping
[1, 2, "3", 4, 5]
   
   
-   
- json_build_object(VARIADIC "any")
-   
+   json_build_object(VARIADIC "any")
+  jsonb_build_object(VARIADIC "any")
+   

  Builds a JSON object out of a variadic argument list.  By
  convention, the argument list consists of alternating
@@ -10344,9 +10358,9 @@ table2-mapping
{"foo": 1, "bar": 2}
   
   
-   
- json_object(text[])
-   
+   json_object(text[])
+  jsonb_object(text[])
+   

  Builds a JSON object out of a text array.  The array must have either
  exactly one dimension with an even number of members, in which case
@@ -10359,9 +10373,9 @@ table2-mapping
{"a": "1", "b": "def", "c": "3.5"}
   
   
-   
- json_object(keys text[], values text[])
-   
+   json_object(keys text[], values text[])
+  json_object(keys text[], values text[])
+   

  This form of json_object takes keys and values pairwise from two separate
  arrays. In all other respects it is identical to the one-argument form.
@@ -10780,7 +10794,8 @@ table2-mapping
 function json_agg which aggregates record
 values as JSON, and the aggregate function
 json_object_agg which aggregates pairs of values
-into a JSON object.
+into a JSON object, and their jsonb equivalents,
+jsonb_agg and jsonb_object_agg.
   
 
  
@@ -12227,6 +12242,22 @@ NULL baz(3 rows)
  
   

+jsonb_agg
+   
+   jsonb_agg(record)
+  
+  
+   record
+  
+  
+   jsonb
+  
+

Re: [HACKERS] jsonb generator functions

2014-12-12 Thread Andrew Dunstan



On 12/08/2014 01:00 PM, Andrew Dunstan wrote:


On 12/08/2014 04:21 AM, Alvaro Herrera wrote:

Andrew Dunstan wrote:


OK, here is a new patch version that

  * uses find_coercion_path() to find the cast function if any, as
discussed elsewhere
  * removes calls to getTypeOutputInfo() except where required
  * honors a cast to json only for rendering both json and jsonb
  * adds processing for the date type that was previously missing in
datum_to_jsonb

Did this go anywhere?



Not, yet. I hope to get to it this week.





OK, here is a new version.

The major change is that the aggregate final functions now clone the 
transition value rather than modifying it directly, avoiding a similar 
nearby error which Tom fixed recently.


Also here is a patch factored out which applies the 
find_coercion_pathway change to json.c. I'm inclined to say we should 
backpatch this to 9.4 (and with a small change 9.3). Thoughts?


cheers

andrew



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

On Fri, Dec 12, 2014 at 10:50:56AM -0500, Tom Lane wrote:
> Also, one part of the point of the review mechanism is that it's supposed
> to provide an opportunity for less-senior reviewers to look at parts of
> the code that they maybe don't know so well, and thereby help grow them
> into senior people.  If we went over to the notion of some one (or a few)
> senior people doing all the reviewing, it might make the review process
> more expeditious but it would lose the training aspect.  Of course, maybe
> the training aspect was never worth anything; I'm not in a position to
> opine on that.  But I don't really think that centralizing that
> responsibility would be a good thing in the long run.

That is a very good point --- we have certainly had people doing reviews
long enough to know if the review process is preparing developers for
more complex tasks.  I don't know the answer myself, which might say
something.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + Everyone has their own god. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Fri, Dec 12, 2014 at 05:19:42PM +0100, Andres Freund wrote:
> On 2014-12-12 11:15:46 -0500, Robert Haas wrote:
> > On Fri, Dec 12, 2014 at 11:12 AM, Andres Freund  wrote:
> > > On 2014-12-12 11:08:52 -0500, Robert Haas wrote:
> > >> Unless I'm missing something, this test is showing that FPW
> > >> compression saves 298MB of WAL for 17.3 seconds of CPU time, as
> > >> against master.  And compressing the whole record saves a further 1MB
> > >> of WAL for a further 13.39 seconds of CPU time.  That makes
> > >> compressing the whole record sound like a pretty terrible idea - even
> > >> if you get more benefit by reducing the lower boundary, you're still
> > >> burning a ton of extra CPU time for almost no gain on the larger
> > >> records.  Ouch!
> > >
> > > Well, that test pretty much doesn't have any large records besides FPWs
> > > afaics. So it's unsurprising that it's not beneficial.
> > 
> > "Not beneficial" is rather an understatement.  It's actively harmful,
> > and not by a small margin.
> 
> Sure, but that's just because it's too simplistic. I don't think it
> makes sense to make any inference about the worthyness of the general
> approach from the, nearly obvious, fact that compressing every tiny
> record is a bad idea.

Well, it seems we need to see some actual cases where compression does
help before moving forward.  I thought Amit had some amazing numbers for
WAL compression --- has that changed?

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + Everyone has their own god. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

Robert Haas wrote:
> On Fri, Dec 12, 2014 at 11:00 AM, Alvaro Herrera
>  wrote:

> > So let's put the whole bunch under src/bin/ and be done with it.
> 
> I'm not really convinced this is a very good idea.  What do we get out
> of moving everything, or even anything, from contrib?

We show that it's no longer "contrib" (== possibly low quality) stuff
anymore.  At the beginning of pg_upgrade, for example, we didn't want it
in src/bin because it wasn't stable enough, it was full of bugs, there
were always going to be scenarios it wouldn't handle.  Now that is all
gone, so we promote it to the next status level.

> It will make back-patching harder,

Yes.  We can deal with that.  It's not that hard anyway.

> but more importantly, it will possibly create the false impression
> that everything we distribute is on equal footing.

Stuff in contrib is of lower quality.  Some items have improved enough
that we can let them out of that sack now.  What we're doing is create
the correct impression that stuff that's no longer in contrib is of
better quality than what remains in contrib.

> Right now, we've got stuff like vacuumlo in contrib which is
> useful but, let's face it, also a cheap hack.

Then we don't move vacuumlo.  I agree we shouldn't move it.  (And
neither oid2names.)

> If we decide that executables can no longer live in contrib,

Nobody is saying that.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On Fri, Dec 12, 2014 at 10:16:05AM -0500, Tom Lane wrote:
> Peter Eisentraut  writes:
> > On 12/9/14 4:32 PM, Bruce Momjian wrote:
> >> On Tue, Dec  9, 2014 at 06:10:02PM -0300, Alvaro Herrera wrote:
> >>> (For pg_upgrade you also need to do something about pg_upgrade_support,
> >>> which is good because that is one very ugly crock.)
> 
> >> FYI, pg_upgrade_support was segregated from pg_upgrade only because we
> >> wanted separate binary and shared object build/install targets.
> 
> > I think the actual reason is that the makefile structure won't let you
> > have them both in the same directory.  I don't see why you would need
> > separate install targets.
> 
> > How about we move these support functions into the backend?  It's not
> > like we don't already have other pg_upgrade hooks baked in all over the
> > place.
> 
> I don't particularly object to having the C code built into the backend;
> there's not that much of it, and if we could static-ize some of the global
> variables that are involved presently, it'd be a Good Thing IMO.  However,
> the current arrangement makes sure that the function are not accessible
> except during pg_upgrade, and that seems like a Good Thing as well.  So
> I think pg_upgrade should continue to have SQL scripts that create and
> delete the SQL function definitions for these.

Oh, hmmm, would pg_upgrade_support still be a separate shared object
file, or would we just link to functions that already exist in the
backend binary, i.e. it is just the SQL-callabiity you want pg_upgrade
to do?

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + Everyone has their own god. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

Andres Freund  writes:
> On 2014-12-12 11:42:57 -0500, Robert Haas wrote:
>> And I don't know how well git cherry-pick will follow
>> the moves.

> Not well if the patch is done in master first.

FWIW, I always patch master first, and have zero intention of changing
that workflow.  (I have given reasons for that in the past, and don't
feel like repeating them right now.)  So I'm really not on board with
moving code around without *very* good reasons.  This thread hasn't
done very well at coming up with good reasons to move stuff out of
contrib.

In the particular case of pg_upgrade, while it may be now on par
usefulness-wise with src/bin stuff, I think it is and always will
be a special case anyway so far as packagers are concerned; the
reason being that it needs to ride along with back-branch executables.
So I'm not sure that we're making their lives easier by moving it.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PATCH: hashjoin - gracefully increasing NTUP_PER_BUCKET instead of batching

On 12.12.2014 14:19, Robert Haas wrote:
> On Thu, Dec 11, 2014 at 5:46 PM, Tomas Vondra  wrote:
>
>> Regarding the "sufficiently small" - considering today's hardware, we're
>> probably talking about gigabytes. On machines with significant memory
>> pressure (forcing the temporary files to disk), it might be much lower,
>> of course. Of course, it also depends on kernel settings (e.g.
>> dirty_bytes/dirty_background_bytes).
> 
> Well, this is sort of one of the problems with work_mem.  When we
> switch to a tape sort, or a tape-based materialize, we're probably far
> from out of memory.  But trying to set work_mem to the amount of
> memory we have can easily result in a memory overrun if a load spike
> causes lots of people to do it all at the same time.  So we have to
> set work_mem conservatively, but then the costing doesn't really come
> out right.  We could add some more costing parameters to try to model
> this, but it's not obvious how to get it right.

Ummm, I don't think that's what I proposed. What I had in mind was a
flag "the batches are likely to stay in page cache". Because when it is
likely, batching is probably faster (compared to increased load factor).

Tomas


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On 2014-12-12 11:42:57 -0500, Robert Haas wrote:
> > I think the amount of effort a simple renamed directory which wholly
> > contains a binary creates is acceptable. Just use patch -p4 instead of
> > patch -p1...
> 
> That is fine if you are manually applying a patch that touches only
> that directory, but if the patch also touches other stuff then it's
> not as simple.

I think backpatchable commits that touch individual binaries and other
code at the same time are (and ought to be!) pretty rare.

> And I don't know how well git cherry-pick will follow
> the moves.

Not well if the patch is done in master first.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On Fri, Dec 12, 2014 at 11:40 AM, Andres Freund  wrote:
> The benefit of moving relevant stuff is that it'll actually be installed
> by default when installing postgres on many platforms. That's currently
> often not the case. The contrib umbrella, as used by many other
> projects, actually justifies not doing so.

Agreed.  See my other response for my thoughts on that topic.

>> It will make back-patching harder
>
> I think the amount of effort a simple renamed directory which wholly
> contains a binary creates is acceptable. Just use patch -p4 instead of
> patch -p1...

That is fine if you are manually applying a patch that touches only
that directory, but if the patch also touches other stuff then it's
not as simple.  And I don't know how well git cherry-pick will follow
the moves.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On Fri, Dec 12, 2014 at 11:26 AM, Tom Lane  wrote:
> Robert Haas  writes:
>> I'm not really convinced this is a very good idea.  What do we get out
>> of moving everything, or even anything, from contrib?  It will make
>> back-patching harder, but more importantly, it will possibly create
>> the false impression that everything we distribute is on equal
>> footing.  Right now, we've got stuff like vacuumlo in contrib which is
>> useful but, let's face it, also a cheap hack.  If we decide that
>> executables can no longer live in contrib, then every time somebody
>> submits something in the future, we've got to decide whether it
>> deserves parity with psql and pg_dump or whether we shouldn't include
>> it at all.  contrib is a nice middle-ground.
>
> Yeah, that's a good point.  I think part of the motivation here is the
> thought that some of these programs, like pg_upgrade, *should* now be
> considered on par with pg_dump et al.  But it does not follow that
> everything in contrib is, or should be, on that level.

Yeah.  We have put enough effort collectively into pg_upgrade that I
think it's fair to say that it is on a part with pg_dump.  I still
think the architecture there is awfully fragile and we should try to
improve it, but it's very widely-used and people rely on it to work,
which it generally does.  And certainly we have put a lot of sweat
into making it work.

I would also say that pg_archivecleanup is a fundamental server tool
and that it belongs in src/bin.

But after that, I get fuzzy.  For me, the next tier of things would
consist of pgbench, pg_test_fsync, pg_test_timing, and pg_xlogdump.
Those are all useful, but I would also classify them as optional.  If
you are running a PostgreSQL installation, you definitely need initdb
and postgres and pg_dump and pg_dumpall and psql, but you don't
definitely need these.  I think they are all robust enough to go in
src/bin, but they are not as necessary as much of the stuff that is in
that directory today, so it's unclear to me whether we want to put
them there.

Finally, there is the stuff that is either hacky or deprecated:
oid2name, pg_standby, vacuumlo.  Putting that stuff in src/bin clearly
makes no sense IMV.  But I wouldn't necessarily want to remove it all
either.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On 2014-12-12 11:14:56 -0500, Robert Haas wrote:
> I'm not really convinced this is a very good idea.  What do we get out
> of moving everything, or even anything, from contrib?

The benefit of moving relevant stuff is that it'll actually be installed
by default when installing postgres on many platforms. That's currently
often not the case. The contrib umbrella, as used by many other
projects, actually justifies not doing so.

I don't think that's a good argument for moving everything, rather the
contrary, but relevant stuff that we properly support should imo be
moved.

> It will make back-patching harder

I think the amount of effort a simple renamed directory which wholly
contains a binary creates is acceptable. Just use patch -p4 instead of
patch -p1...

> Right now, we've got stuff like vacuumlo in contrib which is
> useful but, let's face it, also a cheap hack.

On the other hand, we really don't provide any other solution. Since
large objects are part of core we really ought to provide at least some
support for cleanup.

> If we decide that executables can no longer live in contrib, then
> every time somebody submits something in the future, we've got to
> decide whether it deserves parity with psql and pg_dump or whether we
> shouldn't include it at all.  contrib is a nice middle-ground.

I think it makes sense to still have it as a middleground for future
things.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

Robert Haas  writes:
> I'm not really convinced this is a very good idea.  What do we get out
> of moving everything, or even anything, from contrib?  It will make
> back-patching harder, but more importantly, it will possibly create
> the false impression that everything we distribute is on equal
> footing.  Right now, we've got stuff like vacuumlo in contrib which is
> useful but, let's face it, also a cheap hack.  If we decide that
> executables can no longer live in contrib, then every time somebody
> submits something in the future, we've got to decide whether it
> deserves parity with psql and pg_dump or whether we shouldn't include
> it at all.  contrib is a nice middle-ground.

Yeah, that's a good point.  I think part of the motivation here is the
thought that some of these programs, like pg_upgrade, *should* now be
considered on par with pg_dump et al.  But it does not follow that
everything in contrib is, or should be, on that level.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin



On 12/08/2014 07:50 PM, Tom Lane wrote:

Peter Eisentraut  writes:

Last time this was attempted, the discussion got lost in exactly which
extensions are worthy enough to be considered official or something like
that.  I want to dodge that here by starting at the opposite end:
1. move programs to src/bin/



Here are the contrib programs:



oid2name
pg_archivecleanup
pg_standby
pg_test_fsync
pg_test_timing
pg_upgrade
pg_xlogdump
pgbench
vacuumlo



The proposal would basically be to mv contrib/$x src/bin/$x and also
move the reference pages in the documentation.


Personally, I'm good with moving pg_archivecleanup, pg_standby,
pg_upgrade, pg_xlogdump, and pgbench this way.  (Although wasn't there
just some discussion about pg_standby being obsolete?  If so, shouldn't
we remove it instead of promoting it?)  As for the others:



Let's not forget pg_upgrade which is arguably the most important of 
everything listed.


JD


--
Command Prompt, Inc. - http://www.commandprompt.com/  503-667-4564
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, @cmdpromptinc
"If we send our children to Caesar for their education, we should
 not be surprised when they come back as Romans."


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On 2014-12-12 11:15:46 -0500, Robert Haas wrote:
> On Fri, Dec 12, 2014 at 11:12 AM, Andres Freund  wrote:
> > On 2014-12-12 11:08:52 -0500, Robert Haas wrote:
> >> Unless I'm missing something, this test is showing that FPW
> >> compression saves 298MB of WAL for 17.3 seconds of CPU time, as
> >> against master.  And compressing the whole record saves a further 1MB
> >> of WAL for a further 13.39 seconds of CPU time.  That makes
> >> compressing the whole record sound like a pretty terrible idea - even
> >> if you get more benefit by reducing the lower boundary, you're still
> >> burning a ton of extra CPU time for almost no gain on the larger
> >> records.  Ouch!
> >
> > Well, that test pretty much doesn't have any large records besides FPWs
> > afaics. So it's unsurprising that it's not beneficial.
> 
> "Not beneficial" is rather an understatement.  It's actively harmful,
> and not by a small margin.

Sure, but that's just because it's too simplistic. I don't think it
makes sense to make any inference about the worthyness of the general
approach from the, nearly obvious, fact that compressing every tiny
record is a bad idea.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [Bug] Inconsistent result for inheritance and FOR UPDATE.

Etsuro Fujita  writes:
>> (2014/12/12 10:37), Tom Lane wrote:
>>> Yeah, this is clearly a thinko: really, nothing in the planner should
>>> be using get_parse_rowmark().  I looked around for other errors of the
>>> same type and found that postgresGetForeignPlan() is also using
>>> get_parse_rowmark().  While that's harmless at the moment because we
>>> don't support foreign tables as children, it's still wrong.  Will
>>> fix that too.

> While updating the inheritance patch, I noticed that the fix for
> postgresGetForeignPlan() is not right.  Since PlanRowMarks for foreign
> tables get the ROW_MARK_COPY markType during preprocess_rowmarks(), so
> we can't get the locking strength from the PlanRowMarks, IIUC.

Ugh, you're right.

> In order
> to get the locking strength, I think we need to see the RowMarkClauses
> and thus still need to use get_parse_rowmark() in
> postgresGetForeignPlan(), though I agree with you that that is ugly.

I think this needs more thought; I'm still convinced that having the FDW
look at the parse rowmarks is the Wrong Thing.  However, we don't need
to solve it in existing branches.  With 9.4 release so close, the right
thing is to revert that change for now and consider a HEAD-only patch
later.  (One idea is to go ahead and make a ROW_MARK_COPY item, but
add a field to PlanRowMark to record the original value.  We should
probably also think about allowing FDWs to change these settings if
they want to.  The real source of trouble here is that planner.c
has a one-size-fits-all approach to row locking for FDWs; and we're
now seeing that that one size doesn't fit postgres_fdw.)

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Fri, Dec 12, 2014 at 11:12 AM, Andres Freund  wrote:
> On 2014-12-12 11:08:52 -0500, Robert Haas wrote:
>> Unless I'm missing something, this test is showing that FPW
>> compression saves 298MB of WAL for 17.3 seconds of CPU time, as
>> against master.  And compressing the whole record saves a further 1MB
>> of WAL for a further 13.39 seconds of CPU time.  That makes
>> compressing the whole record sound like a pretty terrible idea - even
>> if you get more benefit by reducing the lower boundary, you're still
>> burning a ton of extra CPU time for almost no gain on the larger
>> records.  Ouch!
>
> Well, that test pretty much doesn't have any large records besides FPWs
> afaics. So it's unsurprising that it's not beneficial.

"Not beneficial" is rather an understatement.  It's actively harmful,
and not by a small margin.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On Fri, Dec 12, 2014 at 11:00 AM, Alvaro Herrera
 wrote:
>> I'm pretty much -1 on relocating anything that's under src/bin already.

I agree.  I can't see packagers putting it anywhere except for
$SOMETHING/bin in the final install, so what do we get out of dividing
it up in some weird way in our tree?

> So let's put the whole bunch under src/bin/ and be done with it.

I'm not really convinced this is a very good idea.  What do we get out
of moving everything, or even anything, from contrib?  It will make
back-patching harder, but more importantly, it will possibly create
the false impression that everything we distribute is on equal
footing.  Right now, we've got stuff like vacuumlo in contrib which is
useful but, let's face it, also a cheap hack.  If we decide that
executables can no longer live in contrib, then every time somebody
submits something in the future, we've got to decide whether it
deserves parity with psql and pg_dump or whether we shouldn't include
it at all.  contrib is a nice middle-ground.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On 2014-12-12 11:08:52 -0500, Robert Haas wrote:
> Unless I'm missing something, this test is showing that FPW
> compression saves 298MB of WAL for 17.3 seconds of CPU time, as
> against master.  And compressing the whole record saves a further 1MB
> of WAL for a further 13.39 seconds of CPU time.  That makes
> compressing the whole record sound like a pretty terrible idea - even
> if you get more benefit by reducing the lower boundary, you're still
> burning a ton of extra CPU time for almost no gain on the larger
> records.  Ouch!

Well, that test pretty much doesn't have any large records besides FPWs
afaics. So it's unsurprising that it's not beneficial.

Greetings,

Andres Freund


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Fri, Dec 12, 2014 at 10:04 AM, Andres Freund  wrote:
>> Note that autovacuum and fsync are off.
>> =# select phase, user_diff, system_diff,
>> pg_size_pretty(pre_update - pre_insert),
>> pg_size_pretty(post_update - pre_update) from results;
>>phase| user_diff | system_diff | pg_size_pretty |
>> pg_size_pretty
>> +---+-++
>>  Compression FPW| 42.990799 |0.868179 | 429 MB | 567 MB
>>  No compression | 25.688731 |1.236551 | 429 MB | 727 MB
>>  Compression record | 56.376750 |0.769603 | 429 MB | 566 MB
>> (3 rows)
>> If we do record-level compression, we'll need to be very careful in
>> defining a lower-bound to not eat unnecessary CPU resources, perhaps
>> something that should be controlled with a GUC. I presume that this stands
>> true as well for the upper bound.
>
> Record level compression pretty obviously would need a lower boundary
> for when to use compression. It won't be useful for small heapam/btree
> records, but it'll be rather useful for large multi_insert, clean or
> similar records...

Unless I'm missing something, this test is showing that FPW
compression saves 298MB of WAL for 17.3 seconds of CPU time, as
against master.  And compressing the whole record saves a further 1MB
of WAL for a further 13.39 seconds of CPU time.  That makes
compressing the whole record sound like a pretty terrible idea - even
if you get more benefit by reducing the lower boundary, you're still
burning a ton of extra CPU time for almost no gain on the larger
records.  Ouch!

(Of course, I'm assuming that Michael's patch is reasonably efficient,
which might not be true.)

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

Tom Lane wrote:
> Heikki Linnakangas  writes:
> > On 12/12/2014 03:07 PM, Peter Eisentraut wrote:
> >> On 12/9/14 4:10 PM, Alvaro Herrera wrote:
> >>> Maybe it makes sense to have a distinction between client programs and
> >>> server programs.  Can we have src/sbin/ and move stuff that involves the
> >>> server side in there?  I think that'd be pg_xlogdump, pg_archivecleanup,
> >>> pg_upgrade, pg_test_timing, pg_test_fsync.  (If we were feeling bold we
> >>> could also move pg_resetxlog, pg_controldata and initdb there.)
> 
> >> I was thinking about that.  What do others think?
> 
> > Sounds good. We already separate server and client programs in the docs, 
> > and packagers put them in different packages too. This should make 
> > packagers' life a little bit easier in the long run.
> 
> I'm pretty much -1 on relocating anything that's under src/bin already.

So let's put the whole bunch under src/bin/ and be done with it.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

Robert Haas  writes:
> On Fri, Dec 12, 2014 at 9:15 AM, Alvaro Herrera
>  wrote:
>> Robert Haas wrote:
>>> (I note that the proposal to have the CFM review everything is merely
>>> one way of meeting the need to have senior people spend more time
>>> reviewing.  But I assure all of you that I spend as much time
>>> reviewing as I can find time for.  If someone wants to pay me the same
>>> salary I'm making now to do nothing but review patches, I'll think
>>> about it.  But even then, that would also mean that I wasn't spending
>>> time writing patches of my own.)

>> I have heard the idea of a "cross-company PostgreSQL foundation" of some
>> sort that would hire a developer just to manage commitfests, do patch
>> reviews, apply bugfixes, etc, without the obligations that come from
>> individual companies' schedules for particular development roadmaps,
>> customer support, and the like.  Of course, only a senior person would
>> be able to fill this role because it requires considerable experience.

> Yeah, that would be great, and even better if we could get 2 or 3
> positions funded so that the success or failure isn't too much tied to
> a single individual.  But even getting 1 position funded in a
> stable-enough fashion that someone would be willing to bet on it seems
> like a challenge.  (Maybe other people here are less risk-averse than
> I am.)

Yeah, it would be hard to sell anyone on that unless the foundation
was so well funded that it could clearly afford to keep paying you
for years into the future.

I'm not really on board with the CFM-reviews-everything idea anyway.
I don't think that can possibly work well, because it supposes that senior
reviewers are interchangeable, which they aren't.  Everybody's got pieces
of the system that they know better than other pieces.

Also, one part of the point of the review mechanism is that it's supposed
to provide an opportunity for less-senior reviewers to look at parts of
the code that they maybe don't know so well, and thereby help grow them
into senior people.  If we went over to the notion of some one (or a few)
senior people doing all the reviewing, it might make the review process
more expeditious but it would lose the training aspect.  Of course, maybe
the training aspect was never worth anything; I'm not in a position to
opine on that.  But I don't really think that centralizing that
responsibility would be a good thing in the long run.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Turning recovery.conf into GUCs

2014-12-12 Thread Alex Shulgin

Alex Shulgin  writes:

> Alex Shulgin  writes:
>>
>> Here's an attempt to revive this patch.
>
> Here's the patch rebased against current HEAD, that is including the
> recently committed action_at_recovery_target option.
>
> The default for the new GUC is 'pause', as in HEAD, and
> pause_at_recovery_target is removed completely in favor of it.
>
> I've also taken the liberty to remove that part that errors out when
> finding $PGDATA/recovery.conf.  Now get your rotten tomatoes ready. ;-)

This was rather short-sighted, so I've restored that part.

Also, rebased on current HEAD, following the rename of
action_at_recovery_target to recovery_target_action.

--
Alex



recovery_guc_v5.5.patch.gz
Description: application/gzip

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On 2014-12-12 10:20:58 -0500, Tom Lane wrote:
> Peter Eisentraut  writes:
> > On 12/12/14 8:13 AM, Andres Freund wrote:
> >> Wouldn't a make install-server/client targets or something similar
> >> actually achieve the same thing? Seems simpler to maintain to me.
> 
> > Adding non-standard makefile targets comes with its own set of
> > maintenance issues.
> 
> It would be of zero value to packagers anyway; certainly so for those
> following the Red Hat tradition, in which you tell the package Makefile
> to install everything and then what goes into which subpackage is
> sorted out in a separate, subsequent step.  Possibly Debian or other
> packaging infrastructures do it differently, but I doubt that.

Debian has that step as well - you don't really have to use it, but the
postgres debian packages do so. They already don't adhere to the current
distinction.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

On 2014-12-12 07:10:40 -0800, David Fetter wrote:
> On Thu, Dec 11, 2014 at 05:55:56PM -0500, Tom Lane wrote:
> > Josh Berkus  writes:
> > > How about *you* run the next one, Tom?
> > 
> > I think the limited amount of time I can put into a commitfest is
> > better spent on reviewing patches than on managing the process.
> 
> With utmost respect,

FWIW, the way you frequently use this phrase doesn't come over as
actually being respectful.

> Tom, you seem to carve off an enormous amount of
> time to follow -bugs and -general.  What say you unsubscribe to those
> lists for the duration of your tenure as CFM?

And why on earth would that be a good idea? These bugs need to be fixed
- we're actually behind on that front. Are we now really trying to
dictate how other developers manage their time? It's one thing to make
up rules that say "one review for one commit" or something, it's
something entirely else to try to assign tasks to them.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

Peter Eisentraut  writes:
> On 12/12/14 8:13 AM, Andres Freund wrote:
>> Wouldn't a make install-server/client targets or something similar
>> actually achieve the same thing? Seems simpler to maintain to me.

> Adding non-standard makefile targets comes with its own set of
> maintenance issues.

It would be of zero value to packagers anyway; certainly so for those
following the Red Hat tradition, in which you tell the package Makefile
to install everything and then what goes into which subpackage is
sorted out in a separate, subsequent step.  Possibly Debian or other
packaging infrastructures do it differently, but I doubt that.

Really, if we want to tell packagers that foo is a client program and
bar is a server-side program, the documentation is where to address it.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

Peter Eisentraut  writes:
> On 12/9/14 4:32 PM, Bruce Momjian wrote:
>> On Tue, Dec  9, 2014 at 06:10:02PM -0300, Alvaro Herrera wrote:
>>> (For pg_upgrade you also need to do something about pg_upgrade_support,
>>> which is good because that is one very ugly crock.)

>> FYI, pg_upgrade_support was segregated from pg_upgrade only because we
>> wanted separate binary and shared object build/install targets.

> I think the actual reason is that the makefile structure won't let you
> have them both in the same directory.  I don't see why you would need
> separate install targets.

> How about we move these support functions into the backend?  It's not
> like we don't already have other pg_upgrade hooks baked in all over the
> place.

I don't particularly object to having the C code built into the backend;
there's not that much of it, and if we could static-ize some of the global
variables that are involved presently, it'd be a Good Thing IMO.  However,
the current arrangement makes sure that the function are not accessible
except during pg_upgrade, and that seems like a Good Thing as well.  So
I think pg_upgrade should continue to have SQL scripts that create and
delete the SQL function definitions for these.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

Heikki Linnakangas  writes:
> On 12/12/2014 03:07 PM, Peter Eisentraut wrote:
>> On 12/9/14 4:10 PM, Alvaro Herrera wrote:
>>> Maybe it makes sense to have a distinction between client programs and
>>> server programs.  Can we have src/sbin/ and move stuff that involves the
>>> server side in there?  I think that'd be pg_xlogdump, pg_archivecleanup,
>>> pg_upgrade, pg_test_timing, pg_test_fsync.  (If we were feeling bold we
>>> could also move pg_resetxlog, pg_controldata and initdb there.)

>> I was thinking about that.  What do others think?

> Sounds good. We already separate server and client programs in the docs, 
> and packagers put them in different packages too. This should make 
> packagers' life a little bit easier in the long run.

I'm pretty much -1 on relocating anything that's under src/bin already.
The history mess and back-patching pain would outweigh any notional
cleanliness --- and AFAICS it's entirely notional.  As an ex-packager
I can tell you that where stuff sits in the source tree makes precisely
*zero* difference to a packager.  She's going to do "make install-world"
and then her package recipe will list out which files in the install tree
go into which sub-package.  Perhaps it would get clearer to packagers if
we also installed stuff into $INSTALLDIR/sbin, but I doubt that such a
change is going to fly with anyone else.  The bin vs sbin distinction
is not universal.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] moving from contrib to bin

On 2014-12-12 11:27:01 -0300, Alvaro Herrera wrote:
> We already have src/bin/; the mixture of "src/" and "bin/" predates us.
> Of course, the stuff we keep in there is not binaries but source code
> that produces binaries.
> 
> As for src/sbin/, we wouldn't install anything to the system's
> /usr/sbin/ of course, only /usr/bin/, just like the stuff in src/bin/.
> But it would be slightly more clear what we keep in each src/ subdir.

I think sbin is a spectactularly bad name, let's not go there. If
anything, make it srvbin or something like that.

> I think our current src/bin/ is a misnomer, but it seems late to fix
> that.  In a greenfield I think we could have "src/clients/" and
> "src/srvtools/" or something like that, and everything would install to
> /usr/bin.  Then there would be no doubt where to move each program from
> contrib.

Maybe. We could just do that now - git's file change tracking is good
enough for that kind of move.

> Maybe there is no point to all of this and we should just move it all to
> src/bin/ as originally proposed, which is simpler anyway.

+1. Packagers already don't use the current boundaries for packaging...

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

2014-12-12 Thread David Fetter

On Thu, Dec 11, 2014 at 05:55:56PM -0500, Tom Lane wrote:
> Josh Berkus  writes:
> > How about *you* run the next one, Tom?
> 
> I think the limited amount of time I can put into a commitfest is
> better spent on reviewing patches than on managing the process.

With utmost respect, Tom, you seem to carve off an enormous amount of
time to follow -bugs and -general.  What say you unsubscribe to those
lists for the duration of your tenure as CFM?

Cheers,
David.
-- 
David Fetter  http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter  XMPP: david.fet...@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_rewind in contrib

Heikki Linnakangas  writes:
> I'd like to include pg_rewind in contrib.

I don't object to adding the tool as such, but let's wait to see what
happens with Peter's proposal to move contrib command-line tools into
src/bin/.  If it should be there it'd be less code churn if it went
into there in the first place.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On 2014-12-12 23:50:43 +0900, Michael Paquier wrote:
> I got curious to see how the compression of an entire record would perform
> and how it compares for small WAL records, and here are some numbers based
> on the patch attached, this patch compresses the whole record including the
> block headers, letting only XLogRecord out of it with a flag indicating
> that the record is compressed (note that this patch contains a portion for
> replay untested, still this patch gives an idea on how much compression of
> the whole record affects user CPU in this test case). It uses a buffer of 4
> * BLCKSZ, if the record is longer than that compression is simply given up.
> Those tests are using the hack upthread calculating user and system CPU
> using getrusage() when a backend.
> 
> Here is the simple test case I used with 512MB of shared_buffers and small
> records, filling up a bunch of buffers, dirtying them and them compressing
> FPWs with a checkpoint.
> #!/bin/bash
> psql < SELECT pg_backend_pid();
> CREATE TABLE aa (a int);
> CREATE TABLE results (phase text, position pg_lsn);
> CREATE EXTENSION IF NOT EXISTS pg_prewarm;
> ALTER TABLE aa SET (FILLFACTOR = 50);
> INSERT INTO results VALUES ('pre-insert', pg_current_xlog_location());
> INSERT INTO aa VALUES (generate_series(1,700)); -- 484MB
> SELECT pg_size_pretty(pg_relation_size('aa'::regclass));
> SELECT pg_prewarm('aa'::regclass);
> CHECKPOINT;
> INSERT INTO results VALUES ('pre-update', pg_current_xlog_location());
> UPDATE aa SET a = 700 + a;
> CHECKPOINT;
> INSERT INTO results VALUES ('post-update', pg_current_xlog_location());
> SELECT * FROM results;
> EOF
> 
> Note that autovacuum and fsync are off.
> =# select phase, user_diff, system_diff,
> pg_size_pretty(pre_update - pre_insert),
> pg_size_pretty(post_update - pre_update) from results;
>phase| user_diff | system_diff | pg_size_pretty |
> pg_size_pretty
> +---+-++
>  Compression FPW| 42.990799 |0.868179 | 429 MB | 567 MB
>  No compression | 25.688731 |1.236551 | 429 MB | 727 MB
>  Compression record | 56.376750 |0.769603 | 429 MB | 566 MB
> (3 rows)
> If we do record-level compression, we'll need to be very careful in
> defining a lower-bound to not eat unnecessary CPU resources, perhaps
> something that should be controlled with a GUC. I presume that this stands
> true as well for the upper bound.

Record level compression pretty obviously would need a lower boundary
for when to use compression. It won't be useful for small heapam/btree
records, but it'll be rather useful for large multi_insert, clean or
similar records...

Greetings,

Andres Freund


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_rewind in contrib

2014-12-12 Thread Heikki Linnakangas


On 12/12/2014 04:20 PM, Andres Freund wrote:

Not sure if the copyright notices in the current form are actually ok?


Hmm. We do have such copyright notices in the source tree, but I know 
that we're trying to avoid it in new code. They had to be there when the 
code lived as a separate project, but now that I'm contributing this to 
PostgreSQL proper, I can remove them if necessary.


- Heikki



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On 2014-12-12 09:46:13 -0500, Bruce Momjian wrote:
> On Fri, Dec 12, 2014 at 03:27:33PM +0100, Andres Freund wrote:
> > On 2014-12-12 09:24:27 -0500, Bruce Momjian wrote:
> > > On Fri, Dec 12, 2014 at 03:22:24PM +0100, Andres Freund wrote:
> > > > > Well, the larger question is why wouldn't we just have the user 
> > > > > compress
> > > > > the entire WAL file before archiving --- why have each backend do it? 
> > > > > Is it the write volume we are saving?  I though this WAL compression
> > > > > gave better performance in some cases.
> > > > 
> > > > Err. Streaming?
> > > 
> > > Well, you can already set up SSL for compression while streaming.  In
> > > fact, I assume many are already using SSL for streaming as the majority
> > > of SSL overhead is from connection start.
> > 
> > That's not really true. The overhead of SSL during streaming is
> > *significant*. Both the kind of compression it does (which is far more
> > expensive than pglz or lz4) and the encyrption itself. In many cases
> > it's prohibitively expensive - there's even a fair number on-list
> > reports about this.
> 
> Well, I am just trying to understand when someone would benefit from WAL
> compression.  Are we saying it is only useful for non-SSL streaming?

No, not at all. It's useful in a lot more situations:

* The amount of WAL in pg_xlog can make up a significant portion of a
  database's size. Especially in large OLTP databases. Compressing
  archives doesn't help with that.
* The original WAL volume itself can be quite problematic because at
  some point its exhausting the underlying IO subsystem. Both due to the
  pure write rate and to the fsync()s regularly required.
* ssl compression can often not be used for WAL streaming because it's
  too slow as it's uses a much more expensive algorithm. Which is why we
  even have a GUC to disable it.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Wed, Dec 10, 2014 at 11:25 PM, Bruce Momjian  wrote:

> On Wed, Dec 10, 2014 at 07:40:46PM +0530, Rahila Syed wrote:
> > The tests ran for around 30 mins.Manual checkpoint was run before each
> test.
> >
> > Compression   WAL generated%compressionLatency-avg   CPU usage
> > (seconds)  TPS
>  Latency
> > stddev
> >
> >
> > on  1531.4 MB  ~35 %  7.351 ms
>
> >   user diff: 562.67s system diff: 41.40s  135.96
>
> >   13.759 ms
> >
> >
> > off  2373.1 MB 6.781
> ms
> >   user diff: 354.20s  system diff: 39.67s147.40
>
> >   14.152 ms
> >
> > The compression obtained is quite high close to 35 %.
> > CPU usage at user level when compression is on is quite noticeably high
> as
> > compared to that when compression is off. But gain in terms of reduction
> of WAL
> > is also high.
>
> I am sorry but I can't understand the above results due to wrapping.
> Are you saying compression was twice as slow?
>

I got curious to see how the compression of an entire record would perform
and how it compares for small WAL records, and here are some numbers based
on the patch attached, this patch compresses the whole record including the
block headers, letting only XLogRecord out of it with a flag indicating
that the record is compressed (note that this patch contains a portion for
replay untested, still this patch gives an idea on how much compression of
the whole record affects user CPU in this test case). It uses a buffer of 4
* BLCKSZ, if the record is longer than that compression is simply given up.
Those tests are using the hack upthread calculating user and system CPU
using getrusage() when a backend.

Here is the simple test case I used with 512MB of shared_buffers and small
records, filling up a bunch of buffers, dirtying them and them compressing
FPWs with a checkpoint.
#!/bin/bash
psql  SizeOfXLogRecord)
+		{
+			memcpy(uncompressed_buffer, hdr_rdt.data + SizeOfXLogRecor

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Fri, Dec 12, 2014 at 03:27:33PM +0100, Andres Freund wrote:
> On 2014-12-12 09:24:27 -0500, Bruce Momjian wrote:
> > On Fri, Dec 12, 2014 at 03:22:24PM +0100, Andres Freund wrote:
> > > > Well, the larger question is why wouldn't we just have the user compress
> > > > the entire WAL file before archiving --- why have each backend do it? 
> > > > Is it the write volume we are saving?  I though this WAL compression
> > > > gave better performance in some cases.
> > > 
> > > Err. Streaming?
> > 
> > Well, you can already set up SSL for compression while streaming.  In
> > fact, I assume many are already using SSL for streaming as the majority
> > of SSL overhead is from connection start.
> 
> That's not really true. The overhead of SSL during streaming is
> *significant*. Both the kind of compression it does (which is far more
> expensive than pglz or lz4) and the encyrption itself. In many cases
> it's prohibitively expensive - there's even a fair number on-list
> reports about this.

Well, I am just trying to understand when someone would benefit from WAL
compression.  Are we saying it is only useful for non-SSL streaming?

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + Everyone has their own god. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Rahila Syed

Hello,

>Well, the larger question is why wouldn't we just have the user compress
>the entire WAL file before archiving --- why have each backend do it?
>Is it the write volume we are saving?

IIUC,  the idea here is to not only save the on disk size of WAL but to
reduce the overhead of flushing WAL records to disk in servers with heavy
write operations. So yes improving the performance by saving write volume
is a part of the requirement.

Thank you,
Rahila Syed


On Fri, Dec 12, 2014 at 7:48 PM, Bruce Momjian  wrote:
>
> On Fri, Dec 12, 2014 at 08:27:59AM -0500, Robert Haas wrote:
> > On Thu, Dec 11, 2014 at 11:34 AM, Bruce Momjian 
> wrote:
> > >> compression = 'on'  : 1838 secs
> > >> = 'off' : 1701 secs
> > >>
> > >> Different is around 140 secs.
> > >
> > > OK, so the compression took 2x the cpu and was 8% slower.  The only
> > > benefit is WAL files are 35% smaller?
> >
> > Compression didn't take 2x the CPU.  It increased user CPU from 354.20
> > s to 562.67 s over the course of the run, so it took about 60% more
> > CPU.
> >
> > But I wouldn't be too discouraged by that.  At least AIUI, there are
> > quite a number of users for whom WAL volume is a serious challenge,
> > and they might be willing to pay that price to have less of it.  Also,
> > we have talked a number of times before about incorporating Snappy or
> > LZ4, which I'm guessing would save a fair amount of CPU -- but the
> > decision was made to leave that out of the first version, and just use
> > pg_lz, to keep the initial patch simple.  I think that was a good
> > decision.
>
> Well, the larger question is why wouldn't we just have the user compress
> the entire WAL file before archiving --- why have each backend do it?
> Is it the write volume we are saving?  I though this WAL compression
> gave better performance in some cases.
>
> --
>   Bruce Momjian  http://momjian.us
>   EnterpriseDB http://enterprisedb.com
>
>   + Everyone has their own god. +
>

Re: [HACKERS] pg_rewind in contrib

On Fri, Dec 12, 2014 at 11:13 PM, Heikki Linnakangas
 wrote:
> I'd like to include pg_rewind in contrib. I originally wrote it as an
> external project so that I could quickly get it working with the existing
> versions, and because I didn't feel it was quite ready for production use
> yet. Now, with the WAL format changes in master, it is a lot more
> maintainable than before. Many bugs have been fixed since the first
> prototypes, and I think it's fairly robust now.
>
> I propose that we include pg_rewind in contrib/ now. Attached is a patch for
> that. It just includes the latest sources from the current pg_rewind
> repository at https://github.com/vmware/pg_rewind. It is released under the
> PostgreSQL license.
>
> For those who are not familiar with pg_rewind, it's a tool that allows
> repurposing an old master server as a new standby server, after promotion,
> even if the old master was not shut down cleanly. That's a very often
> requested feature.
Indeed the code got quite cleaner with the new WAL API. Btw, gitignore
has many unnecessary entries.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Compression of full-page-writes

On Fri, Dec 12, 2014 at 9:34 AM, Michael Paquier
 wrote:
>> I don't think that's a cost worth caring about.
> OK, I thought it was.

Space on the heap that never gets used is basically free.  The OS
won't actually allocate physical memory unless the pages are actually
accessed.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commitfest problems

On Fri, Dec 12, 2014 at 9:15 AM, Alvaro Herrera
 wrote:
> Robert Haas wrote:
>> (I note that the proposal to have the CFM review everything is merely
>> one way of meeting the need to have senior people spend more time
>> reviewing.  But I assure all of you that I spend as much time
>> reviewing as I can find time for.  If someone wants to pay me the same
>> salary I'm making now to do nothing but review patches, I'll think
>> about it.  But even then, that would also mean that I wasn't spending
>> time writing patches of my own.)
>
> I have heard the idea of a "cross-company PostgreSQL foundation" of some
> sort that would hire a developer just to manage commitfests, do patch
> reviews, apply bugfixes, etc, without the obligations that come from
> individual companies' schedules for particular development roadmaps,
> customer support, and the like.  Of course, only a senior person would
> be able to fill this role because it requires considerable experience.
>
> Probably this person should be allowed to work on their own patches if
> they so desire; otherwise there is a risk that experience dilutes.
> Also, no single company should dictate what this person's priorities
> are, other than general guidelines: general stability, submitted patches
> get attention, bugs get closed, releases get out, coffee gets brewed.

Yeah, that would be great, and even better if we could get 2 or 3
positions funded so that the success or failure isn't too much tied to
a single individual.  But even getting 1 position funded in a
stable-enough fashion that someone would be willing to bet on it seems
like a challenge.  (Maybe other people here are less risk-averse than
I am.)

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Compression of full-page-writes

On Fri, Dec 12, 2014 at 9:15 AM, Michael Paquier
 wrote:
> I just meant that the scratch buffers used to store temporarily the
> compressed and uncompressed data should be palloc'd all the time, even
> if the switch is off.

If they're fixed size, you can just put them on the heap as static globals.

static char space_for_stuff[65536];

Or whatever you need.

I don't think that's a cost worth caring about.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Compression of full-page-writes

On Fri, Dec 12, 2014 at 11:32 PM, Robert Haas  wrote:
> On Fri, Dec 12, 2014 at 9:15 AM, Michael Paquier
>  wrote:
>> I just meant that the scratch buffers used to store temporarily the
>> compressed and uncompressed data should be palloc'd all the time, even
>> if the switch is off.
>
> If they're fixed size, you can just put them on the heap as static globals.
> static char space_for_stuff[65536];
Well sure :)

> Or whatever you need.
> I don't think that's a cost worth caring about.
OK, I thought it was.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_rewind in contrib