Re: [HACKERS] Remove or weaken hints about "effective resolution of sleep delays is 10 ms"?

2016-02-16 Thread Andres Freund
On 2016-02-16 09:13:09 +0100, Andres Freund wrote: > What we do we think the resolution is on modern > >systems? I would not have guessed that to be inaccurate. > > Depends in a lot of factors. The biggest being how busy you're system > is. On an mostly idle system (i.e

Re: [HACKERS] pglogical - logical replication contrib module

2016-02-17 Thread Andres Freund
k this is much less a fundamental, and more an implementation issue. Falling back to just re-replicating the table, and then optimizing a few common cases (only immutable DEFALUT/USING involved) should be enough for a while. Lets get the basics right, before reaching for the moon. Greetings,

Re: [HACKERS] Re: [COMMITTERS] pgsql: Introduce group locking to prevent parallel processes from deadl

2016-02-17 Thread Andres Freund
On 2016-02-16 23:33:47 -0500, Tom Lane wrote: > Yes, exactly. I'm not certain if there are any real platforms where > a pointer-sized write wouldn't be atomic (it sure sounds inefficient > for that to be true), but we have not assumed that to date and I'd > just as soon not start here. FWIW, ther

Re: [HACKERS] exposing pg_controldata and pg_config as functions

2016-02-18 Thread Andres Freund
n't understand why you're so opposed to this. Several people said that they're interested in this information in the current discussion and it has been requested repeatedly over the years. For superusers you can already hack access, but it's darn ugly. Greetings

Re: [HACKERS] pg_ctl promote wait

2016-02-18 Thread Andres Freund
Hi, On 2016-02-17 21:47:50 -0500, Peter Eisentraut wrote: > It would be nice if pg_ctl promote supported the -w (wait) option. > > How could pg_ctl determine when the promotion has finished? How about looking into pg_control? ControlFileData->state ought to have the correct information. Regards

Re: [HACKERS] checkpointer continuous flushing - V16

2016-02-18 Thread Andres Freund
On 2016-02-11 19:44:25 +0100, Andres Freund wrote: > The first two commits of the series are pretty close to being ready. I'd > welcome review of those, and I plan to commit them independently of the > rest as they're beneficial independently. The most important bits are >

Re: [HACKERS] checkpointer continuous flushing - V16

2016-02-18 Thread Andres Freund
On 2016-02-18 09:51:20 +0100, Fabien COELHO wrote: > I've looked at these patches, especially the whole bench of explanations and > comments which is a good source for understanding what is going on in the > WAL writer, a part of pg I'm not familiar with. > > When reading the patch 0002 explanatio

Re: [HACKERS] Relaxing SSL key permission checks

2016-02-18 Thread Andres Freund
On 2016-02-18 10:17:49 -0500, Tom Lane wrote: > Christoph Berg writes: > > Currently the server insists on ssl_key_file's permissions to be 0600 > > or less, and be owned by the database user. Debian has been patching > > be-secure.c since forever (the git history goes back to 8.2beta1) to > > rel

Re: [HACKERS] checkpointer continuous flushing - V16

2016-02-19 Thread Andres Freund
Hi, On 2016-02-19 10:16:41 +0100, Fabien COELHO wrote: > Below the results of a lot of tests with pgbench to exercise checkpoints on > the above version when fetched. Wow, that's a great test series. > Overall comments: > - sorting & flushing is basically always a winner > - benchmarking with

Re: [HACKERS] Typo in bufmgr.c that result in waste of memory

2016-02-19 Thread Andres Freund
Hi, Nice catch! On February 19, 2016 2:42:08 PM GMT+01:00, Tom Lane wrote: >> I think we should fix it, but not backpatch. > >I don't think that's particularly good policy. It's a clear bug, why >would we not fix it? Leaving it as-is in the back branches can have >no good effect, and what it d

Re: [HACKERS] PostgreSQL Audit Extension

2016-02-19 Thread Andres Freund
On 2016-02-19 10:14:47 -0500, Bruce Momjian wrote: > We have already hesitated to record DDL changes for > logical replication because of the code size, maintenance overhead, and > testing required. I'm not sure what you're referring to here? It'd be some relatively minor code surgery to also pass

Re: [HACKERS] pg_ctl promote wait

2016-02-19 Thread Andres Freund
On 2016-02-19 13:48:52 -0500, Peter Eisentraut wrote: > Is it safe to read pg_control externally without a lock? pg_controldata > will just report a CRC error and proceed, and if you're not sure you can > just run it again. But if a promotion fails every so often because of > concurrent pg_contro

Re: [HACKERS] pg_ctl promote wait

2016-02-19 Thread Andres Freund
On 2016-02-20 00:06:09 +0900, Fujii Masao wrote: > One concern is that there can be a "time" after the pg_control's state > is changed to DB_IN_PRODUCTION and before the server is able to > start accepting normal (not read-only) connections. So if users try to > start write transaction just after p

Re: [HACKERS] FDW: should GetFdwRoutine be called when drop table?

2016-02-19 Thread Andres Freund
On 2016-02-19 14:18:19 -0500, Peter Eisentraut wrote: > On 2/19/16 12:21 PM, Feng Tian wrote: > > I have an fdw that each foreign table will acquire some persisted resource. > > In my case, some files in file system. To drop the table cleanly, I > > have written > > an object_access_hook that rem

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-19 Thread Andres Freund
On 2016-02-04 16:54:58 +0100, Andres Freund wrote: > Hi, > > Fabien asked me to post a new version of the checkpoint flushing patch > series. While this isn't entirely ready for commit, I think we're > getting closer. > > I don't want to post a full serie

Re: [HACKERS] pg_ctl promote wait

2016-02-19 Thread Andres Freund
On 2016-02-19 15:09:58 -0500, Tom Lane wrote: > Peter Eisentraut writes: > > On 2/19/16 10:06 AM, Fujii Masao wrote: > >> One concern is that there can be a "time" after the pg_control's state > >> is changed to DB_IN_PRODUCTION and before the server is able to > >> start accepting normal (not rea

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-19 Thread Andres Freund
On 2016-02-19 22:46:44 +0100, Fabien COELHO wrote: > > Hello Andres, > > >Here's the next two (the most important) patches of the series: > >0001: Allow to trigger kernel writeback after a configurable number of > >writes. > >0002: Checkpoint sorting and balancing. > > I will look into these tw

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-20 Thread Andres Freund
Hi, On 2016-02-20 20:56:31 +0100, Fabien COELHO wrote: > >* Currently *_flush_after can be set to a nonzero value, even if there's > > no support for flushing on that platform. Imo that's ok, but perhaps > > other people's opinion differ. > > In some previous version I think a warning was shown o

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-21 Thread Andres Freund
;argument to make. > > Yep. My thinking is that doing things in the sleeping interval does not > interfere with the checkpointer scheduling, so it is less likely to go wrong > and falling behind. I don't really see why that's the case. Triggering writeback every N writes doesn't really influence the scheduling in a bad way - the flushing is done *before* computing the sleep time. Triggering the writeback *after* computing the sleep time, and then sleep for that long, in addition of the time for sync_file_range, skews things more. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-21 Thread Andres Freund
e" stuff. I can try to do > that, but it takes time to do so, if someone can test on other OS it would > be much better. I think that if it works it should be kept in, so it is just > a matter of testing it. I'm not arguing for ripping it out, what I mean is that we don't set a nondefault value for the GUCs on platforms with just posix_fadivise available... Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-22 Thread Andres Freund
?id=23d0127096cb91cb6d354bdc71bd88a7bae3a1d5 If you use your 12.04 kernel, that'd not be fixed. Which might be a reason to do it as you suggest. Could you share the exact details of that workload? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-22 Thread Andres Freund
On 2016-02-22 11:05:20 -0500, Tom Lane wrote: > Andres Freund writes: > > Interesting. That doesn't reflect my own tests, even on rotating media, > > at all. I wonder if it's related to: > > https://git.kernel.org/cgit/linux/kernel/gi

Re: [HACKERS] Performance degradation in commit 6150a1b0

2016-02-26 Thread Andres Freund
Hi, On 2016-02-25 12:56:39 +0530, Amit Kapila wrote: > From past few weeks, we were facing some performance degradation in the > read-only performance bench marks in high-end machines. My colleague > Mithun, has tried by reverting commit ac1d794 which seems to degrade the > performance in HEAD on

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-02-26 Thread Andres Freund
On 2016-02-02 13:12:50 +0300, Alexander Korotkov wrote: > On Tue, Feb 2, 2016 at 12:43 AM, Andres Freund wrote: > > > On 2016-02-01 13:06:57 +0300, Alexander Korotkov wrote: > > > On Mon, Feb 1, 2016 at 11:34 AM, Alexander Korotkov < > > > a.korot...@postg

Re: [HACKERS] Proposal: Generic WAL logical messages

2016-02-26 Thread Andres Freund
data += len; > } > break; > + case REORDER_BUFFER_CHANGE_MESSAGE: > + { > + Sizemessage_size = > change->data.msg.message_size; > + Size

Re: [HACKERS] LWLocks in DSM memory

2016-08-17 Thread Andres Freund
On 2016-08-16 23:09:07 -0400, Robert Haas wrote: > On Tue, Aug 16, 2016 at 5:03 PM, Andres Freund wrote: > > On 2016-08-15 18:15:23 -0400, Robert Haas wrote: > >> On Thu, Aug 11, 2016 at 2:19 PM, Robert Haas wrote: > >> > Therefore, I plan to commit thi

Re: [HACKERS] LWLocks in DSM memory

2016-08-17 Thread Andres Freund
On 2016-08-17 08:31:36 -0400, Robert Haas wrote: > On Tue, Aug 16, 2016 at 5:03 PM, Andres Freund wrote: > > On 2016-08-15 18:15:23 -0400, Robert Haas wrote: > >> On Thu, Aug 11, 2016 at 2:19 PM, Robert Haas wrote: > >> > Therefore, I plan to commit thi

Re: [HACKERS] CLUSTER, reform_and_rewrite_tuple(), and parallelism

2016-08-17 Thread Andres Freund
> Does anyone have any ideas on how to: > > 1). Directly address the reform_and_rewrite_tuple() bottleneck. What part of is actually the expensive bit? It does a whole lot of things. Forming/Deforming tuples, the hash lookups in rewrite_heap_tuple(), ...? Andres -- Sent via pgsql-hackers mail

Re: [HACKERS] CLUSTER, reform_and_rewrite_tuple(), and parallelism

2016-08-17 Thread Andres Freund
On 2016-08-17 16:23:29 -0700, Peter Geoghegan wrote: > On Wed, Aug 17, 2016 at 4:16 PM, Andres Freund wrote: > >> Does anyone have any ideas on how to: > >> > >> 1). Directly address the reform_and_rewrite_tuple() bottleneck. > > > > What part of is actua

Re: [HACKERS] CLUSTER, reform_and_rewrite_tuple(), and parallelism

2016-08-17 Thread Andres Freund
On 2016-08-17 16:58:49 -0700, Peter Geoghegan wrote: > On Wed, Aug 17, 2016 at 4:28 PM, Andres Freund wrote: > > Could you also provide a strace -ttt -T -c and a cpu cycles flamegraph? > > Here is the output from that strace invocation, plus a -p (to attach > to th

Re: [HACKERS] CLUSTER, reform_and_rewrite_tuple(), and parallelism

2016-08-17 Thread Andres Freund
On 2016-08-17 21:17:56 -0300, Alvaro Herrera wrote: > Peter Geoghegan wrote: > > > This doesn't seem that interesting, but not sure what you're looking for. > > > > I also attach cycles flamegraph. > > I may be blind, but what are those write() calls attributed to > heap_form_tuple? libc isn't

Re: [HACKERS] CLUSTER, reform_and_rewrite_tuple(), and parallelism

2016-08-17 Thread Andres Freund
On 2016-08-17 17:35:32 -0700, Peter Geoghegan wrote: > On Wed, Aug 17, 2016 at 5:20 PM, Andres Freund wrote: > > libc isn't compiled with -fno-omit-frame-pointer (and even if, it uses > > assembly without setup of the frame pointer), so frame pointer based > > call gra

Re: [HACKERS] Changed SRF in targetlist handling

2016-08-17 Thread Andres Freund
On 2016-08-03 20:22:03 -0700, Andres Freund wrote: > On 2016-08-02 16:30:55 -0700, Andres Freund wrote: > > > > Besides that I'm structurally wondering whether turning the original > > > > query into a subquery is the right thing to do. It requires some kind o

Re: [HACKERS] Patch: initdb: "'" for QUOTE_PATH (non-windows)

2016-08-17 Thread Andres Freund
On 2016-08-18 09:14:44 +0900, Michael Paquier wrote: > On Thu, Aug 18, 2016 at 12:21 AM, Ryan Murphy wrote: > > I have created a better patch (attached) that correctly escapes the shell > > arguments using PQExpBufferStr and the appendShellString function, as per > > Michael and Andres' suggestion

Re: [HACKERS] Patch: initdb: "'" for QUOTE_PATH (non-windows)

2016-08-17 Thread Andres Freund
On August 17, 2016 8:15:56 PM PDT, Michael Paquier wrote: >+{ /* pg_ctl command w path, properly quoted */ >+PQExpBuffer pg_ctl_path = createPQExpBuffer(); >+printfPQExpBuffer(pg_ctl_path, "%s%spg_ctl", >+bin_dir, >+(strlen(bin_di

Re: [HACKERS] Pluggable storage

2016-08-18 Thread Andres Freund
On August 18, 2016 7:44:50 AM PDT, Ants Aasma wrote: >On Tue, Aug 16, 2016 at 9:46 PM, Andres Freund >wrote: >> On 2016-08-15 12:02:18 -0400, Robert Haas wrote: >>> I am somewhat inclined to >>> believe that we need to restructure the executor in a bigger wa

Re: [HACKERS] Pluggable storage

2016-08-18 Thread Andres Freund
On 2016-08-18 08:58:11 +0100, Simon Riggs wrote: > On 16 August 2016 at 19:46, Andres Freund wrote: > > On 2016-08-15 12:02:18 -0400, Robert Haas wrote: > >> Thanks for taking a stab at this. I'd like to throw out a few concerns. > >> > >> One, I

Re: [HACKERS] WIP: About CMake v2

2016-08-18 Thread Andres Freund
On 2016-08-18 15:55:20 -0400, Christian Convey wrote: > * Allow the CMake-based build system to assume a fairly modern version > of CMake. (Maybe 2.8.12, or 3.0.) > > * For systems where the minimum CMake version isn't readily available, > have an alternative build system which is just a simplist

Re: [HACKERS] Patch: initdb: "'" for QUOTE_PATH (non-windows)

2016-08-18 Thread Andres Freund
On 2016-08-18 16:11:27 -0400, Robert Haas wrote: > On Wed, Aug 17, 2016 at 11:18 PM, Andres Freund wrote: > > On August 17, 2016 8:15:56 PM PDT, Michael Paquier > > wrote: > > > >>+{ /* pg_ctl command w path, properly quoted */ > >&

Re: [HACKERS] errno clobbering in reorderbuffer

2016-08-18 Thread Andres Freund
On 2016-08-18 19:06:02 -0300, Alvaro Herrera wrote: > While researching a customer issue with BDR I noticed that one ereport() > call happens after clobbering errno, leading to the wrong strerror being > reported. This patch fixes it by saving before calling > CloseTransientFile and restoring afte

Re: [HACKERS] errno clobbering in reorderbuffer

2016-08-18 Thread Andres Freund
Hi, On 2016-08-18 19:06:02 -0300, Alvaro Herrera wrote: > if (write(fd, rb->outbuf, ondisk->size) != ondisk->size) > { > + int save_errno = errno; > + > CloseTransientFile(fd); > + errno = save_errno; > ereport(ERROR, >

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-18 Thread Andres Freund
On 2016-06-30 18:14:15 -0700, Peter Geoghegan wrote: > On Tue, Dec 15, 2015 at 10:04 AM, Andres Freund wrote: > > Took a while. But here we go. The attached version is a significantly > > revised version of my earlier patch. Notably I've pretty much entirely > > revise

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-18 Thread Andres Freund
On 2016-08-18 17:27:59 -0700, Peter Geoghegan wrote: > On Thu, Aug 18, 2016 at 5:26 PM, Andres Freund wrote: > > Rebased version attached. A review would be welcome. Plan to push this > > forward otherwise in the not too far away future. > > I can review this next week.

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-18 Thread Andres Freund
On 2016-08-18 17:35:47 -0700, Peter Geoghegan wrote: > On Thu, Aug 18, 2016 at 5:28 PM, Andres Freund wrote: > >> I can review this next week. > > > > Thanks > > Given the time frame that you have in mind, I won't revisit the > question the parallel CLUS

Re: [HACKERS] errno clobbering in reorderbuffer

2016-08-18 Thread Andres Freund
On August 18, 2016 7:21:03 PM PDT, Alvaro Herrera wrote: >Andres Freund wrote: >> On 2016-08-18 19:06:02 -0300, Alvaro Herrera wrote: >> > While researching a customer issue with BDR I noticed that one >ereport() >> > call happens after clobbering errno, leadin

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

2016-08-19 Thread Andres Freund
On August 19, 2016 2:50:30 AM PDT, Aleksander Alekseev wrote: >Heikki, Peter, thanks a lot for code review! > >> What's going on here? Surely pg_atomic_init_u64() should initialize >> the value? > >It's because of how pg_atomic_exchange_u64_impl is implemented: > >``` >while (true) >{ >o

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

2016-08-19 Thread Andres Freund
On 2016-08-19 17:55:25 -0400, Tom Lane wrote: > It'd be useful also to figure out why our existing valgrind testing has > not caught this already. The example you give looks like it surely > ought to be replicated well enough in the standard regression tests. The valgrind suppression file explici

Re: [HACKERS] Should we cacheline align PGXACT?

2016-08-21 Thread Andres Freund
On 2016-08-20 14:33:13 -0400, Robert Haas wrote: > On Aug 19, 2016, at 2:12 AM, Alexander Korotkov > wrote: > > Hackers, > > > > originally this idea was proposed by Andres Freund while experimenting with > > lockfree Pin/UnpinBuffer [1]. > > The patch is at

Re: [HACKERS] Should we cacheline align PGXACT?

2016-08-21 Thread Andres Freund
On 2016-08-19 09:46:12 -0400, Tom Lane wrote: > Alexander Korotkov writes: > > originally this idea was proposed by Andres Freund while experimenting with > > lockfree Pin/UnpinBuffer [1]. > > The patch is attached as well as results of pgbench -S on 72-cores > > machin

Re: [HACKERS] Should we cacheline align PGXACT?

2016-08-21 Thread Andres Freund
On 2016-08-22 11:25:55 +0800, Craig Ringer wrote: > On 22 August 2016 at 10:40, Andres Freund wrote: > > > On 2016-08-19 09:46:12 -0400, Tom Lane wrote: > > > Alexander Korotkov writes: > > > > originally this idea was proposed by Andres Freund while experiment

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-22 Thread Andres Freund
27;t > hit that now. I suspect you'd still see contention on bigger hardware, > though, my laptop has oly 4 cores. I'll have to find a real server for the > next round of testing. Yea, I think that's true. I can just about see ProcArrayLock contention on my more powerful lapto

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

2016-08-22 Thread Andres Freund
On 2016-08-22 13:16:34 -0400, Robert Haas wrote: > On Sat, Aug 20, 2016 at 3:46 PM, Tom Lane wrote: > > So to me, it seems like the core of this complaint boils down to "my > > sanitizer doesn't understand the valgrind exclusion patterns that have > > been created for Postgres". We can address th

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-22 Thread Andres Freund
On 2016-08-22 13:41:57 -0400, Robert Haas wrote: > On Mon, Aug 22, 2016 at 1:32 PM, Heikki Linnakangas wrote: > >> But what about the best case? If we create a scenario where there are > >> no open read-write transactions at all and (somehow) lots and lots of > >> ProcArrayLock contention, how mu

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

2016-08-22 Thread Andres Freund
On 2016-08-22 13:49:47 -0400, Robert Haas wrote: > On Mon, Aug 22, 2016 at 1:46 PM, Andres Freund wrote: > > I don't think the runtime overhead is likely to be all that high - if > > you look at valgrind.supp the peformancecritical parts basically are: > > - pgstat_send

Re: [HACKERS] New SQL counter statistics view (pg_stat_sql)

2016-08-22 Thread Andres Freund
On 2016-08-22 13:54:43 -0400, Robert Haas wrote: > On Sat, Aug 20, 2016 at 11:17 AM, Tom Lane wrote: > > I'm inclined to suggest you forget this approach and propose a single > > counter for "SQL commands executed", which avoids all of the above > > definitional problems. People who need more det

Re: [HACKERS] Changed SRF in targetlist handling

2016-08-22 Thread Andres Freund
On 2016-08-17 17:41:28 -0700, Andres Freund wrote: > Tom, do you think this is roughly going in the right direction? My plan > here is to develop two patches, to come before this: > > a) Allow to avoid using a tuplestore for SRF_PERCALL SRFs in ROWS FROM - >otherwise our pe

Re: [HACKERS] Changed SRF in targetlist handling

2016-08-22 Thread Andres Freund
Hi, On 2016-05-23 09:26:03 +0800, Craig Ringer wrote: > SRFs-in-tlist are a lot faster for lockstep iteration etc. They're also > much simpler to write, though if the result result rowcount differs > unexpectedly between the functions you get exciting and unexpected > behaviour. > > WITH ORDINALI

Re: [HACKERS] Logical decoding of sequence advances, part II

2016-08-22 Thread Andres Freund
On 2016-08-22 16:29:12 -0400, Robert Haas wrote: > So, I wish I could give you some better advice on this topic, but > sadly I am not an expert in this area. However, it seems to me that > this is just one facet of a much more general problem: given two > transactions T1 and T2, the order of repla

Re: [HACKERS] [Patch] Temporary tables that do not bloat pg_catalog (a.k.a fast temp tables)

2016-08-22 Thread Andres Freund
On 2016-08-22 17:50:11 -0300, Alvaro Herrera wrote: > > 2. You can't write to unlogged tables on standby servers, so this > > doesn't help solve the problem of wanting to use temporary tables on > > standbys. > > Check. We could think about relaxing this restriction, which would > enable the feat

[HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2016-08-22 Thread Andres Freund
- maybe we should forbid that? As the patch currently stands, the diffstat is 56 files changed, 953 insertions(+), 1599 deletions(-) which isn't bad. I'd guess that a few more lines are needed, but I'd still bet it's a net negative code-wise. Regards, Andres Freund [1] http:/

Re: [HACKERS] Changed SRF in targetlist handling

2016-08-22 Thread Andres Freund
Hi, On 2016-08-22 16:20:58 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2016-08-17 17:41:28 -0700, Andres Freund wrote: > >> Tom, do you think this is roughly going in the right direction? > > I've not had time to look at this patch, I'm afraid. If

Re: [HACKERS] Block level parallel vacuum WIP

2016-08-23 Thread Andres Freund
On 2016-08-23 12:17:30 -0400, Robert Haas wrote: > On Tue, Aug 23, 2016 at 11:17 AM, Alvaro Herrera > wrote: > > Robert Haas wrote: > >> 2. When you finish the heap scan, or when the array of dead tuple IDs > >> is full (or very nearly full?), perform a cycle of index vacuuming. > >> For now, have

Re: [HACKERS] Logical decoding of sequence advances, part II

2016-08-23 Thread Andres Freund
On 2016-08-23 07:26:31 -0500, Kevin Grittner wrote: > On Tue, Aug 23, 2016 at 7:10 AM, Kevin Grittner wrote: > > On Mon, Aug 22, 2016 at 6:39 PM, Craig Ringer wrote: > > >> Could you provide an example of a case where xacts replayed in > >> commit order will produce incorrect results? > > > > ht

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-23 Thread Andres Freund
using Unix domain > sockets instead of TCP to connect. Apparently those things add enough > overhead to mask out the little difference. To make the problem more apparent, how are the differences for something as extreme as just a SELECT 1; or a SELECT 1 FROM pgbench_accounts WHERE aid = 1; G

Re: [HACKERS] Slowness of extended protocol

2016-08-23 Thread Andres Freund
ssue without requiring version-dependant adaptions in all client drivers. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [BUGS] BUG #14244: wrong suffix for pg_size_pretty()

2016-08-23 Thread Andres Freund
On 2016-08-23 14:33:15 -0400, Bruce Momjian wrote: > On Tue, Aug 23, 2016 at 02:31:26PM -0400, Robert Haas wrote: > > On Tue, Aug 23, 2016 at 1:57 PM, Bruce Momjian wrote: > > > That's why I was asking you to comment on the final patch, which I am > > > planning to apply to PG 10 soon. > > > > Oh

Re: [HACKERS] Changed SRF in targetlist handling

2016-08-23 Thread Andres Freund
On 2016-08-17 17:41:28 -0700, Andres Freund wrote: > a) Allow to avoid using a tuplestore for SRF_PERCALL SRFs in ROWS FROM - >otherwise our performance would regress noticeably in some cases. To demonstrate the problem: master: =# COPY (SELECT generate_series(1, 5000)) TO 

Re: [HACKERS] dump/restore doesn't preserve row ordering?

2016-08-23 Thread Andres Freund
On 2016-08-23 17:22:03 -0400, Tom Lane wrote: > I happened to notice, while experimenting with the data set used > in the SPGIST-for-inet thread, that loading the supplied pg_dump > script and immediately dumping it does not reproduce the row order > appearing in the original dump script. I though

Re: [HACKERS] [Patch] Temporary tables that do not bloat pg_catalog (a.k.a fast temp tables)

2016-08-23 Thread Andres Freund
On 2016-08-23 19:18:04 -0300, Claudio Freire wrote: > On Tue, Aug 23, 2016 at 7:11 PM, Tomas Vondra > wrote: > > Could someone please explain how the unlogged tables are supposed to fix the > > catalog bloat problem, as stated in the initial patch submission? We'd still > > need to insert/delete t

Re: [HACKERS] [Patch] Temporary tables that do not bloat pg_catalog (a.k.a fast temp tables)

2016-08-23 Thread Andres Freund
On 2016-08-23 19:33:33 -0300, Claudio Freire wrote: > On Tue, Aug 23, 2016 at 7:20 PM, Andres Freund wrote: > >> Wouldn't more aggressive vacuuming of catalog tables fix the bloat? > > > > Not really in my experience, at least not without more drastic vacuum > >

Re: [HACKERS] recent compiler warnings

2016-08-23 Thread Andres Freund
On 2016-08-23 19:54:39 -0700, Jeff Janes wrote: > Sorry for starting a new thread, I can't find the correct one to reply to. > > Using: gcc version 4.8.3 20140627 [gcc-4_8-branch revision 212064] (SUSE > Linux) > > commit ed0097e4f9e6b1 has introduced two compiler warnings: > > gistutil.c: In f

Re: [HACKERS] [Patch] Temporary tables that do not bloat pg_catalog (a.k.a fast temp tables)

2016-08-24 Thread Andres Freund
On August 24, 2016 9:32:48 AM PDT, Tomas Vondra wrote: > > >On 08/24/2016 12:20 AM, Andres Freund wrote: >> On 2016-08-23 19:18:04 -0300, Claudio Freire wrote: >>> On Tue, Aug 23, 2016 at 7:11 PM, Tomas Vondra >>> wrote: >>>> Could someone please

Re: [HACKERS] increasing the default WAL segment size

2016-08-24 Thread Andres Freund
On 2016-08-24 22:33:49 -0400, Tom Lane wrote: > > Possibly it would make sense for this to be configurable at initdb > > time instead of requiring a recompile; > > ... but I think this is just folly. You'd have to do major amounts > of work to keep, eg, slave servers on the same page as the maste

Re: [HACKERS] increasing the default WAL segment size

2016-08-24 Thread Andres Freund
On 2016-08-24 23:26:51 -0400, Robert Haas wrote: > On Wed, Aug 24, 2016 at 10:54 PM, Andres Freund wrote: > > and I'm also rather doubtful that it's actually without overhead. > > Really? Where do you think the overhead would come from? ATM we do a math involving

Re: [HACKERS] increasing the default WAL segment size

2016-08-24 Thread Andres Freund
On 2016-08-25 00:28:58 -0400, Robert Haas wrote: > On Wed, Aug 24, 2016 at 11:52 PM, Andres Freund wrote: > > On 2016-08-24 23:26:51 -0400, Robert Haas wrote: > >> On Wed, Aug 24, 2016 at 10:54 PM, Andres Freund wrote: > >> > and I'm also rather doubtful

Re: [HACKERS] increasing the default WAL segment size

2016-08-25 Thread Andres Freund
On 2016-08-25 13:45:29 -0400, Robert Haas wrote: > I think you may be forgetting that "the base 3 WAL segments" is no > longer the default configuration. checkpoint_segments=3 is history; > we now have max_wal_size=1GB, which is a maximum of 64 WAL segments, > not 3. Well, but min_wal_size still

Re: [HACKERS] increasing the default WAL segment size

2016-08-25 Thread Andres Freund
On 2016-08-26 13:07:09 +0900, Michael Paquier wrote: > On Fri, Aug 26, 2016 at 12:54 PM, Amit Kapila wrote: > > On Fri, Aug 26, 2016 at 9:04 AM, Michael Paquier > > wrote: > >> On Fri, Aug 26, 2016 at 12:25 PM, Amit Kapila > >> wrote: > >>> If we change the default to 64MB, then I think it won'

Re: [HACKERS] Renaming some binaries

2016-08-26 Thread Andres Freund
On 2016-08-26 22:01:58 +0200, Simon Riggs wrote: > On 26 August 2016 at 18:26, Euler Taveira wrote: > > > I'm bringing this $subject into discussion again. Historically, we are > > carrying binary names that have been confused newbies. createuser is the > > worst name so for. Also, names like cre

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-26 Thread Andres Freund
On 2016-08-26 11:53:21 -0400, Peter Eisentraut wrote: > On 8/25/16 10:39 PM, Michael Paquier wrote: > > I am relaunching $subject as 10 development will begin soon. As far as > > I know, there is agreement that we can do something here. Among the > > different proposals I have found: > > - pg_clog

Re: [HACKERS] Why is a newly created index contains the invalid LSN?

2016-08-26 Thread Andres Freund
On 2016-08-26 18:46:42 +0300, Yury Zhuravlev wrote: > Thanks all. > Now understand LSN strongly connected with WAL. > However how difficult put last system LSN instead 0? > It's not so important but will allow make use LSN more consistent. Maybe explain why you're interested in page lsns, that'd p

Re: [HACKERS] Renaming some binaries

2016-08-26 Thread Andres Freund
On 2016-08-26 13:26:39 -0300, Euler Taveira wrote: > I'm bringing this $subject into discussion again. Historically, we are > carrying binary names that have been confused newbies. createuser is the > worst name so for. Also, names like createdb, initdb, reindexdb, and > droplang does not suggest w

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-26 Thread Andres Freund
On 2016-08-26 17:11:00 -0400, Peter Eisentraut wrote: > On 8/26/16 12:28 PM, Tom Lane wrote: > > Also, I'd just as soon not move/rename things > > that don't really need it. > > I'm just as happy with not changing anything. But if we're going to > rename stuff, let's at least think about something

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-26 Thread Andres Freund
On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote: > I agree with all that. But the subject line is specifically about > moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog, > then that is noted. But if we were to move it, we can think about a > good place to move it to. I t

Re: [HACKERS] Bogus sizing parameters in some AllocSetContextCreate calls

2016-08-27 Thread Andres Freund
Hi, On 2016-08-27 14:08:44 -0400, Tom Lane wrote: > The standard calling pattern for AllocSetContextCreate is > Barring objection, I propose to make these changes in HEAD and 9.6. > I don't feel a great need to adjust the back branches --- although there > might be some argument for adding these n

Re: [HACKERS] Bogus sizing parameters in some AllocSetContextCreate calls

2016-08-27 Thread Andres Freund
On 2016-08-27 15:36:28 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2016-08-27 14:08:44 -0400, Tom Lane wrote: > >> Barring objection, I propose to make these changes in HEAD and 9.6. > > > I think we might also / instead want to think about r

Re: [HACKERS] Bogus sizing parameters in some AllocSetContextCreate calls

2016-08-27 Thread Andres Freund
On 2016-08-27 15:46:26 -0400, Tom Lane wrote: > (Or in other words, the fact that "DefaultContextCreate" is spelled > "AllocSetContextCreate" is a bit of a historical artifact, but I do > not see why changing the spelling is a useful exercise.) Well, if you're going through nearly all of the insta

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2016-08-27 Thread Andres Freund
On 2016-08-27 14:48:29 -0700, Andres Freund wrote: > My next steps are to work on cleaning up the code a bit more, and > increase the regression coverage. Oh, there's one open item I actually don't really know how to handle well: A decent way of enforcing the join order between

Re: [HACKERS] VACUUM's ancillary tasks

2016-08-28 Thread Andres Freund
e was an attempt late in the 9.5 cycle, but Bruce (IIRC) ran out of steam. And nobody picked it up again ... :( Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] Transaction traceability - txid_status(bigint)

2016-08-28 Thread Andres Freund
Hi, On 2016-08-29 11:25:39 +0800, Craig Ringer wrote: > ERROR: could not access status of transaction 778793573 > DETAIL: could not open file "pg_clog/02E6": No such file or directory > > What I'd really like is to be able to ask transam.c to handle the > xid_in_recent_past logic, treating

Re: [HACKERS] PostgreSQL Version 10, missing minor version

2016-08-28 Thread Andres Freund
On 2016-08-29 11:41:00 +0800, Craig Ringer wrote: > On 29 August 2016 at 02:52, Tom Lane wrote: > > "Regina Obe" writes: > >> The routine in PostGIS to parse out the version number from pg_config is > >> breaking in the 10 cycle > > > > TBH, I wonder why you are doing that in the first place; it

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-29 Thread Andres Freund
by far, mean it's ephemeral. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Fix pg_receivexlog --synchronous

2016-08-29 Thread Andres Freund
On 2016-08-29 07:34:52 -0400, Tom Lane wrote: > Simon Riggs writes: > > Fix pg_receivexlog --synchronous > > The buildfarm says you broke the 9.5 branch. > > In general, pushing inessential patches just a few hours before a wrap > deadline is a dangerous business. Pushing them without any testi

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2016-08-29 Thread Andres Freund
On 2016-08-29 12:56:25 +0300, Heikki Linnakangas wrote: > On 08/28/2016 12:48 AM, Andres Freund wrote: > > Attached is a significantly updated patch series (see the mail one up > > for details about what this is, I don't want to quote it in its > > entirety). > >

Re: [HACKERS] [WIP] Patches to enable extraction state of query execution from external session

2016-08-29 Thread Andres Freund
y patch introduces > specific flag *runtime* that indicates whether we explain running query and > does some insertions in source code dedicated to output the statistics of > running query. Unless I'm missing something this doesn't really expose a user of this functionality

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-29 Thread Andres Freund
On 2016-08-29 12:07:51 -0400, David Steele wrote: > >> pg_replslot -> pg_tmp/pg_repslot > > > > That's most certainly not ephemeral. Just because something isn't > > generally appropriate on a standby, doesn't, by far, mean it's ephemeral. > > Yes, ephemeral was a poor choice of words. I really

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-29 Thread Andres Freund
On 2016-08-29 19:27:29 -0500, Jim Nasby wrote: > On 8/27/16 1:11 PM, Tom Lane wrote: > > Alvaro Herrera writes: > > > I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog > > > to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like > > > that. > > > > I think

Re: [HACKERS] Re: [COMMITTERS] pgsql: Change the way that LWLocks for extensions are allocated.

2016-08-29 Thread Andres Freund
On 2016-08-30 07:57:19 +0530, Amit Kapila wrote: > I will write such a test case either in this week or early next week. Great. > I hope this is not super urgent, let me know if you think otherwise. It's not urgent, no. Thanks! Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@po

Re: [HACKERS] [WIP] Patches to enable extraction state of query execution from external session

2016-08-30 Thread Andres Freund
On 2016-08-30 11:22:43 +0300, Maksim Milyutin wrote: > > Hi, > > > > On 2016-08-29 18:22:56 +0300, maksim wrote: > > > Now I complete extension that provides facility to see the current state > > > of > > > query execution working on external session in form of EXPLAIN ANALYZE > > > output. This

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-08-30 Thread Andres Freund
t > if we e.g. get rid of ExecMaterializeSlot() in its current form, would that > be OK? Hm. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] sequences and pg_upgrade

2016-08-30 Thread Andres Freund
On 2016-08-30 08:46:48 -0400, Peter Eisentraut wrote: > I was toying with a couple of ideas that would involve changing the > storage of sequences. (Say, for the sake of discussion, removing the > problematic/useless sequence_name field.) I'd be quite interested to know what changes that are...

<    1   2   3   4   5   6   7   8   9   10   >