Re: [HACKERS] Exclude pg_internal.init from base backup

2017-11-03 Thread Petr Jelinek
> + > + Not specific problem to this patch, but I wonder if it should be made more clear that those files (there are couple above of what you added) are skipped no matter which directory they reside in. -- Petr Jelinek http://www.2ndQuadrant.com/ Pos

Re: [HACKERS] Walsender timeouts and large transactions

2017-11-03 Thread Petr Jelinek
Hi, thanks for checking. On 02/11/17 10:00, Robert Haas wrote: > On Wed, Nov 1, 2017 at 8:19 PM, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote: >> sorry for the delay but I didn't have much time in past weeks to follow >> this thread. > > +Timesta

Re: [HACKERS] Walsender timeouts and large transactions

2017-11-01 Thread Petr Jelinek
the keepalive handling. This code is specific to logical decoding walsender interface so it only happens for logical decoding/replication (which means it should be backported all the way to 9.4). The physical one These two issues happen quite normally in the wild as all we need is big data load i

[HACKERS] Subscriber resets additional columns to NULL on UPDATE

2017-10-26 Thread Petr Jelinek
in slot_modify_cstrings() so the extra columns will fall through to the else block which sets the value to NULL. Attached patch fixes it and adds couple of tests for this scenario. This is rather serious issue so it would be good if we could get it fixed in 10.1. -- Petr Jelinek http://www

Re: [HACKERS] Fix a typo in execReplication.c

2017-10-12 Thread Petr Jelinek
re_row? >> Indeed, that's another copy-pasto. > > Thank you! > I think you're right. It should be trig_delete_before_row and be > back-patched to 10. Attached patch fixes it. > Thanks for the patch, looks correct to me. -- Petr Jelinek http://www.2ndQuad

Re: [HACKERS] Help required to debug pg_repack breaking logical replication

2017-10-10 Thread Petr Jelinek
n't had a > suitable reproducible test case. > >> In the above issue #113, Petr Jelinek commented: >> >>> From quick look at pg_repack, the way it does table rewrite is almost >>> guaranteed >>> to break logical decoding unless there is zero u

[HACKERS] Re: [BUGS] 10.0: Logical replication doesn't execute BEFORE UPDATE OF trigger

2017-10-10 Thread Petr Jelinek
r what has been actually changed by the execution) I don't really see the difference between this and statement triggers except that statement trigger behavior is documented. I understand that it's somewhat confusing and I am not saying it's ideal, but I don't see any other behavior that would

[HACKERS] Re: [BUGS] 10.0: Logical replication doesn't execute BEFORE UPDATE OF trigger

2017-10-10 Thread Petr Jelinek
applying updates. So TriggerEnabled >> always ends up with false. I'll make a patch and submit. >> > > Attached patch store the updated columns bitmap set to RangeTblEntry. > In my environment this bug seems to be fixed by the patch. > Hi, let me start by saying that my view i

Re: [HACKERS] Discussion on missing optimizations

2017-10-07 Thread Petr Jelinek
On 07/10/17 19:59, Tom Lane wrote: > Petr Jelinek <petr.jeli...@2ndquadrant.com> writes: >> On 07/10/17 18:15, Tom Lane wrote: >>> No, I'm afraid you didn't read that comment closely enough. This will >>> flat out fail for cases like "sel

Re: [HACKERS] Discussion on missing optimizations

2017-10-07 Thread Petr Jelinek
On 07/10/17 18:15, Tom Lane wrote: > Petr Jelinek <petr.jeli...@2ndquadrant.com> writes: >> On 07/10/17 04:19, Tom Lane wrote: >>> (edit: a few minutes later, I seem to remember that equivclass.c has >>> to do something special with the X=X case, so maybe it co

Re: [HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-07 Thread Petr Jelinek
On 07/10/17 18:23, Konstantin Knizhnik wrote: > On 10/07/2017 04:26 PM, Petr Jelinek wrote: >> >> Hmm so you start transaction (you have to when running >> CREATE_REPLICATION_SLOT with USE_SNAPSHOT parameter). And while the slot >> is being created the config is

Re: [HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-07 Thread Petr Jelinek
On 06/10/17 16:46, Konstantin Knizhnik wrote: > > > On 06.10.2017 15:29, Petr Jelinek wrote: >> On 06/10/17 12:16, Konstantin Knizhnik wrote: >>> When creating logical replication slots we quite often get the following >>> error: >>> >>>

Re: [HACKERS] Discussion on missing optimizations

2017-10-07 Thread Petr Jelinek
re, we'll just waste equal() call and if it is there the optimization seems worth it as it will lead to orders of magnitude better estimation in many cases. So I wrote prototype of achieving this optimization and it seems to be really quite simple code-wise (see attached). I did only a limited manua

Re: [HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-06 Thread Petr Jelinek
if it is normal situation or something goes wrong? > Hi, no it's not normal situation, it seems you are doing something that assigns xid before you run the CREATE_REPLICATION_SLOT command on that connection. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Devel

Re: [HACKERS] Issues with logical replication

2017-10-03 Thread Petr Jelinek
On 02/10/17 18:59, Petr Jelinek wrote: >> >> Now fix the trigger function: >> CREATE OR REPLACE FUNCTION replication_trigger_proc() RETURNS TRIGGER AS $$ >> BEGIN >>   RETURN NEW; >> END $$ LANGUAGE plpgsql; >> >> And manually perform at master tw

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-03 Thread Petr Jelinek
On 03/10/17 22:01, Thomas Munro wrote: > On Wed, Oct 4, 2017 at 7:32 AM, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote: >> On 03/10/17 19:44, Tom Lane wrote: >>> I wrote: >>>>> So that's trouble waiting to happen, for sure. At the very least we

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-03 Thread Petr Jelinek
we indeed wanted to change the decision about this I think we can simply give latch as a parameter to libpqrcv_connect and store it in the WalReceiverConn struct. The connection does not need to live past the latch (although it currently does, but that's just a matter of reordering the code in

Re: [HACKERS] Issues with logical replication

2017-10-02 Thread Petr Jelinek
08ac, >     fdw_trigtuple=0x0, slot=0x2dd0240) at trigger.c:2748 > #5  0x006d2395 in ExecSimpleRelationUpdate (estate=0x2e36b50, > epqstate=0x7ffc4420eda0, searchslot=0x2dd0358, slot=0x2dd0240) >     at execReplication.c:461 > #6  0x00820894 in apply_handle_updat

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-26 Thread Petr Jelinek
I said before it's > chicken-and-egg. But nobody is asking core to do their work. As much as > I love it, I think logical decoding is a bit half-baked until there is a > single, quality, in-core plugin, as it discourages its usage, because of > the reasons I stated. > Well, in that case

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Petr Jelinek
On 25/09/17 19:48, Joshua D. Drake wrote: > On 09/25/2017 10:43 AM, Andres Freund wrote: >> On 2017-09-25 10:38:52 -0700, Joshua D. Drake wrote: >>> On 09/25/2017 10:32 AM, Petr Jelinek wrote: >>>> On 25/09/17 19:26, Tom Lane wrote: >>>>&

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Petr Jelinek
ins out there so I think we did reasonable job there. The fact that vast majority of that are various json ones gives reasonable hint that we should have that one in core though. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -

Re: [HACKERS] logical replication and statistics

2017-09-25 Thread Petr Jelinek
That was all > > In this example, nothing's been done yet by the actual replication > apply process, only by the initial table sync. Maybe that accounts > for your not seeing stats? > The main replication worker should still be running though. The output of pg_stat_replicatio

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Petr Jelinek
On 25/09/17 18:48, Alvaro Hernandez wrote: > > > On 25/09/17 19:39, Petr Jelinek wrote: >> >> Well, test_decoding is not meant for production use anyway, no need for >> middleware to support it. The pgoutput is primarily used for internal >> replication purpose

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Petr Jelinek
pact representation > format (or that supports several formats, not only json). > JSON is indeed great for interoperability, if you want more compact format, use either pgoutput or write something of your own or do conversion to something else in your consumer. I don't think postgres nee

Re: bgw_type (was Re: [HACKERS] Why does logical replication launcher set application_name?)

2017-09-25 Thread Petr Jelinek
documentation, as well as easier >> to interpret and easier for module developers. > > But then background workers that are not updated for, say, PG11 will not > show anything useful in pg_stat_activity. We should have some amount of > backward compatibility here

Re: [HACKERS] logical replication and statistics

2017-09-25 Thread Petr Jelinek
ication by ALTER SUBSCRIPTION test_sub ENABLE. > > > I used this subscriptions for and it was warking. If there is no pid, the worker is not running. And if there is nothing in pg_stat_replication on master, the walsender is not running either, so it seems like it's not actually working. --

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-20 Thread Petr Jelinek
On 20/09/17 05:53, Tom Lane wrote: > Craig Ringer <cr...@2ndquadrant.com> writes: >> On 19 September 2017 at 18:04, Petr Jelinek <petr.jeli...@2ndquadrant.com> >> wrote: >>> If you are asking why they are not identified by the >>> BackgroundWorker

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Petr Jelinek
On 19/09/17 16:30, Amit Kapila wrote: > On Tue, Sep 19, 2017 at 3:34 PM, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote: >> n 18/09/17 18:42, Tom Lane wrote: >>> Amit Kapila <amit.kapil...@gmail.com> writes: >>>> On Mon, Sep 18, 2017 at 7:46 PM

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Petr Jelinek
On 19/09/17 15:08, Amit Kapila wrote: > On Tue, Sep 19, 2017 at 6:29 PM, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote: >> On 19/09/17 14:33, Amit Kapila wrote: >>> On Tue, Sep 19, 2017 at 3:34 PM, Petr Jelinek >>> <petr.jeli...@2ndquadrant.com> wr

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Petr Jelinek
On 19/09/17 14:33, Amit Kapila wrote: > On Tue, Sep 19, 2017 at 3:34 PM, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote: >> n 18/09/17 18:42, Tom Lane wrote: >> >>> So, frankly, I think we would be best off losing the "logical rep >>> worker slo

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Petr Jelinek
eed slots for that. If you are asking why they are not identified by the BackgroundWorkerHandle, then it's because it's private struct and can't be shared with other processes so there is no way to link the logical worker info with bgworker directly. I mentioned that I am not happy about this during

Re: [HACKERS] More replication race conditions

2017-08-27 Thread Petr Jelinek
On 28/08/17 01:36, Michael Paquier wrote: > On Sun, Aug 27, 2017 at 6:32 PM, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote: >> Attached should fix this. > > +$node_master->poll_query_until('postgres', > +"SELECT EXISTS (SELECT 1 FROM pg_replication_s

Re: [HACKERS] More replication race conditions

2017-08-27 Thread Petr Jelinek
On 27/08/17 04:32, Noah Misch wrote: > On Fri, Aug 25, 2017 at 12:09:00PM +0200, Petr Jelinek wrote: >> On 24/08/17 19:54, Tom Lane wrote: >>> sungazer just failed with >>> >>> pg_recvlogical exited with code '256', stdout '' and stderr >>> 'pg_re

Re: [HACKERS] Proposal: global index

2017-08-25 Thread Petr Jelinek
gainst partitioned tables vs stored in one giant index, right? > No, just global constraints. For example, if you consider unique index to be implementation detail of a unique constraint, there is nothing stopping us to use multiple such indexes (one per partition) as implementation detail to s

Re: [HACKERS] More replication race conditions

2017-08-25 Thread Petr Jelinek
y used by somebody else (slots use their own locking mechanism that does not wait). In this case it seems the walsender which was using slot for previous previous step didn't finish releasing the slot by the time we start new command. We can work around this by changing the test to wait perhaps. -- P

[HACKERS] Re: [COMMITTERS] pgsql: Don't force-assign transaction id when exporting a snapshot.

2017-08-12 Thread Petr Jelinek
which allows doing that when pg_dumping from PG10+. I also wonder if it should be mentioned in release notes. If the attached patch would make it into PG10 it would be no brainer to mention it as feature under pg_dump section, but exporting snapshots alone I am not sure about. -- Petr Jelinek

Re: [HACKERS] Walsender timeouts and large transactions

2017-08-09 Thread Petr Jelinek
do CHECK_FOR_INTERRUPTS() independently of pq_is_send_pending so that it's possible to stop walsender while it's processing large transaction. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More race conditions in logical replication

2017-07-25 Thread Petr Jelinek
f ConditionVariableSleep(). But that's not true, it's essential for preventing race conditions like the one above because it puts the current process into waiting list so we can be sure it will be signaled on broadcast once ConditionVariablePrepareToSleep() has been called. -- Petr Jelinek

Re: [HACKERS] More race conditions in logical replication

2017-07-07 Thread Petr Jelinek
On 06/07/17 18:20, Petr Jelinek wrote: > On 06/07/17 17:33, Petr Jelinek wrote: >> On 03/07/17 01:54, Tom Lane wrote: >>> I noticed a recent failure that looked suspiciously like a race condition: >>> >>> https://buildfarm.postgresql.org/cgi-bin/show_log.pl

[HACKERS] Subscription code improvements

2017-07-07 Thread Petr Jelinek
...@2ndquadrant.com [2] https://www.postgresql.org/message-id/flat/cad21aobd4t2rwtiwd8yfxd+q+m9swx50yjqt5ibj2mzebvn...@mail.gmail.com -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services From 4c1ef64493ee930dfde3aa787c454a5d68ac3efd Mon

Re: [HACKERS] More race conditions in logical replication

2017-07-06 Thread Petr Jelinek
On 06/07/17 17:33, Petr Jelinek wrote: > On 03/07/17 01:54, Tom Lane wrote: >> I noticed a recent failure that looked suspiciously like a race condition: >> >> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hornet=2017-07-02%2018%3A02%3A07 >> >>

Re: [HACKERS] More race conditions in logical replication

2017-07-06 Thread Petr Jelinek
so there is no object locking with wait queue. We could use some latch wait with small timeout but that seems ugly as that function can be called by user without having dropped the slot before so the wait can be quite long (as in "forever"). -- Petr Jelinek http://www.

Re: [HACKERS] Fix doc of DROP SUBSCRIPTION

2017-06-30 Thread Petr Jelinek
sociates with a replication. > > Sorry, I was wrong and missing something... I confirmaed it. > The documentation is right. Sorry for the noise. > It will only fail if the remote site is not reachable during the drop (but then it should hint about the ALTER), maybe that's what happ

Re: [HACKERS] Apparent walsender bug triggered by logical replication

2017-06-30 Thread Petr Jelinek
On 30/06/17 04:46, Tom Lane wrote: > Petr Jelinek <petr.jeli...@2ndquadrant.com> writes: >> On 30/06/17 02:07, Tom Lane wrote: >>> I'm also kind of wondering why the "behind the apply" path out of >>> LogicalRepSyncTableStart exists at all; as far as I

Re: [HACKERS] Apparent walsender bug triggered by logical replication

2017-06-29 Thread Petr Jelinek
interprocess signaling. > Hmm, I don't understand what you mean here. The "letting any extra catchup happen later" would never happen if the sync is behind apply as apply has already skipped relevant transactions. -- Petr Jelinek http://www.2ndQuadrant.com/

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-06-27 Thread Petr Jelinek
he whole subscription (and only conflicting things are DROP and ALTER SUBSCRIPTION), not individual subscription-relation mapping so it doesn't seem to me like there is any higher risk of deadlocks than anything else which works with multiple tables (or compared to previous behavior). -- Petr Je

Re: [HACKERS] Walsender timeouts and large transactions

2017-06-25 Thread Petr Jelinek
On 30/05/17 15:44, Petr Jelinek wrote: > On 30/05/17 11:02, Kyotaro HORIGUCHI wrote: >> >> +TimestampTz now = GetCurrentTimestamp(); >> >> I think it is not recommended to read the current time too >> frequently, especially within a loop that hates sl

Re: [HACKERS] Logical replication: stuck spinlock at ReplicationSlotRelease

2017-06-25 Thread Petr Jelinek
need to be able to modify the catalog for that which may not be possible in an unrecoverable error) but it's not clear to me how to reasonably produce such a list. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Se

Re: [HACKERS] CREATE SUBSCRIPTION log noise

2017-06-25 Thread Petr Jelinek
itly asked for, I would agree it's not needed but given that it's default behavior I prefer to inform the user. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-06-25 Thread Petr Jelinek
it's also probably why this has slipped through). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services From d7038474012769c9c3b50231af76dd7796fe593f Mon Sep 17 00:00:00 2001 From: Petr Jelinek <pjmo...@pjmodos.net> Date: Sat

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-06-15 Thread Petr Jelinek
On 15/06/17 18:36, Peter Eisentraut wrote: > On 6/15/17 12:22, Petr Jelinek wrote: >> On 15/06/17 17:53, Peter Eisentraut wrote: >>> On 6/14/17 18:35, Petr Jelinek wrote: >>>> Attached fixes it (it was mostly about order of calls). >>> >>> So do I u

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-06-15 Thread Petr Jelinek
On 15/06/17 17:53, Peter Eisentraut wrote: > On 6/14/17 18:35, Petr Jelinek wrote: >> Attached fixes it (it was mostly about order of calls). > > So do I understand this right that the actual fix is just moving up the > logicalrep_worker_stop() call in DropSubscriptio

Re: [HACKERS] Refreshing subscription relation state inside a transaction block

2017-06-15 Thread Petr Jelinek
On 15/06/17 15:52, Peter Eisentraut wrote: > On 6/15/17 02:41, Petr Jelinek wrote: >> Hmm, forcibly stopping currently running table sync is not what was >> intended, I'll have to look into it. We should not be forcibly stopping >> anything except the main apply worker dur

Re: [HACKERS] Refreshing subscription relation state inside a transaction block

2017-06-15 Thread Petr Jelinek
On 13/06/17 18:33, Masahiko Sawada wrote: > On Wed, Jun 14, 2017 at 1:02 AM, Masahiko Sawada <sawada.m...@gmail.com> > wrote: >> On Tue, Jun 13, 2017 at 4:53 PM, Petr Jelinek >> <petr.jeli...@2ndquadrant.com> wrote: >>> On 13/06/17 09:06, Masahiko Sawa

Re: [HACKERS] logical replication busy-waiting on a lock

2017-06-14 Thread Petr Jelinek
On 14/06/17 20:57, Andres Freund wrote: > Hi, > > On 2017-06-13 00:50:20 -0700, Andres Freund wrote: >> Just to be clear: The patch, after the first point above (which I did), >> looks ok. I'm just looking for comments. > > And pushed. > Thanks! -- Petr Jelin

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-06-14 Thread Petr Jelinek
y processing it. So it looks to me like we'd need to reread the catalog with new snapshot after the lock was acquired which seems bit wasteful (I wonder if we could just AcceptInvalidationMessages and refetch from syscache). Any better ideas? Other related problem is locking of subscriptions during o

Re: [HACKERS] Refreshing subscription relation state inside a transaction block

2017-06-13 Thread Petr Jelinek
subscription DDLs inside a transaction block. Attached > patch. > Can you be more specific than "It doesn't work fine inside transaction block", what do you expect to happen and what actually happens? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL D

Re: [HACKERS] pg_subscription_rel entry can be updated concurrently

2017-06-12 Thread Petr Jelinek
n, so it would > be the same result as when the worker ends up with an error. But I > think since it's not an appropriate behavior we should deal with it. > Any thoughts? > This has been already reported by tushar in different thread and it's still on my list to fix. -- Petr Jelinek

Re: [HACKERS] logical replication busy-waiting on a lock

2017-06-09 Thread Petr Jelinek
On 09/06/17 03:20, Petr Jelinek wrote: > On 09/06/17 01:06, Andres Freund wrote: >> Hi, >> >> On 2017-06-03 04:50:00 +0200, Petr Jelinek wrote: >>> One thing I don't like is the GetLastLocalTransactionId() that I had to >>> add because we clear the proc-&

Re: [HACKERS] walsender & parallelism

2017-06-09 Thread Petr Jelinek
ch wrote: >>>>> On Fri, Jun 02, 2017 at 11:27:55PM -0400, Peter Eisentraut wrote: >>>>>> On 5/31/17 23:54, Peter Eisentraut wrote: >>>>>>> On 5/29/17 22:01, Noah Misch wrote: >>>>>>>> On Tue, May 23, 2017 at 01:45:5

Re: [HACKERS] logical replication busy-waiting on a lock

2017-06-08 Thread Petr Jelinek
On 09/06/17 01:06, Andres Freund wrote: > Hi, > > On 2017-06-03 04:50:00 +0200, Petr Jelinek wrote: >> One thing I don't like is the GetLastLocalTransactionId() that I had to >> add because we clear the proc->lxid before we get to AtEOXact_Snapshot() >> bu

Re: [HACKERS] walsender termination error messages worse in v10

2017-06-08 Thread Petr Jelinek
On 08/06/17 23:57, Andres Freund wrote: > On 2017-06-03 00:55:22 +0200, Petr Jelinek wrote: >> On 02/06/17 23:45, Andres Freund wrote: >>> Hi Petr, >>> >>> On 2017-06-02 22:57:37 +0200, Petr Jelinek wrote: >>>> On 02/06/17 20:51, Andres Freund w

Re: [HACKERS] Notes on testing Postgres 10b1

2017-06-07 Thread Petr Jelinek
On 08/06/17 03:50, Josh Berkus wrote: > On 06/07/2017 06:25 PM, Petr Jelinek wrote: >> On 08/06/17 03:19, Josh Berkus wrote: >>> >>> Peter and Petr: >>> >>> On 06/07/2017 05:24 PM, Peter Eisentraut wrote: >>>> On 6/7/17 01:01, Josh Berkus w

Re: [HACKERS] Notes on testing Postgres 10b1

2017-06-07 Thread Petr Jelinek
in the > first place, and in some environments (AWS, containers, etc.) logs can > be very hard to access. > > We really need the subscription to fail at step (2), not wait for the > first update to fail. And if it doesn't fail at step 2, then we should > at least give a warning.

Re: [HACKERS] Use of snapshot in logical replication

2017-06-07 Thread Petr Jelinek
gaps or duplicates. That being said, the built-in logical replication isn't using the exported snapshots at all. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@

Re: [HACKERS] Notes on testing Postgres 10b1

2017-06-07 Thread Petr Jelinek
ription gives list of requested publications as option to START_REPLICATION walsender command. The list of publications associated with a subscription are only stored on the subscriber and publisher has no idea what those are. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Dev

Re: [HACKERS] logical replication - possible remaining problem

2017-06-07 Thread Petr Jelinek
but that's separate discussion). > then, and only then, does the session finish and succeed ('replica ok'). > > So to me it looks as if there is an omission of > pg_replication_origin-cleanup when pg_description is deleted. > There is no omission, origin is not supposed to

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-06-07 Thread Petr Jelinek
On 07/06/17 03:00, Andres Freund wrote: > On 2017-06-06 19:36:13 +0200, Petr Jelinek wrote: > >> As a side note, we are starting to have several IsSomeTypeOfProcess >> functions for these kind of things. I wonder if bgworker infrastructure >> should somehow pr

Re: [HACKERS] libpqrcv_PQexec() seems to violate latch protocol

2017-06-06 Thread Petr Jelinek
On 06/06/17 23:42, Andres Freund wrote: > On 2017-06-06 23:24:50 +0200, Petr Jelinek wrote: >> On 06/06/17 23:17, Andres Freund wrote: >>> Right. I found a couple more instance of similarly iffy, although not >>> quite as broken, patterns in launcher.c. It's easy to g

Re: [HACKERS] libpqrcv_PQexec() seems to violate latch protocol

2017-06-06 Thread Petr Jelinek
r.c. It's easy to get this wrong, > but it's a lot easy if you do it differently everywhere you use a > latch. It's not good if code in the same file, by the same author(s), > has different ways of using latches. Huh? I see same pattern everywhere in launcher.c, what am I missing? -- Petr Jel

Re: [HACKERS] logical replication - still unstable after all these months

2017-06-06 Thread Petr Jelinek
users. > +1, I considered the issues solved when snapshot builder issues were fixed so it was good reminder to check again. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-06-06 Thread Petr Jelinek
On 03/06/17 18:53, Peter Eisentraut wrote: > On 6/2/17 14:52, Peter Eisentraut wrote: >> On 5/24/17 15:14, Petr Jelinek wrote: >>> All the locking works just fine the way it is in master. The issue with >>> deadlock with apply comes from the wrong handling of the S

Re: [HACKERS] inconsistent application_name use in logical workers

2017-06-06 Thread Petr Jelinek
On 06/06/17 15:07, Peter Eisentraut wrote: > On 6/6/17 06:51, Petr Jelinek wrote: >> On 06/06/17 04:19, Peter Eisentraut wrote: >>> The logical replication code is supposed to use the subscription name as >>> the fallback_application_name, but in some cases it uses the

Re: [HACKERS] inconsistent application_name use in logical workers

2017-06-06 Thread Petr Jelinek
ntiation has a reason though. The application_name is used for sync rep and having synchronization connection using same application_name might have adverse effects there because synchronization connection can be in-front of main apply one, so sync rep will think something is consumed while it's not.

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-06-05 Thread Petr Jelinek
On 03/06/17 05:18, Peter Eisentraut wrote: > On 6/2/17 16:44, Petr Jelinek wrote: >> However, I am not sure about the bgw_name_extra. I think I would have >> preferred keeping full bgw_name field which would be used where full >> name is needed and bgw_type where only th

Re: [HACKERS] logical replication - still unstable after all these months

2017-06-04 Thread Petr Jelinek
On 03/06/17 16:12, Jeff Janes wrote: > > On Fri, Jun 2, 2017 at 4:10 PM, Petr Jelinek > <petr.jeli...@2ndquadrant.com <mailto:petr.jeli...@2ndquadrant.com>> wrote: > > > While I was testing something for different thread I noticed that I > manage trans

Re: [HACKERS] logical replication - still unstable after all these months

2017-06-02 Thread Petr Jelinek
On 03/06/17 04:45, Mark Kirkwood wrote: > On 03/06/17 11:10, Petr Jelinek wrote: > >> On 02/06/17 22:29, Petr Jelinek wrote: >>> On 02/06/17 08:55, Mark Kirkwood wrote: >>>> On 02/06/17 17:11, Erik Rijkers wrote: >>>> >>>>> On 2017-06-0

Re: [HACKERS] logical replication busy-waiting on a lock

2017-06-02 Thread Petr Jelinek
ss the pid around a bit but I don't think it's too bad. One thing I don't like is the GetLastLocalTransactionId() that I had to add because we clear the proc->lxid before we get to AtEOXact_Snapshot() but I don't have better solution there. -- Petr Jelinek http://www.2ndQuadr

Re: [HACKERS] logical replication busy-waiting on a lock

2017-06-02 Thread Petr Jelinek
On 02/06/17 03:38, Robert Haas wrote: > On Thu, Jun 1, 2017 at 2:28 PM, Andres Freund <and...@anarazel.de> wrote: >> On 2017-06-01 14:17:44 +0200, Petr Jelinek wrote: >>> Thinking more about this, I am not convinced it's a good idea to change >>> exports this late

Re: [HACKERS] logical replication - still unstable after all these months

2017-06-02 Thread Petr Jelinek
On 02/06/17 22:29, Petr Jelinek wrote: > On 02/06/17 08:55, Mark Kirkwood wrote: >> On 02/06/17 17:11, Erik Rijkers wrote: >> >>> On 2017-06-02 00:46, Mark Kirkwood wrote: >>>> On 31/05/17 21:16, Petr Jelinek wrote: >>>> >>>>

Re: [HACKERS] walsender termination error messages worse in v10

2017-06-02 Thread Petr Jelinek
On 02/06/17 23:45, Andres Freund wrote: > Hi Petr, > > On 2017-06-02 22:57:37 +0200, Petr Jelinek wrote: >> On 02/06/17 20:51, Andres Freund wrote: >>> I don't understand why the new block is there, nor does the commit >>> message explain it. >>> >

Re: [HACKERS] walsender termination error messages worse in v10

2017-06-02 Thread Petr Jelinek
(PQerrorMessage(conn->streamConn); > } > } > > note the new /* Verify that there are no more results */ bit. > > I don't understand why the new block is there, nor does the commit > message explain it. > Hmm, that particular change can actually be rev

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-06-02 Thread Petr Jelinek
_name_extra. I think I would have preferred keeping full bgw_name field which would be used where full name is needed and bgw_type where only the worker type is used. The concatenation just doesn't sit well with me, especially if it requires the bgw_name_extra to start with space. --

Re: [HACKERS] logical replication - still unstable after all these months

2017-06-02 Thread Petr Jelinek
On 02/06/17 08:55, Mark Kirkwood wrote: > On 02/06/17 17:11, Erik Rijkers wrote: > >> On 2017-06-02 00:46, Mark Kirkwood wrote: >>> On 31/05/17 21:16, Petr Jelinek wrote: >>> >>> I'm seeing a new failure with the patch applied - this time the >>>

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-02 Thread Petr Jelinek
On 02/06/17 15:37, Amit Kapila wrote: > On Thu, Jun 1, 2017 at 10:36 PM, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote: >> On 01/06/17 15:25, Tom Lane wrote: >>> Robert Haas <robertmh...@gmail.com> writes: >>>> So, are you going to, per

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-06-01 Thread Petr Jelinek
On 01/06/17 17:32, Masahiko Sawada wrote: > On Thu, May 25, 2017 at 5:29 PM, tushar <tushar.ah...@enterprisedb.com> wrote: >> On 05/25/2017 12:44 AM, Petr Jelinek wrote: >>> >>> There is still outstanding issue that sync worker will keep running >>> ins

Re: [HACKERS] [BUGS] Concurrent ALTER SEQUENCE RESTART Regression

2017-06-01 Thread Petr Jelinek
that the code is simpler. So +1 to go ahead with both patches. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscriptio

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-01 Thread Petr Jelinek
nd the continue statement in the else block IMHO. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Create subscription with `create_slot=false` and incorrect slot name

2017-06-01 Thread Petr Jelinek
lly little documentation about this. So > how about the attached documentation patch as well? > > As mentioned earlier, if we want to do HINT messages, that will be a bit > more involved and probably PG11 material. > I think the combination of those patches is probably

Re: [HACKERS] logical replication busy-waiting on a lock

2017-06-01 Thread Petr Jelinek
On 31/05/17 11:21, Petr Jelinek wrote: > On 31/05/17 09:24, Andres Freund wrote: >> On 2017-05-29 23:49:33 +0200, Petr Jelinek wrote: >>> I am not quite sure I understand (both the vxid suggestion and for the >>> session dying badly). Maybe we can discuss bit more when

Re: [HACKERS] walsender & parallelism

2017-06-01 Thread Petr Jelinek
e handled properly (= we need to share SIGHUP/SIGUSR1 handling between postgres.c and walsender.c). The rest can wait for PG11. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing li

Re: [HACKERS] logical replication busy-waiting on a lock

2017-05-31 Thread Petr Jelinek
On 31/05/17 09:24, Andres Freund wrote: > On 2017-05-29 23:49:33 +0200, Petr Jelinek wrote: >> I am not quite sure I understand (both the vxid suggestion and for the >> session dying badly). Maybe we can discuss bit more when you get to >> computer so it's easier for you to ex

Re: [HACKERS] logical replication - still unstable after all these months

2017-05-31 Thread Petr Jelinek
On 29/05/17 23:06, Mark Kirkwood wrote: > On 29/05/17 23:14, Petr Jelinek wrote: > >> On 29/05/17 03:33, Jeff Janes wrote: >> >>> What would you want to look at? Would saving the WAL from the master be >>> helpful? >>> >> Useful info is, l

Re: [HACKERS] Walsender timeouts and large transactions

2017-05-30 Thread Petr Jelinek
On 30/05/17 11:02, Kyotaro HORIGUCHI wrote: > At Thu, 25 May 2017 17:52:50 +0200, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote in > <e082a56a-fd95-a250-3bae-0fff93832...@2ndquadrant.com> >> Hi, >> >> We have had issue with walsender

Re: [HACKERS] logical replication busy-waiting on a lock

2017-05-29 Thread Petr Jelinek
On 29/05/17 21:44, Andres Freund wrote: > > > On May 29, 2017 12:41:26 PM PDT, Petr Jelinek <petr.jeli...@2ndquadrant.com> > wrote: >> On 29/05/17 21:28, Andres Freund wrote: >>> >>> On May 29, 2017 12:25:35 PM PDT, Petr Jelinek >> <pet

Re: [HACKERS] logical replication busy-waiting on a lock

2017-05-29 Thread Petr Jelinek
On 29/05/17 21:28, Andres Freund wrote: > > On May 29, 2017 12:25:35 PM PDT, Petr Jelinek <petr.jeli...@2ndquadrant.com> > wrote: >> >> Looking at the code more, the xid is only used as parameter for >> SnapBuildBuildSnapshot() which never does anything with th

Re: [HACKERS] logical replication busy-waiting on a lock

2017-05-29 Thread Petr Jelinek
On 29/05/17 21:23, Andres Freund wrote: > > > On May 29, 2017 12:21:50 PM PDT, Petr Jelinek <petr.jeli...@2ndquadrant.com> > wrote: >> On 29/05/17 20:59, Andres Freund wrote: >>> >>> >>> On May 29, 2017 11:58:05 AM PDT, Petr Jelinek >>

Re: [HACKERS] logical replication busy-waiting on a lock

2017-05-29 Thread Petr Jelinek
On 29/05/17 21:21, Petr Jelinek wrote: > On 29/05/17 20:59, Andres Freund wrote: >> >> >> On May 29, 2017 11:58:05 AM PDT, Petr Jelinek <petr.jeli...@2ndquadrant.com> >> wrote: >>> On 27/05/17 17:17, Andres Freund wrote: >>>> >>>>

Re: [HACKERS] logical replication busy-waiting on a lock

2017-05-29 Thread Petr Jelinek
On 29/05/17 20:59, Andres Freund wrote: > > > On May 29, 2017 11:58:05 AM PDT, Petr Jelinek <petr.jeli...@2ndquadrant.com> > wrote: >> On 27/05/17 17:17, Andres Freund wrote: >>> >>> >>> On May 27, 2017 9:48:22 AM EDT, Petr Jelinek >> <

Re: [HACKERS] logical replication busy-waiting on a lock

2017-05-29 Thread Petr Jelinek
On 27/05/17 17:17, Andres Freund wrote: > > > On May 27, 2017 9:48:22 AM EDT, Petr Jelinek <petr.jeli...@2ndquadrant.com> > wrote: >> Actually, I guess it's the pid 47457 (COPY process) who is actually >> running the xid 73322726. In that case that's the same thi

  1   2   3   4   5   6   7   8   9   10   >