Re: [HACKERS] logical replication - still unstable after all these months

2017-05-29 Thread Petr Jelinek
mention LSNs), logs from subscriber (the lines about when sync workers finished), contents of the pg_subscription_rel (with srrelid casted to regclass so we know which table is which), and pg_waldump output around the LSNs found in the logs and in the pg_subscription_rel (+ few lines before and some after to get contex

Re: [HACKERS] logical replication busy-waiting on a lock

2017-05-27 Thread Petr Jelinek
On 27/05/17 15:44, Petr Jelinek wrote: > On 27/05/17 01:25, Jeff Janes wrote: >> When I create a subscription in the disabled state, and then later doing >> "alter subscription sub enable;", on the master I sometimes get a tight >> loop of the deadlock

Re: [HACKERS] logical replication busy-waiting on a lock

2017-05-27 Thread Petr Jelinek
kTableWait releases the lock immediately after acquiring it. In fact AFAICS everything that acquires ShareLock on xid releases it immediately after acquiring as it's only used for waits. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training

Re: [HACKERS] ALTER SUBSCRIPTION ..SET PUBLICATION refresh is not throwing error.

2017-05-27 Thread Petr Jelinek
On 27/05/17 04:00, Euler Taveira wrote: > 2017-05-26 21:29 GMT-03:00 Petr Jelinek <petr.jeli...@2ndquadrant.com > <mailto:petr.jeli...@2ndquadrant.com>>: > > > Actually another possibility would be to remove the REFRESH keyword > completely and jus

Re: [HACKERS] ALTER SUBSCRIPTION ..SET PUBLICATION refresh is not throwing error.

2017-05-26 Thread Petr Jelinek
On 27/05/17 02:13, Euler Taveira wrote: > 2017-05-26 17:58 GMT-03:00 Peter Eisentraut > <peter.eisentr...@2ndquadrant.com > <mailto:peter.eisentr...@2ndquadrant.com>>: > > On 5/24/17 15:38, Petr Jelinek wrote: > >>> I wonder if w

Re: [HACKERS] logical replication - still unstable after all these months

2017-05-26 Thread Petr Jelinek
from the tests (do they log some intermediary states of those md5 checksums, maybe numbers of rows etc?), description of the problems, errors from logs, etc. I for example don't get any issues from similar test as the one described in this thread so without more info it might be hard to reprod

Re: [HACKERS] logical replication - still unstable after all these months

2017-05-26 Thread Petr Jelinek
Hi, Hmm, I was under the impression that the changes we proposed in the snapbuild thread fixed your issues, does this mean they didn't? Or the modified versions of those that were eventually committed didn't? Or did issues reappear at some point? -- Petr Jelinek http://www

Re: [HACKERS] Create subscription with `create_slot=false` and incorrect slot name

2017-05-25 Thread Petr Jelinek
sion, etc.). We don't want to give the hint that is effectively > "just forget about the slot then" for all of them. So we would need > some way to distinguish "you can't do this right now" from "this would > never work" (400 vs 500 errors). > This

Re: [HACKERS] No parameter values checking while creating Alter subscription...Connection

2017-05-25 Thread Petr Jelinek
ons it's quite useful to be able to do so > even if the other hosts aren't reachable that second. > Yes, it's intended behavior for this very reason, we want ability to (re)configure downstream without any existing upstream. It's also reason why we have the WITH (connect = false)

[HACKERS] Walsender timeouts and large transactions

2017-05-25 Thread Petr Jelinek
might ignore them for too long on large transactions. Thoughts? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services 0001-Fix-walsender-timeouts-when-decoding-large-transacti.patch Description: binary/octet-stream --

Re: [HACKERS] ALTER SUBSCRIPTION ..SET PUBLICATION refresh is not throwing error.

2017-05-24 Thread Petr Jelinek
On 24/05/17 21:28, Petr Jelinek wrote: > On 24/05/17 21:21, Petr Jelinek wrote: >> On 24/05/17 21:07, Euler Taveira wrote: >>> 2017-05-23 6:00 GMT-03:00 tushar <tushar.ah...@enterprisedb.com >>> <mailto:tushar.ah...@enterprisedb.com>>: >>> >&

Re: [HACKERS] ALTER SUBSCRIPTION ..SET PUBLICATION refresh is not throwing error.

2017-05-24 Thread Petr Jelinek
On 24/05/17 21:21, Petr Jelinek wrote: > On 24/05/17 21:07, Euler Taveira wrote: >> 2017-05-23 6:00 GMT-03:00 tushar <tushar.ah...@enterprisedb.com >> <mailto:tushar.ah...@enterprisedb.com>>: >> >> >> s=# alter subscription s1 set public

Re: [HACKERS] ALTER SUBSCRIPTION ..SET PUBLICATION refresh is not throwing error.

2017-05-24 Thread Petr Jelinek
e "REFRESH [ WITH ... ]" when user wants to refresh, so if REFRESH is not specified, we can just behave as if SKIP REFRESH was used, it's not like there is 3rd possible behavior. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Tra

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-05-24 Thread Petr Jelinek
worker will keep running inside the long COPY because the invalidation messages are also not processed until it finishes but all the original issues reported here disappear for me with the attached patches applied. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] walsender & parallelism

2017-05-23 Thread Petr Jelinek
On 23/05/17 19:45, Andres Freund wrote: > > > On May 23, 2017 1:42:41 PM EDT, Petr Jelinek <petr.jeli...@2ndquadrant.com> > wrote: >> Hi, >> >> so this didn't really move anywhere AFAICS, do we think the approach >> I've chosen is good or do we want t

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-05-23 Thread Petr Jelinek
ndicates that the worker is not using specific user (this can be easily set in InitializeSessionUserIdStandalone()) and will work for multiple things. About application_name, perhaps we should just add bgw_type or bgw_group and show it as worker_type in activity and that's it? I think this should be open item btw so I

Re: [HACKERS] walsender & parallelism

2017-05-23 Thread Petr Jelinek
Hi, so this didn't really move anywhere AFAICS, do we think the approach I've chosen is good or do we want to do something else here? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mai

Re: [HACKERS] Multiple table synchronizations are processed serially

2017-05-19 Thread Petr Jelinek
, it > has to wait for that. > Well IMHO this concrete example is partially effect of the snapshort builder fixes we did with Andres. Before those fixes the ondisk snapshot may have been used in this situation. Makes me think if we want to mark the exported snapshot as containing all transacti

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-05-19 Thread Petr Jelinek
cesses however, the workers need to be able to access the catalog during clean shutdown in some situations. What we need is to block starting of new workers for that subscription so only those code paths would need to block. So I think we might want to do both finer-grained locking and decrea

Re: [HACKERS] snapbuild woes

2017-05-14 Thread Petr Jelinek
On 13/05/17 22:23, Andres Freund wrote: > On 2017-05-12 10:57:55 +0200, Petr Jelinek wrote: >> Hmm, well it helps but actually now that we don't track individual >> running transactions anymore it got much less effective (my version of >> 0005 does as well). >> >

Re: [HACKERS] snapbuild woes

2017-05-12 Thread Petr Jelinek
On 12/05/17 10:57, Petr Jelinek wrote: > On 12/05/17 03:31, Andres Freund wrote: >> On 2017-05-11 14:54:26 -0700, Andres Freund wrote: >>> On 2017-05-11 14:51:55 -0700, wrote: >>>> On 2017-05-08 00:10:12 -0700, Andres Freund wrote: >>>>> I plan to

Re: [HACKERS] If subscription to foreign table valid ?

2017-05-12 Thread Petr Jelinek
On 12/05/17 15:55, Neha Khatri wrote: > On 5/11/17, Petr Jelinek <petr.jeli...@2ndquadrant.com> wrote: >> Hi, >> >> On 11/05/17 14:25, tushar wrote: >>> Hi, >>> >>> I observed that -we cannot publish "foreign table" in

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-12 Thread Petr Jelinek
On 12/05/17 15:02, Peter Eisentraut wrote: > On 5/9/17 11:43, Petr Jelinek wrote: >> Here is rebased version of the other patch (the interface rework). I >> also fixed the tab completion there. > > Committed. > > One small thing I changed, to make it look m

Re: [HACKERS] Time based lag tracking for logical replication

2017-05-12 Thread Petr Jelinek
ATE SUBSCRIPTION: > > CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=postgres host=localhost > user=neha port=5432' PUBLICATION mypub; > NOTICE: synchronized table states > ERROR: could not create replication slot "sub1": ERROR: could not > load library "/home/neha/p

Re: [HACKERS] If subscription to foreign table valid ?

2017-05-12 Thread Petr Jelinek
On 11/05/17 15:43, Petr Jelinek wrote: > Hi, > > On 11/05/17 14:25, tushar wrote: >> Hi, >> >> I observed that -we cannot publish "foreign table" in Publication >> >> postgres=# create foreign table t (n int) server db1_server options >> (t

Re: [HACKERS] snapbuild woes

2017-05-12 Thread Petr Jelinek
akes about 20 mins on my laptop without patches, minute and half with your patches for 0004 and 0005 (or with your 0004 and my 0005) and about 2s with my original 0004 and 0005. What makes it slow is the constant qsorting IIRC. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] snapbuild woes

2017-05-11 Thread Petr Jelinek
On 11/05/17 16:33, Stas Kelvich wrote: > >> On 10 May 2017, at 11:43, Petr Jelinek <petr.jeli...@2ndquadrant.com> wrote: >> >> On 09/05/17 22:11, Erik Rijkers wrote: >>> On 2017-05-09 21:00, Petr Jelinek wrote: >>>> On 09/05/17 19:54, Erik Rijker

Re: [HACKERS] If subscription to foreign table valid ?

2017-05-11 Thread Petr Jelinek
TION does not mean that it won't be something else half hour later. I think it does make sense to add check for this into CREATE/ALTER SUBSCRIBER though so that user is informed immediately about the mistake rather than by errors in the logs later. I'll look into writing patch for this. I don't

Re: [HACKERS] Time based lag tracking for logical replication

2017-05-11 Thread Petr Jelinek
On 11/05/17 15:01, Simon Riggs wrote: > On 11 May 2017 at 08:32, Noah Misch <n...@leadboat.com> wrote: >> On Sun, Apr 23, 2017 at 01:10:32AM +0200, Petr Jelinek wrote: >>> The time based lag tracking commit [1] added interface for logging >>> progress of repl

Re: [HACKERS] Re: logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-11 Thread Petr Jelinek
y send > a status update within 24 hours, and include a date for your subsequent status > update. Refer to the policy on open item ownership: > https://www.postgresql.org/message-id/20170404140717.GA2675809%40tornado.leadboat.com > The patch Peter mentioned was committed. There

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-05-11 Thread Petr Jelinek
n apply worker (or reusing the existing one) that searches for all the other workers for same subscription and shuts them down as well sounds like solution to this. [1] https://www.postgresql.org/message-id/cahgqgwhpi8ky-yanffe0sgmhktsycqltnkx07bw9s7-rn17...@mail.gmail.com -- Petr Jelinek

Re: [HACKERS] Should pg_current_wal_location() become pg_current_wal_lsn()

2017-05-10 Thread Petr Jelinek
voted for doing nothing. > > Hm, well, anybody else want to vote? > I am for standardizing (although I don't have preference of location vs lsn). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Se

Re: [HACKERS] Function to move the position of a replication slot

2017-05-10 Thread Petr Jelinek
As Craig said this will not work correctly for logical slots so it should throw error on those at the moment (at least until we write something that works, but that's far more complex than this patch). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Supp

Re: [HACKERS] 'nocopy data' option is set in SUBSCRIPTION but still data is getting migrated

2017-05-10 Thread Petr Jelinek
> Yes, (no)copy data only affects existing data at the time of running the command, any additional data are always replicated. The "alter subscription sub refresh publication" does nothing unless you added/removed tables to/from publication. -- Petr Jelinek ht

Re: [HACKERS] [BUGS] Concurrent ALTER SEQUENCE RESTART Regression

2017-05-10 Thread Petr Jelinek
, Andres raised concern about performance of nextval() if we do this, did you try to run any benchmark on this? Looking at the changes to open_share_lock(), I wonder if we need to have lock level as parameter so that lastval() is not blocked. -- Petr Jelinek http://www.2ndQua

Re: [HACKERS] snapbuild woes

2017-05-10 Thread Petr Jelinek
On 09/05/17 22:11, Erik Rijkers wrote: > On 2017-05-09 21:00, Petr Jelinek wrote: >> On 09/05/17 19:54, Erik Rijkers wrote: >>> On 2017-05-09 11:50, Petr Jelinek wrote: >>> >> >> Ah okay, so this is same issue that's reported by both Masahiko Sawada >&g

Re: [HACKERS] snapbuild woes

2017-05-09 Thread Petr Jelinek
On 09/05/17 19:54, Erik Rijkers wrote: > On 2017-05-09 11:50, Petr Jelinek wrote: > >> I rebased the above mentioned patch to apply to the patches Andres sent, >> if you could try to add it on top of what you have and check if it still >> fails, that would be help

Re: [HACKERS] logical replication deranged sender

2017-05-09 Thread Petr Jelinek
On 09/05/17 19:13, Jeff Janes wrote: > On Tue, May 9, 2017 at 9:18 AM, Petr Jelinek > <petr.jeli...@2ndquadrant.com <mailto:petr.jeli...@2ndquadrant.com>> wrote: > > On 08/05/17 13:47, Petr Jelinek wrote: > > On 08/05/17 01:17, Jeff Janes wrote: >

Re: [HACKERS] logical replication deranged sender

2017-05-09 Thread Petr Jelinek
On 08/05/17 13:47, Petr Jelinek wrote: > On 08/05/17 01:17, Jeff Janes wrote: >> After dropping a subscription, it says it succeeded and that it dropped >> the slot on the publisher. >> >> But the publisher still has the slot, and a full-tilt process described >&

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-09 Thread Petr Jelinek
On 09/05/17 16:28, Peter Eisentraut wrote: > On 5/9/17 04:39, Petr Jelinek wrote: >>>> What we want to simulate instead is an "auto" dependency of the slot on >>>> the subscription. So you can drop the slot separately (subject to other >>>> re

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-09 Thread Petr Jelinek
On 09/05/17 11:44, Masahiko Sawada wrote: > On Tue, May 9, 2017 at 5:57 PM, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote: >> On 09/05/17 10:51, Masahiko Sawada wrote: >>> On Tue, May 9, 2017 at 5:39 PM, Petr Jelinek >>> <petr.jeli...@2ndquadrant.co

Re: [HACKERS] snapbuild woes

2017-05-09 Thread Petr Jelinek
On 09/05/17 10:59, Erik Rijkers wrote: > On 2017-05-09 10:50, Petr Jelinek wrote: >> On 09/05/17 00:03, Erik Rijkers wrote: >>> On 2017-05-05 02:00, Andres Freund wrote: >>>> >>>> Could you have a look? >>> >>> Running tests with

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-09 Thread Petr Jelinek
On 09/05/17 10:51, Masahiko Sawada wrote: > On Tue, May 9, 2017 at 5:39 PM, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote: >> On 09/05/17 07:07, Peter Eisentraut wrote: >>> On 5/8/17 23:23, Peter Eisentraut wrote: >>>> The way this uses RESTRICT and C

Re: [HACKERS] SUBSCRIPTIONS and pg_upgrade

2017-05-09 Thread Petr Jelinek
send a valid status >>> update within 24 hours. Refer to the policy on open item ownership: >>> https://www.postgresql.org/message-id/20170404140717.GA2675809%40tornado.leadboat.com >> >> Fair enough. I have set myself a reminder to report back on May 30. > >

Re: [HACKERS] snapbuild woes

2017-05-09 Thread Petr Jelinek
l patch will not apply on top of this change). Alternatively there is a bug in this patch. Did you see high CPU usage during the test when there were those "startup" state walsenders? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-09 Thread Petr Jelinek
t whether the slot exists. If the >> subscription is associated with the slot, it should be there. > > Here is your patch amended for that. > I am fine with this mechanism as well. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-08 Thread Petr Jelinek
On 08/05/17 22:55, Peter Eisentraut wrote: > On 5/5/17 13:01, Petr Jelinek wrote: >> What do you think of attached. I actually did add RESTRICT/CASCADE, in a >> way that if there is slot RESTRICT will refuse to drop subscription and >> CASCADE will try to drop it. Still all

Re: [HACKERS] Logical replication - TRAP: FailedAssertion in pgstat.c

2017-05-08 Thread Petr Jelinek
On 08/05/17 18:12, Peter Eisentraut wrote: > On 5/5/17 07:13, Petr Jelinek wrote: >> Yes that's different thing that we've been discussing a bit in snapbuild >> woes thread. >> >>> the 'TRAP: FailedAssertion in pgstat.c' anymore. >>> >>> (If t

Re: [HACKERS] Logical replication - TRAP: FailedAssertion in pgstat.c

2017-05-08 Thread Petr Jelinek
On 08/05/17 17:52, Masahiko Sawada wrote: > On Fri, May 5, 2017 at 8:13 PM, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote: >> On 03/05/17 13:23, Erik Rijkers wrote: >>> On 2017-05-03 08:17, Petr Jelinek wrote: >>>> On 02/05/17 20:43, Robert Haas wrote

Re: [HACKERS] logical replication deranged sender

2017-05-08 Thread Petr Jelinek
> statement: drop subscription foobar ; > Yeah that's normal because DropSubscription holds exclusive lock on pg_subscription. I am guessing 21270 is the logical replication launcher? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training &

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-05-08 Thread Petr Jelinek
top itself if the subscription got removed, but of course again the dependencies are an issue, so we should probably kill those explicitly as well. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgs

Re: [HACKERS] Draft release notes for next week's back-branch releases

2017-05-08 Thread Petr Jelinek
On 06/05/17 19:38, Tom Lane wrote: > Petr Jelinek <petr.jeli...@2ndquadrant.com> writes: >> On 06/05/17 19:15, Tom Lane wrote: >>> (Or, wait a minute. That documentation only applies to v10, but we >>> need to be writing this relnote for 9.6 users. What terminol

Re: [HACKERS] Draft release notes for next week's back-branch releases

2017-05-06 Thread Petr Jelinek
On 06/05/17 19:15, Tom Lane wrote: > Petr Jelinek <petr.jeli...@2ndquadrant.com> writes: >> On 06/05/17 18:16, Tom Lane wrote: >>> Hmm, I'm hoping for something more user-oriented. Is the corruption >>> time-limited? What's an "exported snap

Re: [HACKERS] Draft release notes for next week's back-branch releases

2017-05-06 Thread Petr Jelinek
On 06/05/17 18:16, Tom Lane wrote: > Petr Jelinek <petr.jeli...@2ndquadrant.com> writes: >> On 06/05/17 17:25, Tom Lane wrote: >>> OK, can you suggest better wording? > >> Something like the attached (it requires some polishing of English >> probably). >

Re: [HACKERS] Draft release notes for next week's back-branch releases

2017-05-06 Thread Petr Jelinek
On 06/05/17 17:25, Tom Lane wrote: > Petr Jelinek <petr.jeli...@2ndquadrant.com> writes: >> Hmm, the 2bef06d51 and 56e19d938 are fixes for different bugs, we can >> keep them together since result of both is corrupted snapshot, but the >> description can't just mangle pi

Re: [HACKERS] Draft release notes for next week's back-branch releases

2017-05-06 Thread Petr Jelinek
isk snapshot (which only contain transactions changing catalogs) as full snapshots, it's more about creating multiple logical replication slots while concurrent writes happen. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Service

Re: [HACKERS] Not getting error if ALTER SUBSCRIPTION syntax is wrong.

2017-05-06 Thread Petr Jelinek
On 05/05/17 19:51, Petr Jelinek wrote: > On 05/05/17 14:40, tushar wrote: >> Hi, >> >> While testing 'logical replication' against v10 , i encountered one >> issue where data stop migrating after ALTER PUBLICATION. >> >> X Server >> \\ Make sure wal_le

Re: [HACKERS] Not getting error if ALTER SUBSCRIPTION syntax is wrong.

2017-05-05 Thread Petr Jelinek
t * from test; > n > --- > 1 > (1 row) > > I think probably syntax of alter subscription is not correct but > surprisingly it is not throwing an error. > Syntax of ALTER command is correct, syntax of the connection string is not, you are probably getting errors in log from

Re: [HACKERS] snapbuild woes

2017-05-05 Thread Petr Jelinek
On 05/05/17 18:18, Andres Freund wrote: > On 2017-05-05 13:53:16 +0200, Petr Jelinek wrote: >> On 05/05/17 02:42, Andres Freund wrote: >>> On 2017-05-04 17:00:04 -0700, Andres Freund wrote: >>>> Attached is a prototype patch for that. >>> >>

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-05 Thread Petr Jelinek
On 02/05/17 15:31, Tom Lane wrote: > Petr Jelinek <petr.jeli...@2ndquadrant.com> writes: >> Let me expand, if we don't drop the slot by default when dropping >> subscription, we'll have a lot of users with dead slots laying around >> holding back WAL/catalog_xmin, that'

Re: [HACKERS] compiler warning with VS 2017

2017-05-05 Thread Petr Jelinek
On 05/05/17 16:56, Tom Lane wrote: > Petr Jelinek <petr.jeli...@2ndquadrant.com> writes: >> On 05/05/17 06:50, Tom Lane wrote: >>> Actually, looking around a bit there, it's not even clear why >>> we should be booby-trapping the value of an unchanged column in

Re: [HACKERS] [patch] Build pgoutput with MSVC

2017-05-05 Thread Petr Jelinek
n middle of line patch: Only garbage was found > in the patch input.". But I can just copy/paste it manually :) > Thanks for fixing this, I admit I have no idea how our windows build system works. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Devel

Re: [HACKERS] PG 10 release notes

2017-05-05 Thread Petr Jelinek
n core, the decoding just provides C API for plugins to consume the decoded info (ie, heap tuples and similar). The test_decoding *tests* the decoding API and the external projects use the decoding API as well but they don't really decode anything, their role is of filtering and deciding the wire proto

Re: [HACKERS] PG 10 release notes

2017-05-05 Thread Petr Jelinek
that we'd get a decent json plugin > into care soon-ish. While there was some initial work towards that, it > unfortunately stalled at some point. > > Euler, are you perchance interested in working towards that? ;) > I am happy to review any work in this area BTW. -- Petr Jelinek

Re: [HACKERS] compiler warning with VS 2017

2017-05-05 Thread Petr Jelinek
the comment is inadequate too. Hmm, as far as I can recollect this is just leftover debugging code that was intended to help ensure that we are checking the "changed" everywhere we are supposed to (since I changed handling of these structured quite a bit during development). Should be chan

Re: [HACKERS] snapbuild woes

2017-05-05 Thread Petr Jelinek
ant to store it in the ondisk snapshot as well? About better name for it what about something like oldest_full_xact? Otherwise the logic seems to be right on first read, will ponder it a bit more -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Suppo

Re: [HACKERS] Logical replication - TRAP: FailedAssertion in pgstat.c

2017-05-05 Thread Petr Jelinek
On 03/05/17 13:23, Erik Rijkers wrote: > On 2017-05-03 08:17, Petr Jelinek wrote: >> On 02/05/17 20:43, Robert Haas wrote: >>> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut > >>>> code path that calls CommitTransactionCommand() should have one, n

Re: [HACKERS] snapbuild woes

2017-05-05 Thread Petr Jelinek
On 05/05/17 02:00, Andres Freund wrote: > Hi, > > On 2017-05-02 08:55:53 +0200, Petr Jelinek wrote: >> Aah, now I understand we talked about slightly different things, I >> considered the running thing to be first step towards tracking aborted >> txes everywher

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-04 Thread Petr Jelinek
On 04/05/17 23:29, Tom Lane wrote: > Robert Haas <robertmh...@gmail.com> writes: >> On Wed, May 3, 2017 at 12:38 AM, Petr Jelinek >> <petr.jeli...@2ndquadrant.com> wrote: >>> Ok, Let me be clear, I actually happen to agree with your proposal. The >>&

Re: [HACKERS] CTE inlining

2017-05-04 Thread Petr Jelinek
ase they can easily figure where to add the >> MATERIALIZED option and regain the original performance. > > +1 for option 1. This change will be welcome for a large number of > queries, but forced materialization is a real need and I use it often. > This comes off as a very rea

Re: [HACKERS] snapbuild woes

2017-05-04 Thread Petr Jelinek
but I'm worried this isn't on track to end. > What is missing to make it end? It's tricky 5 bugs, and they are quite sensitive on rare timing/concurrency events. First two are fixed, we can live with 5th being done later (as it's not correctness fix, but very bad performance one). We hav

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-03 Thread Petr Jelinek
On 02/05/17 16:14, Petr Jelinek wrote: > On 02/05/17 15:31, Tom Lane wrote: >> Petr Jelinek <petr.jeli...@2ndquadrant.com> writes: >>> Let me expand, if we don't drop the slot by default when dropping >>> subscription, we'll have a lot of users with dead slots l

Re: [HACKERS] Time based lag tracking for logical replication

2017-05-03 Thread Petr Jelinek
On 03/05/17 08:28, Simon Riggs wrote: > On 23 April 2017 at 01:10, Petr Jelinek <petr.jeli...@2ndquadrant.com> wrote: >> Hi, >> >> The time based lag tracking commit [1] added interface for logging >> progress of replication so that we can report lag as time

Re: [HACKERS] Logical replication - TRAP: FailedAssertion in pgstat.c

2017-05-03 Thread Petr Jelinek
On 02/05/17 20:43, Robert Haas wrote: > On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut > <peter.eisentr...@2ndquadrant.com> wrote: >> On 4/16/17 16:11, Petr Jelinek wrote: >>> Yeah it is, it needs to be fenced to happen only after commit, which is >>> not

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 18:25, Alvaro Herrera wrote: > Petr Jelinek wrote: >> On 02/05/17 18:00, Robert Haas wrote: >>> On Tue, May 2, 2017 at 11:49 AM, Alvaro Herrera >>> <alvhe...@2ndquadrant.com> wrote: >>>> Petr Jelinek wrote: >>>>> So the only

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 22:40, Robert Haas wrote: > On Tue, May 2, 2017 at 3:02 PM, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote: >> That sounds okay. I know PeterE didn't like the lower case and >> underscore separated words for options in the original patch, so I'd >

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 19:42, Robert Haas wrote: > On Tue, May 2, 2017 at 8:45 AM, Petr Jelinek > <petr.jeli...@2ndquadrant.com> wrote: >> I am happy to implement something different, it's quite trivial to >> change. But I am not going to propose anything different as I can't >>

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 18:00, Robert Haas wrote: > On Tue, May 2, 2017 at 11:49 AM, Alvaro Herrera > <alvhe...@2ndquadrant.com> wrote: >> Petr Jelinek wrote: >>> So the only way to fulfill the requirement you stated is to just not try >>> to drop the slot, ever, on DROP

Re: [HACKERS] Logical replication in the same cluster

2017-05-02 Thread Petr Jelinek
On 02/05/17 16:37, Andres Freund wrote: > On 2017-05-02 09:17:27 +0200, Petr Jelinek wrote: >> Yes because otherwise we risk leaving slot on the upstream if the >> command fails downstream. > > Shouldn't temporary slots be able to solve that concern? Create it as > te

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 15:31, Tom Lane wrote: > Petr Jelinek <petr.jeli...@2ndquadrant.com> writes: >> Let me expand, if we don't drop the slot by default when dropping >> subscription, we'll have a lot of users with dead slots laying around >> holding back WAL/catalog_xmin, that'

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 15:14, Petr Jelinek wrote: > On 02/05/17 15:10, Tom Lane wrote: >> Robert Haas <robertmh...@gmail.com> writes: >>>> On Thu, Apr 20, 2017 at 7:46 AM, Petr Jelinek >>>> <petr.jeli...@2ndquadrant.com> wrote: >>>>> DROP S

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 15:10, Tom Lane wrote: > Robert Haas <robertmh...@gmail.com> writes: >>> On Thu, Apr 20, 2017 at 7:46 AM, Petr Jelinek >>> <petr.jeli...@2ndquadrant.com> wrote: >>>> DROP SUBSCRIPTION mysub NODROP SLOT; > >>> I'm pretty uninspi

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 14:13, Tom Lane wrote: > Robert Haas <robertmh...@gmail.com> writes: >> On Thu, Apr 20, 2017 at 7:46 AM, Petr Jelinek >> <petr.jeli...@2ndquadrant.com> wrote: >>> DROP SUBSCRIPTION mysub NODROP SLOT; > >> I'm pretty uninspired by this ch

Re: [HACKERS] Logical replication in the same cluster

2017-05-02 Thread Petr Jelinek
On 02/05/17 04:14, Peter Eisentraut wrote: > On 4/30/17 20:31, Andres Freund wrote: >> On 2017-04-26 23:41:51 +0200, Petr Jelinek wrote: >>> Yes that's result of how logical replication slots work, the transaction >>> that needs to finish is your transacti

Re: [HACKERS] logical replication and PANIC during shutdown checkpoint in publisher

2017-05-02 Thread Petr Jelinek
there - CREATE_REPLICATION_SLOT PHYSICAL will not WAL log and CREATE_REPLICATION_SLOT LOGICAL can't be run without being connected to db, neither can logical decoding and SQL queries, so that coves all. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development,

Re: [HACKERS] snapbuild woes

2017-05-02 Thread Petr Jelinek
On 01/05/17 21:14, Andres Freund wrote: > On 2017-05-01 11:09:44 +0200, Petr Jelinek wrote: >> On 01/05/17 10:03, Andres Freund wrote: >>> On 2017-05-01 03:54:49 +0200, Petr Jelinek wrote: > >>>> But, I still think we need to restart the tracking after

Re: [HACKERS] snapbuild woes

2017-05-01 Thread Petr Jelinek
On 01/05/17 10:03, Andres Freund wrote: > On 2017-05-01 03:54:49 +0200, Petr Jelinek wrote: >> I agree with adding running, I think that's good thing even for the per >> transaction tracking and snapshot exports - we could use the newly added >> field to get rid of the issue w

Re: [HACKERS] snapbuild woes

2017-04-30 Thread Petr Jelinek
On 01/05/17 04:29, Craig Ringer wrote: > On 1 May 2017 at 09:54, Petr Jelinek <petr.jeli...@2ndquadrant.com> wrote: > >> But, I still think we need to restart the tracking after new >> xl_running_xacts. Reason for that is afaics any of the catalog snapshots >> th

Re: [HACKERS] snapbuild woes

2017-04-30 Thread Petr Jelinek
f the supposedly running txes was actually already committed and that transaction might have done catalog changes. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-ha

Re: [HACKERS] Time based lag tracking for logical replication

2017-04-29 Thread Petr Jelinek
On 23/04/17 01:10, Petr Jelinek wrote: > Hi, > > The time based lag tracking commit [1] added interface for logging > progress of replication so that we can report lag as time interval > instead of just bytes. But the patch didn't contain patch for the > builtin logical repl

Re: [HACKERS] Interval for launching the table sync worker

2017-04-27 Thread Petr Jelinek
On 27/04/17 21:00, Peter Eisentraut wrote: > On 4/27/17 06:47, Petr Jelinek wrote: >> One thing I am missing in your patch however is cleanup of entries for >> relations that finished sync. I wonder if it would be enough to just >> destroy the hash when we get to empty lis

Re: [HACKERS] Interval for launching the table sync worker

2017-04-27 Thread Petr Jelinek
I wonder if it would be enough to just destroy the hash when we get to empty list. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes t

Re: [HACKERS] some review comments on logical rep code

2017-04-27 Thread Petr Jelinek
sahiko Sawada <sawada.m...@gmail.com> >>> wrote in >>> <CAD21AoDMy8a6UwMrRh8pigQbDC+JAOQ4m_tXT41VRP4SYp23=w...@mail.gmail.com> >>>> On Wed, Apr 26, 2017 at 12:35 PM, Petr Jelinek >>>> <petr.jeli...@2ndquadrant.com> wrote: >>>&

Re: [HACKERS] Logical replication in the same cluster

2017-04-27 Thread Petr Jelinek
her not hide even more errors in the log. Note that the workaround for all of this is not all that complex, you do same thing (create slot manually) you'd do for physical replication with slots. Thoughts? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Su

Re: [HACKERS] Logical replication in the same cluster

2017-04-26 Thread Petr Jelinek
r transaction. It can be worked around by creating the slot manually via the SQL interface for example and create the subscription using WITH (NOCREATE SLOT, SLOT NAME = 'your slot') . -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Traini

Re: [HACKERS] some review comments on logical rep code

2017-04-25 Thread Petr Jelinek
ing here for this issue. >> >> I agree this as a whole. But I think that the main issue here is >> not false wakeups, but 'possible delay of launching new workers >> by 3 minutes at most' (this is centainly a kind of false wakeups, >> though). We can live with this failure when usin

Re: [HACKERS] Separation walsender & normal backends

2017-04-25 Thread Petr Jelinek
t's good idea, the usefulness if much lower without the initial copy. The original patch for this added new commands to replication protocol, adding generic SQL interface was result of request in the reviews. I personally don't mind moving back my original idea of special commands if that was the cons

Re: [HACKERS] tablesync patch broke the assumption that logical rep depends on?

2017-04-25 Thread Petr Jelinek
On 22/04/17 22:09, Petr Jelinek wrote: > On 21/04/17 16:31, Petr Jelinek wrote: >> On 21/04/17 16:23, Peter Eisentraut wrote: >>> On 4/21/17 10:11, Petr Jelinek wrote: >>>> On 21/04/17 16:09, Peter Eisentraut wrote: >>>>> On 4/20/17 14:29, Petr Jelinek

Re: [HACKERS] PG 10 release notes

2017-04-25 Thread Petr Jelinek
really more than a proof-of-concept > until now. > Huh? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscrip

Re: [HACKERS] PG 10 release notes

2017-04-24 Thread Petr Jelinek
replication=database kind of connection). Or the ability of logical decoding to follow timeline switches. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

<    1   2   3   4   5   6   7   8   9   10   >