Re: [HACKERS] snapshot too large error when initializing logical replication (9.4)
On 2015-03-20 08:41:38 -0400, Bruce Momjian wrote: On Mon, Nov 17, 2014 at 03:52:38PM +0100, Andres Freund wrote: On 2014-11-17 11:51:54 -0300, Alvaro Herrera wrote: Andres Freund wrote: Hi, On 2014-10-25 18:09:36 -0400, Steve Singer wrote: I sometimes get the error snapshot too large from my logical replication walsender process when in response to a CREATE_REPLICATION_SLOT. Yes. That's possible if 'too much' was going on until a consistent point was reached. I think we can just use a much larger size for the array if necessary. I've attached patch for this. Could you try whether that helps? I don't have a testcase handy that reproduces the problem. You haven't pushed this, have you? No, but it's on my todo list. Uh, where are we on this? Nowhere, unfortunately. The fix I'd developed isn't actually correct. The problem is that we can only import a transaction with fewer than GetMaxSnapshotXidCount() transactions atm. The only reason it worked in Steve's case is because he probably wasn't importin gthe snapshot in the test case... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] snapshot too large error when initializing logical replication (9.4)
On Mon, Nov 17, 2014 at 03:52:38PM +0100, Andres Freund wrote: On 2014-11-17 11:51:54 -0300, Alvaro Herrera wrote: Andres Freund wrote: Hi, On 2014-10-25 18:09:36 -0400, Steve Singer wrote: I sometimes get the error snapshot too large from my logical replication walsender process when in response to a CREATE_REPLICATION_SLOT. Yes. That's possible if 'too much' was going on until a consistent point was reached. I think we can just use a much larger size for the array if necessary. I've attached patch for this. Could you try whether that helps? I don't have a testcase handy that reproduces the problem. You haven't pushed this, have you? No, but it's on my todo list. Uh, where are we on this? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + Everyone has their own god. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] snapshot too large error when initializing logical replication (9.4)
Andres Freund wrote: Hi, On 2014-10-25 18:09:36 -0400, Steve Singer wrote: I sometimes get the error snapshot too large from my logical replication walsender process when in response to a CREATE_REPLICATION_SLOT. Yes. That's possible if 'too much' was going on until a consistent point was reached. I think we can just use a much larger size for the array if necessary. I've attached patch for this. Could you try whether that helps? I don't have a testcase handy that reproduces the problem. You haven't pushed this, have you? -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] snapshot too large error when initializing logical replication (9.4)
On 2014-11-17 11:51:54 -0300, Alvaro Herrera wrote: Andres Freund wrote: Hi, On 2014-10-25 18:09:36 -0400, Steve Singer wrote: I sometimes get the error snapshot too large from my logical replication walsender process when in response to a CREATE_REPLICATION_SLOT. Yes. That's possible if 'too much' was going on until a consistent point was reached. I think we can just use a much larger size for the array if necessary. I've attached patch for this. Could you try whether that helps? I don't have a testcase handy that reproduces the problem. You haven't pushed this, have you? No, but it's on my todo list. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] snapshot too large error when initializing logical replication (9.4)
On 10/28/2014 01:27 PM, Andres Freund wrote: Hi, On 2014-10-25 18:09:36 -0400, Steve Singer wrote: I sometimes get the error snapshot too large from my logical replication walsender process when in response to a CREATE_REPLICATION_SLOT. Yes. That's possible if 'too much' was going on until a consistent point was reached. I think we can just use a much larger size for the array if necessary. I've attached patch for this. Could you try whether that helps? I don't have a testcase handy that reproduces the problem. This patch seems to fix things. I've done numerous runs of the test with I was doing before with your patch applied and don't seem to be having this issue anymore. This is in SnapBuildExportSnapshot in snapbuild.c newxcnt is 212 at that point I have max_connections = 200 procArray-maxProcs=212 Should we be testing newxcnt GetMaxSnapshotXidCount() instead of newxcnt = GetMaxSnapshotXidCount() It actually looks correct to me new - newxcnt is used as an offset into an array of size GetMaxSnapshotXidCount(). Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] snapshot too large error when initializing logical replication (9.4)
Hi,
On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
I sometimes get the error snapshot too large from my logical replication
walsender process when in response to a CREATE_REPLICATION_SLOT.
Yes. That's possible if 'too much' was going on until a consistent point
was reached. I think we can just use a much larger size for the array
if necessary.
I've attached patch for this. Could you try whether that helps? I don't
have a testcase handy that reproduces the problem.
This is in SnapBuildExportSnapshot in snapbuild.c
newxcnt is 212 at that point
I have max_connections = 200
procArray-maxProcs=212
Should we be testing
newxcnt GetMaxSnapshotXidCount()
instead of
newxcnt = GetMaxSnapshotXidCount()
It actually looks correct to me new - newxcnt is used as an offset into
an array of size GetMaxSnapshotXidCount().
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training Services
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 5e59c6b..2df1905 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -519,6 +519,7 @@ SnapBuildExportSnapshot(SnapBuild *builder)
TransactionId xid;
TransactionId *newxip;
int newxcnt = 0;
+ int newxcnt_max;
if (builder-state != SNAPBUILD_CONSISTENT)
elog(ERROR, cannot export a snapshot before reaching a consistent state);
@@ -557,8 +558,9 @@ SnapBuildExportSnapshot(SnapBuild *builder)
MyPgXact-xmin = snap-xmin;
/* allocate in transaction context */
+ newxcnt_max = GetMaxSnapshotXidCount();
newxip = (TransactionId *)
- palloc(sizeof(TransactionId) * GetMaxSnapshotXidCount());
+ palloc(sizeof(TransactionId) * newxcnt_max);
/*
* snapbuild.c builds transactions in an inverted manner, which means it
@@ -579,8 +581,11 @@ SnapBuildExportSnapshot(SnapBuild *builder)
if (test == NULL)
{
- if (newxcnt = GetMaxSnapshotXidCount())
-elog(ERROR, snapshot too large);
+ if (newxcnt = newxcnt_max)
+ {
+newxcnt_max *= 2;
+newxip = repalloc(newxip, sizeof(TransactionId) * newxcnt_max);
+ }
newxip[newxcnt++] = xid;
}
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] snapshot too large error when initializing logical replication (9.4)
I sometimes get the error snapshot too large from my logical replication walsender process when in response to a CREATE_REPLICATION_SLOT. This is in SnapBuildExportSnapshot in snapbuild.c newxcnt is 212 at that point I have max_connections = 200 procArray-maxProcs=212 Should we be testing newxcnt GetMaxSnapshotXidCount() instead of newxcnt = GetMaxSnapshotXidCount() -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
