On 2016-05-05 13:30:42 +1200, Thomas Munro wrote:
> That was a red herring. I was confused because SUSv2 and POSIX call
> this argument 'errorfds' and say that sockets *also* tell you about
> errors this way. (Many/most real OSs call the argument 'exceptfds'
> instead and only use it to tell you
On Tue, Mar 29, 2016 at 8:17 PM, Michael Paquier
wrote:
> On Tue, Mar 29, 2016 at 1:11 PM, Thomas Munro
> wrote:
>> (BTW, isn't the select call in libpq_select
>> lacking an exceptfds set, and can't it therefore block forever when
>> there is an error condition on the socket and no timeout?)
>
>
On Tue, Apr 5, 2016 at 7:21 PM, Thomas Munro
wrote:
> Thanks. I see a way to move that loop and change the overflow
> behaviour along those lines but due to other commitments I won't be
> able to post a well tested patch and still leave time for reviewers
> and committer before the looming deadli
On Tue, Apr 5, 2016 at 4:17 AM, Robert Haas wrote:
> On Wed, Mar 30, 2016 at 2:22 AM, Thomas Munro
> wrote:
>> On Wed, Mar 30, 2016 at 2:36 PM, Robert Haas wrote:
>>> OK, I committed this, with a few tweaks. In particular, I added a
>>> flag variable instead of relying on "latch set" == "need t
On Wed, Mar 30, 2016 at 2:22 AM, Thomas Munro
wrote:
> On Wed, Mar 30, 2016 at 2:36 PM, Robert Haas wrote:
>> OK, I committed this, with a few tweaks. In particular, I added a
>> flag variable instead of relying on "latch set" == "need to send
>> reply"; the other changes were cosmetic.
>>
>> I'
On Tue, Mar 29, 2016 at 5:47 PM, Thomas Munro
wrote:
> On Wed, Mar 30, 2016 at 6:04 AM, Robert Haas wrote:
>> On Tue, Mar 29, 2016 at 3:17 AM, Michael Paquier
>> wrote:
>>> OK, so I am switching this patch as "Ready for committer", for 0001.
>>> It is in better shape now.
>>
>> Well... I have a
On Wed, Mar 30, 2016 at 6:04 AM, Robert Haas wrote:
> On Tue, Mar 29, 2016 at 3:17 AM, Michael Paquier
> wrote:
>> OK, so I am switching this patch as "Ready for committer", for 0001.
>> It is in better shape now.
>
> Well... I have a few questions yet.
>
> The new argument to SyncRepWaitForLSN
On Tue, Mar 29, 2016 at 3:17 AM, Michael Paquier
wrote:
> OK, so I am switching this patch as "Ready for committer", for 0001.
> It is in better shape now.
Well... I have a few questions yet.
The new argument to SyncRepWaitForLSN is called "bool commit", but
RecordTransactionAbortPrepared passe
On Tue, Mar 29, 2016 at 1:11 PM, Thomas Munro
wrote:
> On Tue, Mar 29, 2016 at 2:28 AM, Michael Paquier
> wrote:
>> I definitely prefer that, that's neater! libpq_select could be
>> simplified because a timeout does not matter much.
>
> Ok, here is a new version that exits the streaming loop corr
On Mon, Mar 28, 2016 at 10:08 PM, Thomas Munro
wrote:
> On Tue, Mar 29, 2016 at 1:56 AM, Thomas Munro
> wrote:
>> On Mon, Mar 28, 2016 at 8:54 PM, Michael Paquier
>> wrote:
>>> I have been also thinking a lot about this patch, and the fact that
>>> the WAL receiver latch is being used within the
On Tue, Mar 29, 2016 at 1:56 AM, Thomas Munro
wrote:
> On Mon, Mar 28, 2016 at 8:54 PM, Michael Paquier
> wrote:
>> I have been also thinking a lot about this patch, and the fact that
>> the WAL receiver latch is being used within the internals of
>> libpqwalreceiver has been bugging me a lot, be
On Sun, Mar 27, 2016 at 7:30 AM, Thomas Munro
wrote:
> On Sat, Mar 26, 2016 at 2:48 AM, Michael Paquier
> wrote:
>> Should we worried about potential backward-incompatibilities with the
>> new return values of walrcv_receive?
>
> There are three changes to the walrcv_receive interface:
>
> 1. It
On Fri, Mar 25, 2016 at 4:51 PM, Thomas Munro
wrote:
> On Thu, Mar 24, 2016 at 12:34 AM, Thomas Munro
> wrote:
>> On Wed, Mar 23, 2016 at 12:37 PM, Robert Haas wrote:
>>> +static void WalRcvUnblockSigUsr2(void)
>>>
>>> And again here.
>>
>> Fixed.
>>
>>> +WalRcvUnblockSigUsr2();
On Thu, Mar 24, 2016 at 11:20 PM, Robert Haas wrote:
> On Thu, Mar 24, 2016 at 2:11 AM, Michael Paquier
> wrote:
>> - SyncRepWaitForLSN(gxact->prepare_end_lsn);
>> + SyncRepWaitForLSN(gxact->prepare_end_lsn, false);
>> Isn't it important to ensure that a PREPARE LSN is applied as well on
>> t
On Thu, Mar 24, 2016 at 2:11 AM, Michael Paquier
wrote:
> On Wed, Mar 23, 2016 at 11:32 PM, Robert Haas wrote:
>> Well, I wouldn't go that far. It seems pretty clear that remote_apply
>> by itself is useful - I can't imagine anybody seriously arguing the
>> contrary, whatever they think of this
On Wed, Mar 23, 2016 at 11:32 PM, Robert Haas wrote:
> Well, I wouldn't go that far. It seems pretty clear that remote_apply
> by itself is useful - I can't imagine anybody seriously arguing the
> contrary, whatever they think of this implementation. My view,
> though, is that by itself that's p
On Wed, Mar 23, 2016 at 8:43 AM, Michael Paquier
wrote:
> On Wed, Mar 23, 2016 at 8:39 PM, Robert Haas wrote:
>> On Wed, Mar 23, 2016 at 7:34 AM, Thomas Munro
>> wrote:
>>> synchronous_commitTPS
>>>
>>> off 9234
>>> local1223
>>> rem
On Wed, Mar 23, 2016 at 8:39 PM, Robert Haas wrote:
> On Wed, Mar 23, 2016 at 7:34 AM, Thomas Munro
> wrote:
>> synchronous_commitTPS
>>
>> off 9234
>> local1223
>> remote_write 907
>> on587
>> remote_appl
On Wed, Mar 23, 2016 at 7:34 AM, Thomas Munro
wrote:
> synchronous_commitTPS
>
> off 9234
> local1223
> remote_write 907
> on587
> remote_apply 555
>
> synchronous_commitTPS
> =
On Mon, Mar 14, 2016 at 2:38 PM, Thomas Munro
wrote:
>> What's with the extra block?
>
> Yeah, that's silly, thanks. Tidied up for the next version.
Some more comments on 0001:
+remote_write, remote_apply,
local, and off.
Extra space.
+ * apply when the transaction eventualll
On Tue, Mar 15, 2016 at 6:58 AM, Robert Haas wrote:
> On Sun, Mar 13, 2016 at 11:50 PM, Thomas Munro
> wrote:
>> The last patches I posted don't apply today due to changes in master,
>> so here's a freshly merged patch series.
>
> +from the current synchronous stanbyindicates it has recei
On Sun, Mar 13, 2016 at 11:50 PM, Thomas Munro
wrote:
> The last patches I posted don't apply today due to changes in master,
> so here's a freshly merged patch series.
+from the current synchronous stanbyindicates it has received the
Uh, no.
-SyncRepWaitForLSN(gxact->prepare_end_ls
On Thu, Mar 3, 2016 at 7:34 PM, Michael Paquier
wrote:
> WAL replay for 2PC should also
> call XLogRequestWalReceiverReply() when needed.
Ah yes, I missed this important sentence. I will address that in the
next version after some testing.
--
Thomas Munro
http://www.enterprisedb.com
--
Sent
On Thu, Mar 3, 2016 at 7:34 PM, Michael Paquier
wrote:
> On Tue, Mar 1, 2016 at 11:53 AM, Thomas Munro
> wrote:
>> On Tue, Mar 1, 2016 at 2:46 PM, Amit Langote
>> wrote:
>>>
>>> Hi,
>>>
>>> On 2016/02/29 18:05, Thomas Munro wrote:
On Mon, Feb 29, 2016 at 9:05 PM, Amit Langote wrote:
> +
On Thu, Mar 3, 2016 at 3:34 PM, Michael Paquier
wrote:
> That's one of my concerns about this patch now: it is trying to do too
> much. I think that there is definitely a need for both things:
> applications may be fine to pay the lagging price when remote_apply is
> used by not having an extra er
On Tue, Mar 1, 2016 at 11:53 AM, Thomas Munro
wrote:
> On Tue, Mar 1, 2016 at 2:46 PM, Amit Langote
> wrote:
>>
>> Hi,
>>
>> On 2016/02/29 18:05, Thomas Munro wrote:
>>> On Mon, Feb 29, 2016 at 9:05 PM, Amit Langote wrote:
+ servers. A transaction that is run with
causal_reads
On Tue, Mar 1, 2016 at 2:46 PM, Amit Langote
wrote:
>
> Hi,
>
> On 2016/02/29 18:05, Thomas Munro wrote:
>> On Mon, Feb 29, 2016 at 9:05 PM, Amit Langote wrote:
>>> + servers. A transaction that is run with
>>> causal_reads set
>>> + to on is guaranteed either to see the effects o
Hi,
On 2016/02/29 18:05, Thomas Munro wrote:
> On Mon, Feb 29, 2016 at 9:05 PM, Amit Langote wrote:
>> + servers. A transaction that is run with
>> causal_reads set
>> + to on is guaranteed either to see the effects of all
>> + completed transactions run on the primary wi
On Mon, Feb 29, 2016 at 6:05 PM, Thomas Munro
wrote:
> "All changes made by the transaction become visible to others ..." --
> which others? But I backed out, that succinct account of COMMIT is 20
> years old, and in any case visibility is tied to committing, not
> specifically to the COMMIT comm
On Mon, Feb 29, 2016 at 9:05 PM, Amit Langote
wrote:
>
> Hi Thomas,
>
> On 2016/02/29 15:20, Thomas Munro wrote:
>> Thanks for looking at the patch! Here is a new version with the
>> following changes:
>>
>> 1. Some draft user documentation has been added, as requested.
>
> Just to clarify, in:
Hi Thomas,
On 2016/02/29 15:20, Thomas Munro wrote:
> Thanks for looking at the patch! Here is a new version with the
> following changes:
>
> 1. Some draft user documentation has been added, as requested.
Just to clarify, in:
+ servers. A transaction that is run with
causal_reads s
On 27 February 2016 at 13:20, Michael Paquier wrote:
> On Mon, Feb 22, 2016 at 9:39 AM, Thom Brown wrote:
>> On 21 February 2016 at 23:18, Thomas Munro
>> wrote:
>> The replay_lag is particularly cool. Didn't think it was possible to
>> glean this information on the primary, but the timings are
On Mon, Feb 22, 2016 at 9:39 AM, Thom Brown wrote:
> On 21 February 2016 at 23:18, Thomas Munro
> wrote:
> The replay_lag is particularly cool. Didn't think it was possible to
> glean this information on the primary, but the timings are correct in
> my tests.
>
> +1 for this patch. Looks like t
On 21 February 2016 at 23:18, Thomas Munro
wrote:
> On Mon, Feb 22, 2016 at 2:10 AM, Thom Brown wrote:
>> On 3 February 2016 at 10:46, Thomas Munro
>> wrote:
>>> On Wed, Feb 3, 2016 at 10:59 PM, Amit Langote
>>> wrote:
There seems to be a copy-pasto there - shouldn't that be:
+
On Mon, Feb 22, 2016 at 2:10 AM, Thom Brown wrote:
> On 3 February 2016 at 10:46, Thomas Munro
> wrote:
>> On Wed, Feb 3, 2016 at 10:59 PM, Amit Langote
>> wrote:
>>> There seems to be a copy-pasto there - shouldn't that be:
>>>
>>> + if (walsndctl->lsn[SYNC_REP_WAIT_FLUSH] < MyWalSnd->flush)
>
On 3 February 2016 at 10:46, Thomas Munro wrote:
> On Wed, Feb 3, 2016 at 10:59 PM, Amit Langote
> wrote:
>> There seems to be a copy-pasto there - shouldn't that be:
>>
>> + if (walsndctl->lsn[SYNC_REP_WAIT_FLUSH] < MyWalSnd->flush)
>
> Indeed, thanks! New patch attached.
I've given this a tes
On Wed, Feb 3, 2016 at 10:59 PM, Amit Langote
wrote:
> There seems to be a copy-pasto there - shouldn't that be:
>
> + if (walsndctl->lsn[SYNC_REP_WAIT_FLUSH] < MyWalSnd->flush)
Indeed, thanks! New patch attached.
--
Thomas Munro
http://www.enterprisedb.com
causal-reads-v6.patch
Description:
Hi Thomas,
On 2016/01/20 13:12, Thomas Munro wrote:
> That one conflicts with b1a9bad9e744857291c7d5516080527da8219854, so
> here is a new version.
-if (walsndctl->lsn[SYNC_REP_WAIT_WRITE] < MyWalSnd->write)
+if (is_highest_priority_sync_standby)
[ ... ]
-if (walsndctl->lsn[SYNC_RE
On Wed, Jan 20, 2016 at 1:12 PM, Thomas Munro
wrote:
> On Wed, Dec 30, 2015 at 5:15 PM, Thomas Munro
> wrote:
>> On Wed, Nov 18, 2015 at 11:50 PM, Thomas Munro
>> wrote:
>>> Here is a new version of the patch with a few small improvements:
>>> ...
>>> [causal-reads-v3.patch]
>>
>> That didn't ap
On Wed, Dec 30, 2015 at 5:15 PM, Thomas Munro
wrote:
> On Wed, Nov 18, 2015 at 11:50 PM, Thomas Munro
> wrote:
>> Here is a new version of the patch with a few small improvements:
>> ...
>> [causal-reads-v3.patch]
>
> That didn't apply after 6e7b3359 (which fix a typo in a comment that I
> moved)
+1 to both the feature and the concept of how it's implemented.
Haven't looked at the code though.
This feature would be very useful for us at Trustly.
This would mean we got get rid of an entire system component in our
architecture (=memcached) which we only use to write data which must
be immedi
On Wed, Nov 18, 2015 at 11:50 PM, Thomas Munro
wrote:
> Here is a new version of the patch with a few small improvements:
> ...
> [causal-reads-v3.patch]
That didn't apply after 6e7b3359 (which fix a typo in a comment that I
moved). Here is a new version that does.
--
Thomas Munro
http://www.e
Hi,
Here is a new version of the patch with a few small improvements:
1. Adopted the term '[read] lease', replacing various hand-wavy language
in the comments and code. That seems to be the established term for this
approach[1].
2. Reduced the stalling time on failure. When things go wrong w
On 11/12/15 1:11 PM, Thomas Munro wrote:
It's true that a pooling system/middleware could spy on your sessions
and insert causality token handling imposing a global ordering of
visibility for you, so that naive users don't have to deal with them.
Whenever it sees a COMMIT result (assuming they ar
On Mon, Nov 16, 2015 at 5:44 AM, Simon Riggs wrote:
> On 15 November 2015 at 14:50, Robert Haas wrote:
>> On Sun, Nov 15, 2015 at 5:41 AM, Simon Riggs
>> wrote:
>> > Hmm, if that's where we're at, I'll summarize my thoughts.
>> >
>> > All of this discussion presupposes we are distributing/load b
On Tue, Nov 17, 2015 at 12:44 AM, Simon Riggs wrote:
> On 15 November 2015 at 10:41, Simon Riggs wrote:
>
>
>> So anyway, consider me nudged to finish my patch to provide capability
>> for that by 1 Jan.
>>
>
> My earlier patch aimed to allow WALReceiver to wait on both a latch and a
> socket a
On 15 November 2015 at 10:41, Simon Riggs wrote:
> So anyway, consider me nudged to finish my patch to provide capability
> for that by 1 Jan.
>
My earlier patch aimed to allow WALReceiver to wait on both a latch and a
socket as well as allow WALWriter to be active, so that WALReceiver could
r
On 16 November 2015 at 11:01, Craig Ringer wrote:
> On 16 November 2015 at 18:44, Simon Riggs wrote:
>
>>
>> The pooler knows which statements are reads and writes
>>
>
> I think that's an iffy assumption.
>
It's not an assumption, its a requirement. If it can't do this in some
manner then you
On 16 November 2015 at 18:44, Simon Riggs wrote:
>
> The pooler knows which statements are reads and writes
>
I think that's an iffy assumption. It's one we tend to make because
otherwise read/write pooling won't work, but in PostgreSQL there's really
no way to know when calling a function.
Wha
On 15 November 2015 at 14:50, Robert Haas wrote:
> On Sun, Nov 15, 2015 at 5:41 AM, Simon Riggs
> wrote:
> > Hmm, if that's where we're at, I'll summarize my thoughts.
> >
> > All of this discussion presupposes we are distributing/load balancing
> > queries so that reads and writes might occur o
On Sun, Nov 15, 2015 at 11:41 PM, Simon Riggs wrote:
> On 12 November 2015 at 18:25, Thomas Munro
> wrote:
>
>
>> I don't want to get bogged down in details, while we're talking about
>> the 30,000 foot view).
>>
>
> Hmm, if that's where we're at, I'll summarize my thoughts.
>
> All of this dis
On Sun, Nov 15, 2015 at 5:41 AM, Simon Riggs wrote:
> Hmm, if that's where we're at, I'll summarize my thoughts.
>
> All of this discussion presupposes we are distributing/load balancing
> queries so that reads and writes might occur on different nodes.
Agreed. I think that's a pretty common pat
On 12 November 2015 at 18:25, Thomas Munro
wrote:
> I don't want to get bogged down in details, while we're talking about the
> 30,000 foot view).
>
Hmm, if that's where we're at, I'll summarize my thoughts.
All of this discussion presupposes we are distributing/load balancing
queries so that
On Fri, Nov 13, 2015 at 1:16 AM, Simon Riggs wrote:
> On 11 November 2015 at 09:22, Thomas Munro
> wrote:
>
>
>> 1. Reader waits with exposed LSNs, as Heikki suggests. This is what
>> BerkeleyDB does in "read-your-writes" mode. It means that application
>> developers have the responsibility f
On Fri, Nov 13, 2015 at 1:16 AM, Simon Riggs wrote:
> On 11 November 2015 at 09:22, Thomas Munro
> wrote:
>
>
>> 1. Reader waits with exposed LSNs, as Heikki suggests. This is what
>> BerkeleyDB does in "read-your-writes" mode. It means that application
>> developers have the responsibility f
On 11 November 2015 at 09:22, Thomas Munro
wrote:
> 1. Reader waits with exposed LSNs, as Heikki suggests. This is what
> BerkeleyDB does in "read-your-writes" mode. It means that application
> developers have the responsibility for correctly identifying transactions
> with causal dependencie
On Thu, Nov 12, 2015 at 12:10 AM, Ants Aasma wrote:
> On Wed, Nov 11, 2015 at 11:22 AM, Thomas Munro
> wrote:
> > On Wed, Nov 11, 2015 at 9:42 PM, Heikki Linnakangas
> wrote:
> >> On 11/11/2015 10:23 AM, Simon Riggs wrote:
> >>> Thanks for working on this issue.
> >>
> >> +1.
>
> +1. I have see
On 11/11/15 4:22 AM, Thomas Munro wrote:
> My thinking was that the reason for wanting to load balance over a set
> of hot standbys is because you have a very read-heavy workload, so it
> makes sense to tax the writers and leave the many dominant readers
> unburdened, so (3) should be better than (
On Wed, Nov 11, 2015 at 3:23 AM, Simon Riggs wrote:
> This causes every writer to wait.
>
> What we want is to isolate the wait only to people performing a write-read
> sequence, so I think it should be readers that wait. Let's have that debate
> up front before we start reviewing the patch.
One
On Wed, Nov 11, 2015 at 11:22 AM, Thomas Munro
wrote:
> On Wed, Nov 11, 2015 at 9:42 PM, Heikki Linnakangas wrote:
>> On 11/11/2015 10:23 AM, Simon Riggs wrote:
>>> Thanks for working on this issue.
>>
>> +1.
+1. I have seen a lot of interest for something along these lines.
>> I'm thinking the
On Wed, Nov 11, 2015 at 9:42 PM, Heikki Linnakangas wrote:
> On 11/11/2015 10:23 AM, Simon Riggs wrote:
>
>> On 11 November 2015 at 05:37, Thomas Munro > >
>> wrote:
>>
>> Many sites use hot standby servers to spread read-heavy workloads over
>> more
>>
>>> hardware, or at least would like to. T
> I'm thinking the client should get some kind of a token back from the
commit, and it could use the token on the standby, to wait for that commit
to be applied. The token could be just the XID, or the LSN of the commit
record. Or the application could generate the token and pass it to the
server i
On 11/11/2015 10:23 AM, Simon Riggs wrote:
On 11 November 2015 at 05:37, Thomas Munro
wrote:
Many sites use hot standby servers to spread read-heavy workloads over more
hardware, or at least would like to. This works well today if your
application can tolerate some time lag on standbys. The
On 11 November 2015 at 05:37, Thomas Munro
wrote:
Many sites use hot standby servers to spread read-heavy workloads over more
> hardware, or at least would like to. This works well today if your
> application can tolerate some time lag on standbys. The problem is that
> there is no guarantee of
Hi hackers,
Many sites use hot standby servers to spread read-heavy workloads over more
hardware, or at least would like to. This works well today if your
application can tolerate some time lag on standbys. The problem is that
there is no guarantee of when a particular commit will become visible
65 matches
Mail list logo