Patch in the previous letter is full of faulties. Please, use new version.
Also, here we fixed the problem with loosing CSN value in a parallel
worker (TAP test 003_parallel_safe.pl). Thanks for a.pyhalov for the
problem detection and a bugfix.
--
regards,
Andrey Lepikhov
Postgres Professional
Next version of CSN implementation in snapshots to achieve a proper
snapshot isolation in the case of a cross-instance distributed transaction.
--
regards,
Andrey Lepikhov
Postgres Professional
>From bbb7dd1d7621c091f11e697d3d894fe7a36918a6 Mon Sep 17 00:00:00 2001
From: Andrey Lepikhov
Date: W
From: Andrey V. Lepikhov
> Current state of the patch set rebased on master, 5aed6a1fc2.
>
> It is development version. Here some problems with visibility still detected
> in
> two tests:
> 1. CSN Snapshot module - TAP test on time skew.
> 2. Clock SI implementation - TAP test on emulation of ba
From: Andrey V. Lepikhov
> After researching in depth, I think this is the real problem.
> My idea was that we are not using real clocks, we only use clock ticks to
> measure time intervals. It can also be interpreted as a kind of clock.
Yes, patent claims tend to be written to cover broad interp
On 1/1/21 8:14 AM, tsunakawa.ta...@fujitsu.com wrote:
--
11. A method comprising:
receiving information relating to a distributed database transaction operating
on data in data stores associated with respective participating nodes
associated with
Hello, Andrey-san, all,
Based on the request at HighGo's sharding meeting, I'm re-sending the
information on Commitment Ordering that could be used for global visibility.
Their patents have already expired.
--
Have anyone examined the followin
On 2021/01/01 12:14, tsunakawa.ta...@fujitsu.com wrote:
Hello,
Fujii-san and I discussed how to move the scale-out development forward. We
are both worried that Clock-SI is (highly?) likely to infringe the said
Microsoft's patent. So we agreed we are going to investigate the Clock-SI and
Hello,
Fujii-san and I discussed how to move the scale-out development forward. We
are both worried that Clock-SI is (highly?) likely to infringe the said
Microsoft's patent. So we agreed we are going to investigate the Clock-SI and
the patent, and if we have to conclude that we cannot embra
Fujii-san, Sawada-san, all,
From: Fujii Masao
> Yeah, so if we need to guarantee the transaction linearizability even
> in distributed env (probably this is yes. Right?), using only Clock-SI
> is not enough. We would need to implement something more
> in addition to Clock-SI or adopt the differen
On 2020/10/23 11:58, Masahiko Sawada wrote:
On Thu, 15 Oct 2020 at 01:41, Fujii Masao wrote:
On 2020/09/17 15:56, Amit Kapila wrote:
On Thu, Sep 10, 2020 at 4:20 PM Fujii Masao wrote:
One alternative is to add only hooks into PostgreSQL core so that we can
implement the global transa
On Thu, 15 Oct 2020 at 01:41, Fujii Masao wrote:
>
>
>
> On 2020/09/17 15:56, Amit Kapila wrote:
> > On Thu, Sep 10, 2020 at 4:20 PM Fujii Masao
> > wrote:
> >>
> One alternative is to add only hooks into PostgreSQL core so that we can
> implement the global transaction management outs
On 2020/09/17 15:56, Amit Kapila wrote:
On Thu, Sep 10, 2020 at 4:20 PM Fujii Masao wrote:
One alternative is to add only hooks into PostgreSQL core so that we can
implement the global transaction management outside. This idea was
discussed before as the title "eXtensible Transaction Manag
Hi Andrey san, all,
From: tsunakawa.ta...@fujitsu.com
> And please wait. As below, the patent holder just says that Clock-SI is not
> based on the patent and an independent development. He doesn't say
> Clock-SI does not overlap with the patent or implementing Clock-SI does not
> infringe on th
From: Andrey Lepikhov
> Thank you for this work!
> As I can see, main development difficulties placed in other areas: CSN,
> resolver,
> global deadlocks, 2PC commit... I'm not lawyer too. But if we get remarks from
> the patent holders, we can rewrite our Clock-SI implementation.
Yeah, I unders
22.09.2020 03:47, tsunakawa.ta...@fujitsu.com пишет:
Does this make sense from your viewpoint, and can we think that we can use
Clock-SI without infrindging on the patent? According to the patent holder,
the difference between Clock-SI and the patent seems to be fewer than the
similarities
Hi Andrey-san, all,
From: Andrey V. Lepikhov
> On 7/27/20 11:22 AM, tsunakawa.ta...@fujitsu.com wrote:
> > Could you take a look at this patent? I'm afraid this is the Clock-SI for
> > MVCC.
> Microsoft holds this until 2031. I couldn't find this with the keyword
> "Clock-SI.""
> >
> >
> > US8
On 2020-09-18 00:54, Bruce Momjian wrote:
On Tue, Sep 8, 2020 at 01:36:16PM +0300, Alexey Kondratov wrote:
Thank you for the link!
After a quick look on the Sawada-san's patch set I think that there
are two
major differences:
1. There is a built-in foreign xacts resolver in the [1], which s
On Tue, Sep 8, 2020 at 01:36:16PM +0300, Alexey Kondratov wrote:
> Thank you for the link!
>
> After a quick look on the Sawada-san's patch set I think that there are two
> major differences:
>
> 1. There is a built-in foreign xacts resolver in the [1], which should be
> much more convenient fro
On Thu, Sep 10, 2020 at 4:20 PM Fujii Masao wrote:
>
> >> One alternative is to add only hooks into PostgreSQL core so that we can
> >> implement the global transaction management outside. This idea was
> >> discussed before as the title "eXtensible Transaction Manager API".
> >
> > Yeah, I read t
On 2020-09-09 20:29, Fujii Masao wrote:
On 2020/09/09 2:00, Alexey Kondratov wrote:
According to the Sawada-san's v25 0002 the logic is pretty much the
same there:
+2. Pre-Commit phase (1st phase of two-phase commit)
+3. Commit locally
+Once we've prepared all of them, commit the transactio
On 2020/09/10 18:01, tsunakawa.ta...@fujitsu.com wrote:
From: Fujii Masao
But I'm concerned about that it's really hard to say there is no patent risk
around that. I'm not sure who can judge there is no patent risk,
in the community. Maybe no one? Anyway, I was thinking that Google Spanner,
From: Fujii Masao
> But I'm concerned about that it's really hard to say there is no patent risk
> around that. I'm not sure who can judge there is no patent risk,
> in the community. Maybe no one? Anyway, I was thinking that Google Spanner,
> YugabyteDB, etc use the global transaction approach ba
On 2020/09/10 10:38, tsunakawa.ta...@fujitsu.com wrote:
Hi Andrey san,
From: Andrey V. Lepikhov > > From:
tsunakawa.ta...@fujitsu.com
While Clock-SI seems to be considered the best promising for global
Could you take a look at this patent? I'm afraid this is the Clock-SI for MVCC.
Micro
Hi Andrey san,
From: Andrey V. Lepikhov > > From:
tsunakawa.ta...@fujitsu.com
> >> While Clock-SI seems to be considered the best promising for global
> >>> > Could you take a look at this patent? I'm afraid this is the Clock-SI
> >>> > for MVCC.
> Microsoft holds this until 2031. I couldn't
On 2020/09/09 2:00, Alexey Kondratov wrote:
On 2020-09-08 14:48, Fujii Masao wrote:
On 2020/09/08 19:36, Alexey Kondratov wrote:
On 2020-09-08 05:49, Fujii Masao wrote:
On 2020/09/05 3:31, Alexey Kondratov wrote:
Attached is a patch, which implements a plain 2PC in the postgres_fdw and ad
On 2020-09-09 08:35, Masahiko Sawada wrote:
On Wed, 9 Sep 2020 at 02:00, Alexey Kondratov
wrote:
On 2020-09-08 14:48, Fujii Masao wrote:
>
> IIUC, yes, the information required for automatic resolution is
> WAL-logged and the standby tries to resolve those orphan transactions
> from WAL after
On Wed, 9 Sep 2020 at 02:00, Alexey Kondratov
wrote:
>
> On 2020-09-08 14:48, Fujii Masao wrote:
> > On 2020/09/08 19:36, Alexey Kondratov wrote:
> >> On 2020-09-08 05:49, Fujii Masao wrote:
> >>> On 2020/09/05 3:31, Alexey Kondratov wrote:
>
> Attached is a patch, which implements a pla
On 2020-09-08 14:48, Fujii Masao wrote:
On 2020/09/08 19:36, Alexey Kondratov wrote:
On 2020-09-08 05:49, Fujii Masao wrote:
On 2020/09/05 3:31, Alexey Kondratov wrote:
Attached is a patch, which implements a plain 2PC in the
postgres_fdw and adds a GUC 'postgres_fdw.use_twophase'. Also it
On 2020/09/08 19:36, Alexey Kondratov wrote:
On 2020-09-08 05:49, Fujii Masao wrote:
On 2020/09/05 3:31, Alexey Kondratov wrote:
Attached is a patch, which implements a plain 2PC in the postgres_fdw and adds
a GUC 'postgres_fdw.use_twophase'. Also it solves these errors handling issues
ab
On 2020-09-08 05:49, Fujii Masao wrote:
On 2020/09/05 3:31, Alexey Kondratov wrote:
Attached is a patch, which implements a plain 2PC in the postgres_fdw
and adds a GUC 'postgres_fdw.use_twophase'. Also it solves these
errors handling issues above and tries to add proper comments
everywhere.
First, IIUC atomicity of the distributed transaction in the postgres_fdw is
achieved by the usage of 2PC. I think that this postgres_fdw 2PC support should
be separated from global snapshots.
Agreed.
It could be useful to have such atomic distributed transactions even without a
proper visib
buted transaction in the postgres_fdw
is achieved by the usage of 2PC. I think that this postgres_fdw 2PC
support should be separated from global snapshots. It could be useful to
have such atomic distributed transactions even without a proper
visibility, which is guaranteed by the global snapshot. Espec
On 7/27/20 11:22 AM, tsunakawa.ta...@fujitsu.com wrote:
Hi Andrey san, Movead san,
From: tsunakawa.ta...@fujitsu.com
While Clock-SI seems to be considered the best promising for global
serializability here,
* Why does Clock-SI gets so much attention? How did Clock-SI become the
only choice?
Hi Andrey san, Movead san,
From: tsunakawa.ta...@fujitsu.com
> While Clock-SI seems to be considered the best promising for global
> serializability here,
>
> * Why does Clock-SI gets so much attention? How did Clock-SI become the
> only choice?
>
> * Clock-SI was devised in Microsoft Researc
On Mon, 13 Jul 2020 at 20:18, Amit Kapila wrote:
>
> On Fri, Jul 10, 2020 at 8:46 AM Masahiko Sawada
> wrote:
> >
> > On Wed, 8 Jul 2020 at 21:35, Amit Kapila wrote:
> > >
> > >
> > > Cool. While studying, if you can try to think whether this approach is
> > > different from the global coordinat
Hello,
While I'm thinking of the following issues of the current approach Andrey
raised, I'm getting puzzled and can't help asking certain things. Please
forgive me if I'm missing some discussions in the past.
> 1. Dependency on clocks synchronization
> 2. Needs guarantees of monotonically inc
On Fri, Jul 10, 2020 at 8:46 AM Masahiko Sawada
wrote:
>
> On Wed, 8 Jul 2020 at 21:35, Amit Kapila wrote:
> >
> >
> > Cool. While studying, if you can try to think whether this approach is
> > different from the global coordinator based approach then it would be
> > great. Here is my initial th
On Wed, 8 Jul 2020 at 21:35, Amit Kapila wrote:
>
> On Wed, Jul 8, 2020 at 11:16 AM Masahiko Sawada
> wrote:
> >
> > On Tue, 7 Jul 2020 at 15:40, Amit Kapila wrote:
> > >
> > >
> > > Okay, but isn't there some advantage with this approach (manage 2PC at
> > > postgres_fdw level) as well which is
On Wed, Jul 8, 2020 at 11:16 AM Masahiko Sawada
wrote:
>
> On Tue, 7 Jul 2020 at 15:40, Amit Kapila wrote:
> >
> >
> > Okay, but isn't there some advantage with this approach (manage 2PC at
> > postgres_fdw level) as well which is that any node will be capable of
> > handling global transactions
t; > >
> > > >Also, can you let us know if this
> > > > > supports 2PC in some way and if so how is it different from what the
> > > > > other thread on the same topic [1] is trying to achieve?
> > > > Yes, the patch '0003-p
2PC in some way and if so how is it different from what the
> > > > other thread on the same topic [1] is trying to achieve?
> > > Yes, the patch '0003-postgres_fdw-support-for-global-snapshots' contains
> > > 2PC machinery. Now I'd not judge whic
e
> > distributed transaction manager.
> >
>
> When you say "single distributed transaction manager" do you mean
> something like pg_dtm which is inspired by Postgres-XL?
>
> >Also, can you let us know if this
> > > supports 2PC in some way and if s
On Mon, Jun 22, 2020 at 8:36 PM Bruce Momjian wrote:
>
> On Sat, Jun 20, 2020 at 05:51:21PM +0530, Amit Kapila wrote:
> > I feel if Sawada-San or someone involved in another patch also once
> > studies this approach and try to come up with some form of comparison
> > then we might be able to make
On Sat, Jun 20, 2020 at 05:51:21PM +0530, Amit Kapila wrote:
> I feel if Sawada-San or someone involved in another patch also once
> studies this approach and try to come up with some form of comparison
> then we might be able to make better decision. It is possible that
> there are few good thing
On Sat, Jun 20, 2020 at 05:54:18PM +0530, Amit Kapila wrote:
> On Fri, Jun 19, 2020 at 6:33 PM Bruce Momjian wrote:
> >
> > On Fri, Jun 19, 2020 at 05:03:20PM +0800, movead...@highgo.ca wrote:
> > >
> > > >> would like to know if the patch related to CSN based snapshot [2] is a
> > > >> precursor
On Sat, Jun 20, 2020 at 5:51 PM Amit Kapila wrote:
>
>
> So, is anyone working on improving these parts of the patch. AFAICS
> from what Bruce has shared [1],
>
oops, forgot to share the link [1] -
https://wiki.postgresql.org/wiki/WIP_PostgreSQL_Sharding
--
With Regards,
Amit Kapila.
Enterpris
know if this
> > supports 2PC in some way and if so how is it different from what the
> > other thread on the same topic [1] is trying to achieve?
> Yes, the patch '0003-postgres_fdw-support-for-global-snapshots' contains
> 2PC machinery. Now I'd not judge which appro
On Fri, Jun 19, 2020 at 6:33 PM Bruce Momjian wrote:
>
> On Fri, Jun 19, 2020 at 05:03:20PM +0800, movead...@highgo.ca wrote:
> >
> > >> would like to know if the patch related to CSN based snapshot [2] is a
> > >> precursor for this, if not, then is it any way related to this patch
> > >> because
On Fri, Jun 19, 2020 at 05:03:20PM +0800, movead...@highgo.ca wrote:
>
> >> would like to know if the patch related to CSN based snapshot [2] is a
> >> precursor for this, if not, then is it any way related to this patch
> >> because I see the latest reply on that thread [2] which says it is an
>
>> would like to know if the patch related to CSN based snapshot [2] is a
>> precursor for this, if not, then is it any way related to this patch
>> because I see the latest reply on that thread [2] which says it is an
>> infrastructure of sharding feature but I don't understand completely
>> whet
n you let us know if this
supports 2PC in some way and if so how is it different from what the
other thread on the same topic [1] is trying to achieve?
Yes, the patch '0003-postgres_fdw-support-for-global-snapshots' contains
2PC machinery. Now I'd not judge which approach is better.
On Wed, Jun 10, 2020 at 8:36 AM Andrey V. Lepikhov
wrote:
>
>
> On 09.06.2020 11:41, Fujii Masao wrote:
> >
> >
> > The patches seem not to be registered in CommitFest yet.
> > Are you planning to do that?
> Not now. It is a sharding-related feature. I'm not sure that this
> approach is fully cons
de/storage/lwlock.h
@@ -197,6 +197,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_XACT_BUFFER = NUM_INDIVIDUAL_LWLOCKS,
LWTRANCHE_COMMITTS_BUFFER,
LWTRANCHE_SUBTRANS_BUFFER,
+ LWTRANCHE_GLOBAL_CSN_LOG_BUFFERS,
LWTRANCHE_MULTIXACTOFFSET_BUFFER,
LWTRANCHE_MULTIXACTMEMBER_BUFFER,
LWTRANCHE_NOT
On 2020/05/12 19:24, Andrey Lepikhov wrote:
Rebased onto current master (fb544735f1).
Thanks for the patches!
These patches are no longer applied cleanly and caused the compilation failure.
So could you rebase and update them?
The patches seem not to be registered in CommitFest yet.
Are yo
OCKS,
LWTRANCHE_COMMITTS_BUFFERS,
LWTRANCHE_SUBTRANS_BUFFERS,
+ LWTRANCHE_GLOBAL_CSN_LOG_BUFFERS,
LWTRANCHE_MXACTOFFSET_BUFFERS,
LWTRANCHE_MXACTMEMBER_BUFFERS,
LWTRANCHE_ASYNC_BUFFERS,
diff --git a/src/include/utils/snapshot.h b/src/include/utils/snapshot.h
index 4796edb63a..57d2dfaa67 100644
--
Hi!
> 30 нояб. 2018 г., в 18:00, Stas Kelvich написал(а):
>
>
> <0001-GlobalCSNLog-SLRU-v3.patch><0002-Global-snapshots-v3.patch><0003-postgres_fdw-support-for-global-snapshots-v3.patch>
In spite of recent backup discussions I realized that we need to backup
> On 31 Jan 2019, at 18:42, Andres Freund wrote:
>
> Hi,
>
> On 2018-11-30 16:00:17 +0300, Stas Kelvich wrote:
>>> On 29 Nov 2018, at 18:21, Dmitry Dolgov <9erthali...@gmail.com> wrote:
>>> Is there any resulting patch where the ideas how to implement this are
>>> outlined?
>>
>> Not yet. I
Hi,
On 2018-11-30 16:00:17 +0300, Stas Kelvich wrote:
> > On 29 Nov 2018, at 18:21, Dmitry Dolgov <9erthali...@gmail.com> wrote:
> > Is there any resulting patch where the ideas how to implement this are
> > outlined?
>
> Not yet. I’m going to continue work on this in January. And probably try t
> On Wed, Jul 25, 2018 at 1:35 PM Arseny Sher wrote:
>
> Hello,
>
> I have looked through the patches and found them pretty accurate. I'd
> fixed a lot of small issues here and there; updated patchset is
> attached.
Hi,
Thank you for working on this patch. Unfortunately, the patch has some
confl
dlock.
> Can we do something with this deadlock? Will placing an upper limit on
> time of InDoubt state fix the issue? I understand that aborting
> automatically is kind of dangerous...
Sure, this is just a generalization of basic deadlock problem to
distributed system. To deal with it,
Hi!
I want to review this patch set. Though I understand that it probably will be
quite long process.
I like the idea that with this patch set universally all postgres instances are
bound into single distributed DB, even if they never heard about each other
before :) This is just amazing. Or d
_SUBTRANS_BUFFERS,
+ LWTRANCHE_GLOBAL_CSN_LOG_BUFFERS,
LWTRANCHE_MXACTOFFSET_BUFFERS,
LWTRANCHE_MXACTMEMBER_BUFFERS,
LWTRANCHE_ASYNC_BUFFERS,
diff --git a/src/include/utils/snapshot.h b/src/include/utils/snapshot.h
index a8a5a8f4c0..318d41e6f7 100644
--- a/src/include/utils/snapshot.h
+++ b/s
> On 15 May 2018, at 15:53, Robert Haas wrote:
>
> Actually, I think if we're going to pursue that approach, we ought to
> back off a bit from thinking about global snapshots and think about
> what kind of general mechanism we want. For example, maybe you can
> imagin
want that thing in general. It means we don't want yours. :-)
Actually, I think if we're going to pursue that approach, we ought to
back off a bit from thinking about global snapshots and think about
what kind of general mechanism we want. For example, maybe you can
imagine it like a messag
.
> With this patch, can we start a remote transaction at READ COMMITTED
> with imported a global snapshot if the local transaction started at
> READ COMMITTED?
In theory it is possible, one just need to send new snapshot before each
statement. With some amount of careful work it is poss
> On 9 May 2018, at 17:51, Robert Haas wrote:
>
> Ouch. That's not so bad at READ COMMITTED, but at higher isolation
> levels failure becomes extremely likely. Any multi-statement
> transaction that lasts longer than global_snapshot_defer_time is
> pretty much doomed.
Ouch indeed. Current xm
On Fri, May 4, 2018 at 2:11 AM, Stas Kelvich wrote:
>
>
>> On 3 May 2018, at 18:28, Masahiko Sawada wrote:
>>
>> On Wed, May 2, 2018 at 1:27 AM, Stas Kelvich
>> wrote:
>>> 1) To achieve commit atomicity of different nodes intermediate step is
>>> introduced: at first running transaction is ma
read by another, but if there's a portion of the data
that is only read and not written, or at least not written by any
transactions that were active around the same time, then it's fine.
While the details aren't really clear to me, I'm inclined to think
that any s
> On 7 May 2018, at 20:04, Robert Haas wrote:
>
> But what happens if a transaction starts on node A at time T0 but
> first touches node B at a much later time T1, such that T1 - T0 >
> global_snapshot_defer_time?
>
Such transaction will get "global snapshot too old" error.
In principle such
On Sun, May 6, 2018 at 6:22 AM, Stas Kelvich wrote:
> Also each second GetSnapshotData writes globalxmin (as it was before
> procArray->global_snapshot_xmin was taken into account) into a circle
> buffer with a size equal to global_snapshot_defer_time value. That more
> or less the same thing as w
> On 4 May 2018, at 22:09, Robert Haas wrote:
>
> So, is the idea that we'll definitely find out about any remote
> transactions within 30 seconds, and then after we know about remote
> transactions, we'll hold back OldestXmin some other way?
Yes, kind of. There is a procArray->global_snapshot
On Tue, May 1, 2018 at 5:02 PM, Stas Kelvich wrote:
> Yes, that totally possible. On both systems you need:
Cool.
> * set track_global_snapshots='on' -- this will start writing each
> transaction commit sequence number to SRLU.
> * set global_snapshot_defer_time to 30 seconds, for example -- t
> On 3 May 2018, at 18:28, Masahiko Sawada wrote:
>
> On Wed, May 2, 2018 at 1:27 AM, Stas Kelvich wrote:
>> 1) To achieve commit atomicity of different nodes intermediate step is
>> introduced: at first running transaction is marked as InDoubt on all nodes,
>> and only after that each nod
On Wed, May 2, 2018 at 1:27 AM, Stas Kelvich wrote:
> 1) To achieve commit atomicity of different nodes intermediate step is
>introduced: at first running transaction is marked as InDoubt on all nodes,
>and only after that each node commit it and stamps with a given GlobalCSN.
>All rea
> On 2 May 2018, at 05:58, Peter Eisentraut
> wrote:
>
> On 5/1/18 12:27, Stas Kelvich wrote:
>> Clock-SI is described in [5] and here I provide a small overview, which
>> supposedly should be enough to catch the idea. Assume that each node runs
>> Commit
>> Sequence Number (CSN) based visibi
On 5/1/18 12:27, Stas Kelvich wrote:
> Clock-SI is described in [5] and here I provide a small overview, which
> supposedly should be enough to catch the idea. Assume that each node runs
> Commit
> Sequence Number (CSN) based visibility: database tracks one counter for each
> transaction start (xi
> On 1 May 2018, at 22:43, Robert Haas wrote:
>
> I'm concerned about the provisioning aspect of this problem. Suppose
> I have two existing database systems with, perhaps, wildly different
> XID counters. On a certain date, I want to start using this system.
Yes, that totally possible. On b
On Tue, May 1, 2018 at 12:27 PM, Stas Kelvich wrote:
> Here proposed a set of patches that allow achieving proper snapshot isolation
> semantics in the case of cross-node transactions. Provided infrastructure to
> synchronize snapshots between different Postgres instances and a way to
> atomically
78 matches
Mail list logo
