On 8 Sep. 2016 1:38 pm, "Andres Freund" <and...@anarazel.de> wrote: > > I kind of dislike this approach for a different reason than already > mentioned in this thread: Namely it's not re-usable for implementing > streaming logical replication of large transaction, i.e. allow to decode > & apply un-committed changes. The issue there is that one really needs > to have multiple transactions active in the same connection, which this > approach does not allow.
I've been thinking about this recently with an eye to handling the majority of transactions on a typical system that neither perform DDL nor are concurrent with it. The following might be fairly nonsensical if I've misunderstood the problem as I've not had a chance to more than glance at it, but: I wonder if some kind of speculative concurrent decoding+output is possible without being able to have multiple xacts on a backend. We could use a shared xact and snapshot for all concurrent xacts. If any of them do anything that requires invalidations etc we dump the speculatively processed ones and fall back to reorder buffering and serial output at commit time until we pass the xact that caused an invalidation and don't have more pending. Add a callback to the output plugin to tell it that speculative decoding of an xact has been aborted. Or maybe even just put that speculstive decoding on hold and pick up where we left off partway through the reorder buffer when we get around to processing it serially. That way we don't have to discard work already done. More usefully we could avoid having to enqueue stuff into the reorder buffer just in case we have to switch to fallback serial decoding. Failing that: Since logical decoding only needs read only xacts that roll back, it only needs multiple backend private virtualxacts. They don't need SERIALIZABLE/SSI which I think (?) means other backends don't need to be able to wait on them. That seems simpler than what autonomous xacts would need since there we need them exposed in PgXact, waitable-on for txid locks, etc. Right? In the same way that historical snapshots are cut-down maybe logical decoding xacts can be too. I suspect that waiting for full in-process multiple xact support to do interleaved decoding/replay will mean a long wait indeed.