On Tue, Sep 13, 2022 at 03:02:34PM +0100, Simon Riggs wrote:
> Thanks Robert. I've tried to accommodate all of your thoughts, plus Alvaro's.
>
> New v5 attached.
>
> Happy to receive further comments.
I liked this patch very much. It gives details on a lot of the
internals we expose to users. Some of my changes were:
* tightening the wording
* restructuring the flow
* splitting out user-visible details (prepared transactions) from
internals, e.g., xid, vxid, subtransactions
* adding references from places in our docs to these new sections
I plan to apply this and backpatch it to all supported versions since
these details apply to all versions. These docs should enable our users
to much better understand and monitor Postgres.
Updated patch attached.
--
Bruce Momjian <br...@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Indecision is a decision. Inaction is an action. Mark Batterson
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 66312b53b8..024a3c5101 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7224,12 +7224,14 @@ local0.* /var/log/postgresql
</row>
<row>
<entry><literal>%v</literal></entry>
- <entry>Virtual transaction ID (backendID/localXID)</entry>
+ <entry>Virtual transaction ID (backendID/localXID); see
+ <xref linkend="transaction-id"/></entry>
<entry>no</entry>
</row>
<row>
<entry><literal>%x</literal></entry>
- <entry>Transaction ID (0 if none is assigned)</entry>
+ <entry>Transaction ID (0 if none is assigned); see
+ <xref linkend="transaction-id"/></entry>
<entry>no</entry>
</row>
<row>
diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml
index b030b36002..fdffba4442 100644
--- a/doc/src/sgml/datatype.sgml
+++ b/doc/src/sgml/datatype.sgml
@@ -4992,7 +4992,8 @@ WHERE ...
<structfield>xmin</structfield> and <structfield>xmax</structfield>. Transaction identifiers are 32-bit quantities.
In some contexts, a 64-bit variant <type>xid8</type> is used. Unlike
<type>xid</type> values, <type>xid8</type> values increase strictly
- monotonically and cannot be reused in the lifetime of a database cluster.
+ monotonically and cannot be reused in the lifetime of a database
+ cluster. See <xref linkend="transaction-id"/> for more details.
</para>
<para>
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index de450cd661..0d6be9a2fa 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -104,6 +104,7 @@
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
+<!ENTITY transaction SYSTEM "xact.sgml">
<!ENTITY tablesample-method SYSTEM "tablesample-method.sgml">
<!ENTITY generic-wal SYSTEM "generic-wal.sgml">
<!ENTITY custom-rmgr SYSTEM "custom-rmgr.sgml">
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index b5a2f94c4e..1e0d587932 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24676,7 +24676,10 @@ SELECT collation for ('foo' COLLATE "de_DE");
<para>
Returns the current transaction's ID. It will assign a new one if the
current transaction does not have one already (because it has not
- performed any database updates).
+ performed any database updates); see <xref
+ linkend="transaction-id"/> for details. If executed in a
+ subtransaction this will return the top-level xid; see <xref
+ linkend="subxacts"/> for details.
</para></entry>
</row>
@@ -24693,6 +24696,7 @@ SELECT collation for ('foo' COLLATE "de_DE");
ID is assigned yet. (It's best to use this variant if the transaction
might otherwise be read-only, to avoid unnecessary consumption of an
XID.)
+ If executed in a subtransaction this will return the top-level xid.
</para></entry>
</row>
@@ -24736,6 +24740,8 @@ SELECT collation for ('foo' COLLATE "de_DE");
<para>
Returns a current <firstterm>snapshot</firstterm>, a data structure
showing which transaction IDs are now in-progress.
+ Only top-level xids are included in the snapshot; subxids are not
+ shown; see <xref linkend="subxacts"/> for details.
</para></entry>
</row>
@@ -24790,7 +24796,8 @@ SELECT collation for ('foo' COLLATE "de_DE");
Is the given transaction ID <firstterm>visible</firstterm> according
to this snapshot (that is, was it completed before the snapshot was
taken)? Note that this function will not give the correct answer for
- a subtransaction ID.
+ a subtransaction ID (subxid); see <xref linkend="subxacts"/> for
+ details.
</para></entry>
</row>
</tbody>
@@ -24802,8 +24809,9 @@ SELECT collation for ('foo' COLLATE "de_DE");
wraps around every 4 billion transactions. However,
the functions shown in <xref linkend="functions-pg-snapshot"/> use a
64-bit type <type>xid8</type> that does not wrap around during the life
- of an installation, and can be converted to <type>xid</type> by casting if
- required. The data type <type>pg_snapshot</type> stores information about
+ of an installation and can be converted to <type>xid</type> by casting if
+ required; see <xref linkend="transaction-id"/> for details.
+ The data type <type>pg_snapshot</type> stores information about
transaction ID visibility at a particular moment in time. Its components
are described in <xref linkend="functions-pg-snapshot-parts"/>.
<type>pg_snapshot</type>'s textual representation is
@@ -24849,7 +24857,7 @@ SELECT collation for ('foo' COLLATE "de_DE");
xmax</literal> and not in this list was already completed at the time
of the snapshot, and thus is either visible or dead according to its
commit status. This list does not include the transaction IDs of
- subtransactions.
+ subtransactions (subxids).
</entry>
</row>
</tbody>
diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml
index d6d0a3a814..fe138a47bb 100644
--- a/doc/src/sgml/glossary.sgml
+++ b/doc/src/sgml/glossary.sgml
@@ -1710,7 +1710,8 @@
<literal>3</literal> (values under that are reserved) and the
epoch value is incremented by one.
In some contexts, the epoch and xid values are
- considered together as a single 64-bit value.
+ considered together as a single 64-bit value; see <xref
+ linkend="transaction-id"/> for more details.
</para>
<para>
For more information, see
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 342b20ebeb..3a9cf68578 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -918,7 +918,8 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<structfield>backend_xid</structfield> <type>xid</type>
</para>
<para>
- Top-level transaction identifier of this backend, if any.
+ Top-level transaction identifier of this backend, if any; see
+ <xref linkend="transaction-id"/>.
</para></entry>
</row>
@@ -1890,7 +1891,8 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</row>
<row>
<entry><literal>virtualxid</literal></entry>
- <entry>Waiting to acquire a virtual transaction ID lock.</entry>
+ <entry>Waiting to acquire a virtual transaction ID lock; see
+ <xref linkend="transaction-id"/>.</entry>
</row>
</tbody>
</tgroup>
diff --git a/doc/src/sgml/pgrowlocks.sgml b/doc/src/sgml/pgrowlocks.sgml
index 2914bf6e6d..ad15cda668 100644
--- a/doc/src/sgml/pgrowlocks.sgml
+++ b/doc/src/sgml/pgrowlocks.sgml
@@ -57,7 +57,8 @@ pgrowlocks(text) returns setof record
<row>
<entry><structfield>locker</structfield></entry>
<entry><type>xid</type></entry>
- <entry>Transaction ID of locker, or multixact ID if multitransaction</entry>
+ <entry>Transaction ID of locker, or multixact ID if
+ multitransaction; see <xref linkend="transaction-id"/></entry>
</row>
<row>
<entry><structfield>multi</structfield></entry>
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 73439c049e..2e271862fc 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -271,6 +271,7 @@ break is not needed in a wider output rendering.
&brin;
&hash;
&storage;
+ &transaction;
&bki;
&planstats;
&backup-manifest;
diff --git a/doc/src/sgml/ref/commit.sgml b/doc/src/sgml/ref/commit.sgml
index 5f244cdd3c..53d830998c 100644
--- a/doc/src/sgml/ref/commit.sgml
+++ b/doc/src/sgml/ref/commit.sgml
@@ -62,6 +62,9 @@ COMMIT [ WORK | TRANSACTION ] [ AND [ NO ] CHAIN ]
linkend="sql-set-transaction"/>) as the just finished one. Otherwise,
no new transaction is started.
</para>
+ <para>
+ The SQL Standard describes this as a chained transaction.
+ </para>
</listitem>
</varlistentry>
</variablelist>
diff --git a/doc/src/sgml/ref/rollback.sgml b/doc/src/sgml/ref/rollback.sgml
index 142f71e774..02b118fc04 100644
--- a/doc/src/sgml/ref/rollback.sgml
+++ b/doc/src/sgml/ref/rollback.sgml
@@ -56,11 +56,14 @@ ROLLBACK [ WORK | TRANSACTION ] [ AND [ NO ] CHAIN ]
<term><literal>AND CHAIN</literal></term>
<listitem>
<para>
- If <literal>AND CHAIN</literal> is specified, a new transaction is
+ If <literal>AND CHAIN</literal> is specified, a new unaborted transaction is
immediately started with the same transaction characteristics (see <xref
linkend="sql-set-transaction"/>) as the just finished one. Otherwise,
no new transaction is started.
</para>
+ <para>
+ The SQL Standard describes this as a chained transaction.
+ </para>
</listitem>
</varlistentry>
</variablelist>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 1ca7c3f9bf..6a37cb8b30 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -1395,7 +1395,8 @@
</para>
<para>
Virtual ID of the transaction targeted by the lock,
- or null if the target is not a virtual transaction ID
+ or null if the target is not a virtual transaction ID; see
+ <xref linkend="transactions"/>
</para></entry>
</row>
@@ -1404,8 +1405,8 @@
<structfield>transactionid</structfield> <type>xid</type>
</para>
<para>
- ID of the transaction targeted by the lock,
- or null if the target is not a transaction ID
+ ID of the transaction targeted by the lock, or null if the target
+ is not a transaction ID; <xref linkend="transactions"/>
</para></entry>
</row>
diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgml
index 6a38b53744..4ac5f49b9e 100644
--- a/doc/src/sgml/wal.sgml
+++ b/doc/src/sgml/wal.sgml
@@ -4,8 +4,9 @@
<title>Reliability and the Write-Ahead Log</title>
<para>
- This chapter explains how the Write-Ahead Log is used to obtain
- efficient, reliable operation.
+ This chapter explains how the control the reliability of
+ <productname>PostgreSQL</productname>, including details about the
+ Write-Ahead Log.
</para>
<sect1 id="wal-reliability">
@@ -909,4 +910,36 @@
seem to be a problem in practice.
</para>
</sect1>
+
+ <sect1 id="two-phase">
+
+ <title>Two-Phase Transactions</title>
+
+ <para>
+ <productname>PostgreSQL</productname> supports a two-phase commit
+ (2PC) protocol that allows multiple distributed systems to work
+ together in a transactional manner. The commands <command>PREPARE
+ TRANSACTION</command>, <command>COMMIT PREPARED</command> and
+ <command>ROLLBACK PREPARED</command>. Two-phase transactions
+ are intended for use by external transaction management systems.
+ <productname>PostgreSQL</productname> follows the features and model
+ proposed by the X/Open XA standard, but does not implement some less
+ often used aspects.
+ </para>
+
+ <para>
+ When the user executes <command>PREPARE TRANSACTION</command>, the
+ only possible next commands are <command>COMMIT PREPARED</command>
+ or <command>ROLLBACK PREPARED</command>. In general, this prepared
+ state is intended to be of very short duration, but external
+ availability issues might mean transactions stay in this state
+ for an extended interval. Short-lived prepared
+ transactions are stored only in shared memory and WAL.
+ Transactions that span checkpoints are recorded in the
+ <filename>pg_twophase</filename> directory. Currently-prepared
+ transactions can be inspected using <link
+ linkend="view-pg-prepared-xacts"><structname>pg_prepared_xacts</structname></link>.
+ </para>
+ </sect1>
+
</chapter>
diff --git a/doc/src/sgml/xact.sgml b/doc/src/sgml/xact.sgml
new file mode 100644
index 0000000000..44d34233c9
--- /dev/null
+++ b/doc/src/sgml/xact.sgml
@@ -0,0 +1,171 @@
+<!-- doc/src/sgml/mvcc.sgml -->
+
+ <chapter id="transactions">
+
+ <title>Transaction Management</title>
+
+ <para>
+ This chapter provides an overview of the internals of
+ <productname>PostgreSQL</productname>'s transaction management system.
+ The word transaction is often abbreviated as "xact".
+ </para>
+
+ <sect1 id="transaction-id">
+
+ <title>Transactions and Identifiers</title>
+
+ <para>
+ Transactions can be created explicitly using <command>BEGIN</command>
+ and <command>COMMIT</command>, which creates a transaction block.
+ An SQL statement outside of a transaction block automatically uses
+ a single-statement transaction.
+ </para>
+
+ <para>
+ Every transaction is identified by a unique
+ <literal>VirtualTransactionId</literal> (also called
+ <literal>virtualXID</literal> or <literal>vxid</literal>), which
+ is comprised of a backend ID (or <literal>backendID</literal>)
+ and a sequentially-assigned number local to each backend, known as
+ <literal>localXID</literal>. For example, the virtual transaction
+ ID <literal>4/12532</literal> has a <literal>backendID</literal>
+ of <literal>4</literal> and a <literal>localXID</literal> of
+ <literal>12532</literal>.
+ </para>
+
+ <para>
+ Once a transaction writes to the database, it is assigned a
+ non-virtual <literal>TransactionId</literal> (or <type>xid</type>),
+ e.g., <literal>278394</literal>. Xids are assigned sequentially
+ using a global counter used by all databases within the
+ <productname>PostgreSQL</productname> cluster. This property is used by
+ the transaction system to order transactions by their first database
+ write, i.e., lower-numbered xids started writing before higher-numbered
+ xids. Of course, transactions might start in a different order.
+ </para>
+
+ <para>
+ The internal transaction ID type <type>xid</type> is 32-bits wide
+ and <link linkend="vacuum-for-wraparound">wraps around</link> every
+ 4 billion transactions. A 32-bit epoch is incremented during each
+ wrap around. There is also a 64-bit type <type>xid8</type> which
+ includes this epoch and therefore does not wrap around during the
+ life of an installation and can be converted to xid by casting.
+ The functions in <xref linkend="functions-pg-snapshot"/>
+ return <type>xid8</type> values. Xids are used as the
+ basis for <productname>PostgreSQL</productname>'s <link
+ linkend="mvcc">MVCC</link> concurrency mechanism, <link
+ linkend="hot-standby">Hot Standby</link>, and Read Replica servers.
+ </para>
+
+ <para>
+ When a top-level transaction with a (non-virtual) xid commits,
+ it is marked as committed in the <filename>pg_xact</filename>
+ directory. Additional information is recorded in the
+ <filename>pg_commit_ts</filename> directory if <xref
+ linkend="guc-track-commit-timestamp"/> is enabled.
+ </para>
+
+ <para>
+ In addition to <literal>vxid</literal> and <type>xid</type>,
+ when a transaction is prepared it is also identified by a Global
+ Transaction Identifier (<acronym>GID</acronym>). GIDs
+ are string literals up to 200 bytes long, which must be
+ unique amongst other currently prepared transactions.
+ The mapping of GID to xid is shown in <link
+ linkend="view-pg-prepared-xacts"><structname>pg_prepared_xacts</structname></link>.
+ </para>
+ </sect1>
+
+ <sect1 id="xact-locking">
+
+ <title>Transactions and Locking</title>
+
+ <para>
+ Currently-executing transactions are shown in <link
+ linkend="view-pg-locks"><structname>pg_locks</structname></link>
+ in columns <structfield>virtualxid</structfield> and
+ <structfield>transactionid</structfield>. Read-only
+ transactions will have <structfield>virtualxid</structfield>s but
+ NULL <structfield>transactionid</structfield>s, while read-write
+ transactions will have both as non-NULL.
+ </para>
+
+ <para>
+ Lock waits on table-level locks are shown waiting for
+ <structfield>virtualxid</structfield>, while lock waits on row-level
+ locks are shown waiting for <structfield>transactionid</structfield>.
+ Row-level read and write locks are recorded directly in locked
+ rows and can be inspected using the <xref linkend="pgrowlocks"/>
+ extension. Row-level read locks might also require the assignment
+ of multixact IDs (<literal>mxid</literal>). Mxids are recorded in
+ the <filename>pg_multixact</filename> directory.
+ </para>
+ </sect1>
+
+ <sect1 id="subxacts">
+
+ <title>Subtransactions</title>
+
+ <para>
+ Subtransactions are started inside transactions, allowing large
+ transactions to be broken into smaller units. Subtransactions can
+ commit or abort without affecting their parent transactions, allowing
+ parent transactions to continue. This allows errors to be handled
+ more easily, which is a common application development pattern.
+ The word subtransaction is often abbreviated as
+ <literal>subxact</literal>.
+ </para>
+
+ <para>
+ Subtransactions can be started explicitly by using
+ <command>SAVEPOINT</command>, but can also be started in other
+ ways, such as PL/pgSQL's <command>EXCEPTION</command> clause.
+ PL/Python and PL/TCL also support explicit subtransactions.
+ Subtransactions can also be started from other subtransactions.
+ This causes the main transaction and its child subtransactions form
+ a hierarchy or tree, which is why we refer to the main transaction
+ as the top-level transaction.
+ </para>
+
+ <para>
+ If a subtransaction is assigned a non-virtual transaction ID,
+ its transaction ID is referred to as a <literal>subxid</literal>.
+ Read-only subtransactions are not assigned subxids, but once they
+ attempt to write, they will be assigned one. This also causes all of
+ a subxid's parents, up to and including the top-level transaction,
+ to be assigned non-virtual transaction ids. We ensure that a parent
+ xid is always lower than any of its child subxids.
+ </para>
+
+ <para>
+ The parent xid of each subxid is recorded in the
+ <filename>pg_subtrans</filename> directory. No entry is made for
+ top-level xids since they do not have a parent, nor is an entry made
+ for read-only subtransactions.
+ </para>
+
+ <para>
+ When a subtransaction commits, all of its committed child
+ subtransactions with subxids will also be considered committed in that
+ session. When a subtransaction aborts, all of its child subtransactions
+ will also be considered aborted.
+ </para>
+
+ <para>
+ When a top-level transaction with an xid commits, all of its committed
+ child subtransactions are also marked as globally committed in the
+ <filename>pg_xact</filename> directory. If the top-level transaction
+ aborts, all its subtransactions are also aborted.
+ </para>
+
+ <para>
+ The more subtransactions each transaction keeps open (not rolled back
+ or released), the greater the transaction management overhead. Up to
+ 64 open subxids are cached in shared memory for each backend; after
+ that point, the overhead increases significantly since we must look
+ up subxid entries in <filename>pg_subtrans</filename>.
+ </para>
+ </sect1>
+ </chapter>
+
