pgsql: Fix replica identity check for MERGE.
Fix replica identity check for MERGE. When executing a MERGE, check that the target relation supports all actions mentioned in the MERGE command. Specifically, check that it has a REPLICA IDENTITY if it publishes updates or deletes and the MERGE command contains update or delete actions. Failing to do this can silently break replication. Author: Zhijie Hou Reviewed-by: Ashutosh Bapat Reviewed-by: Dean Rasheed Tested-by: Chao Li Discussion: https://postgr.es/m/os3pr01mb57180c87e43a679a730482df94...@os3pr01mb5718.jpnprd01.prod.outlook.com Backpatch-through: 15 Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/5481cc332b0fc8ca242966813a6675a514c6917b Modified Files -- src/backend/executor/execMain.c | 27 --- src/backend/executor/execPartition.c | 6 -- src/backend/executor/nodeModifyTable.c| 6 +- src/include/executor/executor.h | 3 ++- src/test/regress/expected/publication.out | 28 src/test/regress/sql/publication.sql | 31 +++ 6 files changed, 94 insertions(+), 7 deletions(-)
pgsql: Fix replica identity check for INSERT ON CONFLICT DO UPDATE.
Fix replica identity check for INSERT ON CONFLICT DO UPDATE. If an INSERT has an ON CONFLICT DO UPDATE clause, the executor must check that the target relation supports UPDATE as well as INSERT. In particular, it must check that the target relation has a REPLICA IDENTITY if it publishes updates. Formerly, it was not doing this check, which could lead to silently breaking replication. Fix by adding such a check to CheckValidResultRel(), which requires adding a new onConflictAction argument. In back-branches, preserve ABI compatibility by introducing a wrapper function with the original signature. Author: Zhijie Hou Reviewed-by: Ashutosh Bapat Reviewed-by: Dean Rasheed Tested-by: Chao Li Discussion: https://postgr.es/m/os3pr01mb57180c87e43a679a730482df94...@os3pr01mb5718.jpnprd01.prod.outlook.com Backpatch-through: 13 Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/0c4d5a45dbd7d11c9ddff4f7cf4dae3940b083be Modified Files -- src/backend/executor/execMain.c | 23 ++- src/backend/executor/execPartition.c | 8 ++-- src/backend/executor/nodeModifyTable.c| 3 ++- src/include/executor/executor.h | 2 ++ src/test/regress/expected/publication.out | 23 +++ src/test/regress/sql/publication.sql | 26 ++ 6 files changed, 81 insertions(+), 4 deletions(-)
pgsql: Fix replica identity check for INSERT ON CONFLICT DO UPDATE.
Fix replica identity check for INSERT ON CONFLICT DO UPDATE. If an INSERT has an ON CONFLICT DO UPDATE clause, the executor must check that the target relation supports UPDATE as well as INSERT. In particular, it must check that the target relation has a REPLICA IDENTITY if it publishes updates. Formerly, it was not doing this check, which could lead to silently breaking replication. Fix by adding such a check to CheckValidResultRel(), which requires adding a new onConflictAction argument. In back-branches, preserve ABI compatibility by introducing a wrapper function with the original signature. Author: Zhijie Hou Reviewed-by: Ashutosh Bapat Reviewed-by: Dean Rasheed Tested-by: Chao Li Discussion: https://postgr.es/m/os3pr01mb57180c87e43a679a730482df94...@os3pr01mb5718.jpnprd01.prod.outlook.com Backpatch-through: 13 Branch -- REL_18_STABLE Details --- https://git.postgresql.org/pg/commitdiff/344662848ac15efc89e09b7d39662d7c71b30e06 Modified Files -- src/backend/commands/copyfrom.c | 2 +- src/backend/executor/execMain.c | 12 +++- src/backend/executor/execPartition.c | 9 +++-- src/backend/executor/nodeModifyTable.c| 3 ++- src/include/executor/executor.h | 1 + src/test/regress/expected/publication.out | 23 +++ src/test/regress/sql/publication.sql | 27 +++ 7 files changed, 72 insertions(+), 5 deletions(-)
pgsql: Fix replica identity check for INSERT ON CONFLICT DO UPDATE.
Fix replica identity check for INSERT ON CONFLICT DO UPDATE. If an INSERT has an ON CONFLICT DO UPDATE clause, the executor must check that the target relation supports UPDATE as well as INSERT. In particular, it must check that the target relation has a REPLICA IDENTITY if it publishes updates. Formerly, it was not doing this check, which could lead to silently breaking replication. Fix by adding such a check to CheckValidResultRel(), which requires adding a new onConflictAction argument. In back-branches, preserve ABI compatibility by introducing a wrapper function with the original signature. Author: Zhijie Hou Reviewed-by: Ashutosh Bapat Reviewed-by: Dean Rasheed Tested-by: Chao Li Discussion: https://postgr.es/m/os3pr01mb57180c87e43a679a730482df94...@os3pr01mb5718.jpnprd01.prod.outlook.com Backpatch-through: 13 Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/451b22efd9ac4a1014af8add5405d809d92ff590 Modified Files -- src/backend/executor/execMain.c | 23 ++- src/backend/executor/execPartition.c | 8 ++-- src/backend/executor/nodeModifyTable.c| 3 ++- src/include/executor/executor.h | 2 ++ src/test/regress/expected/publication.out | 23 +++ src/test/regress/sql/publication.sql | 26 ++ 6 files changed, 81 insertions(+), 4 deletions(-)
pgsql: Move dynamically-allocated LWLock tranche names to shared memory
Move dynamically-allocated LWLock tranche names to shared memory. There are two ways for shared libraries to allocate their own LWLock tranches. One way is to call RequestNamedLWLockTranche() in a shmem_request_hook, which requires the library to be loaded via shared_preload_libraries. The other way is to call LWLockNewTrancheId(), which is not subject to the same restrictions. However, LWLockNewTrancheId() does require each backend to store the tranche's name in backend-local memory via LWLockRegisterTranche(). This API is a little cumbersome and leads to things like unhelpful pg_stat_activity.wait_event values in backends that haven't loaded the library. This commit moves these LWLock tranche names to shared memory, thus eliminating the need for each backend to call LWLockRegisterTranche(). Instead, the tranche name must be provided to LWLockNewTrancheId(), which immediately makes the name available to all backends. Since the tranche name array is append-only, lookups can ordinarily avoid locking as long as their local copy of the LWLock counter is greater than the requested tranche ID. One downside of this approach is that we now have a hard limit on both the length of tranche names (NAMEDATALEN-1 bytes) and the number of dynamically-allocated tranches (256). Besides a limit of NAMEDATALEN-1 bytes for tranche names registered via RequestNamedLWLockTranche(), no such limits previously existed. We could avoid these new limits by using dynamic shared memory, but the complexity involved didn't seem worth it. We briefly considered making the tranche limit user-configurable but ultimately decided against that, too. Since there is still a lot of time left in the v19 development cycle, it's possible we will revisit this choice. Author: Sami Imseih Reviewed-by: Bertrand Drouvot Reviewed-by: Tom Lane Reviewed-by: Rahila Syed Reviewed-by: Andres Freund Discussion: https://postgr.es/m/CAA5RZ0vvED3naph8My8Szv6DL4AxOVK3eTPS0qXsaKi%3DbVdW2A%40mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/38b602b0289fe1dbaf31d5737fba2d42a1e90371 Modified Files -- contrib/pg_prewarm/autoprewarm.c | 3 +- doc/src/sgml/xfunc.sgml| 15 +- src/backend/postmaster/launch_backend.c| 6 +- src/backend/storage/ipc/dsm_registry.c | 12 +- src/backend/storage/lmgr/lwlock.c | 216 ++--- src/include/storage/lwlock.h | 25 +-- src/test/modules/test_dsa/test_dsa.c | 6 +- .../modules/test_dsm_registry/test_dsm_registry.c | 3 +- src/test/modules/test_radixtree/test_radixtree.c | 9 +- src/test/modules/test_slru/test_slru.c | 6 +- src/test/modules/test_tidstore/test_tidstore.c | 3 +- 11 files changed, 130 insertions(+), 174 deletions(-)
pgsql: Fix corruption of pgstats shared hashtable due to OOM failures
Fix corruption of pgstats shared hashtable due to OOM failures A new pgstats entry is created as a two-step process: - The entry is looked at in the shared hashtable of pgstats, and is inserted if not found. - When not found and inserted, its fields are then initialized. This part include a DSA chunk allocation for the stats data of the new entry. As currently coded, if the DSA chunk allocation fails due to an out-of-memory failure, an ERROR is generated, leaving in the pgstats shared hashtable an inconsistent entry due to the first step, as the entry has already been inserted in the hashtable. These broken entries can then be found by other backends, crashing them. There are only two callers of pgstat_init_entry(), when loading the pgstats file at startup and when creating a new pgstats entry. This commit changes pgstat_init_entry() so as we use dsa_allocate_extended() with DSA_ALLOC_NO_OOM, making it return NULL on allocation failure instead of failing. This way, a backend failing an entry creation can take appropriate cleanup actions in the shared hashtable before throwing an error. Currently, this means removing the entry from the shared hashtable before throwing the error for the allocation failure. Out-of-memory errors unlikely happen in the wild, and we do not bother with back-patches when these are fixed, usually. However, the problem dealt with here is a degree worse as it breaks the shared memory state of pgstats, impacting other processes that may look at an inconsistent entry that a different process has failed to create. Author: Mikhail Kot Discussion: https://postgr.es/m/caai9e7jelo5_-sbenftnc2e8xhw2pkzjwftc3i2y-gmqd2b...@mail.gmail.com Backpatch-through: 15 Branch -- REL_17_STABLE Details --- https://git.postgresql.org/pg/commitdiff/3e6dfcfb0529a9d003e07a56f58d0ff3c46556bf Modified Files -- src/backend/utils/activity/pgstat.c | 10 ++ src/backend/utils/activity/pgstat_shmem.c | 28 +++- 2 files changed, 37 insertions(+), 1 deletion(-)
pgsql: Fix corruption of pgstats shared hashtable due to OOM failures
Fix corruption of pgstats shared hashtable due to OOM failures A new pgstats entry is created as a two-step process: - The entry is looked at in the shared hashtable of pgstats, and is inserted if not found. - When not found and inserted, its fields are then initialized. This part include a DSA chunk allocation for the stats data of the new entry. As currently coded, if the DSA chunk allocation fails due to an out-of-memory failure, an ERROR is generated, leaving in the pgstats shared hashtable an inconsistent entry due to the first step, as the entry has already been inserted in the hashtable. These broken entries can then be found by other backends, crashing them. There are only two callers of pgstat_init_entry(), when loading the pgstats file at startup and when creating a new pgstats entry. This commit changes pgstat_init_entry() so as we use dsa_allocate_extended() with DSA_ALLOC_NO_OOM, making it return NULL on allocation failure instead of failing. This way, a backend failing an entry creation can take appropriate cleanup actions in the shared hashtable before throwing an error. Currently, this means removing the entry from the shared hashtable before throwing the error for the allocation failure. Out-of-memory errors unlikely happen in the wild, and we do not bother with back-patches when these are fixed, usually. However, the problem dealt with here is a degree worse as it breaks the shared memory state of pgstats, impacting other processes that may look at an inconsistent entry that a different process has failed to create. Author: Mikhail Kot Discussion: https://postgr.es/m/caai9e7jelo5_-sbenftnc2e8xhw2pkzjwftc3i2y-gmqd2b...@mail.gmail.com Backpatch-through: 15 Branch -- REL_18_STABLE Details --- https://git.postgresql.org/pg/commitdiff/f256a7bba7286058bbf9d5ce160e3237dfe83b79 Modified Files -- src/backend/utils/activity/pgstat.c | 11 +++ src/backend/utils/activity/pgstat_shmem.c | 28 +++- 2 files changed, 38 insertions(+), 1 deletion(-)
pgsql: Fix corruption of pgstats shared hashtable due to OOM failures
Fix corruption of pgstats shared hashtable due to OOM failures A new pgstats entry is created as a two-step process: - The entry is looked at in the shared hashtable of pgstats, and is inserted if not found. - When not found and inserted, its fields are then initialized. This part include a DSA chunk allocation for the stats data of the new entry. As currently coded, if the DSA chunk allocation fails due to an out-of-memory failure, an ERROR is generated, leaving in the pgstats shared hashtable an inconsistent entry due to the first step, as the entry has already been inserted in the hashtable. These broken entries can then be found by other backends, crashing them. There are only two callers of pgstat_init_entry(), when loading the pgstats file at startup and when creating a new pgstats entry. This commit changes pgstat_init_entry() so as we use dsa_allocate_extended() with DSA_ALLOC_NO_OOM, making it return NULL on allocation failure instead of failing. This way, a backend failing an entry creation can take appropriate cleanup actions in the shared hashtable before throwing an error. Currently, this means removing the entry from the shared hashtable before throwing the error for the allocation failure. Out-of-memory errors unlikely happen in the wild, and we do not bother with back-patches when these are fixed, usually. However, the problem dealt with here is a degree worse as it breaks the shared memory state of pgstats, impacting other processes that may look at an inconsistent entry that a different process has failed to create. Author: Mikhail Kot Discussion: https://postgr.es/m/caai9e7jelo5_-sbenftnc2e8xhw2pkzjwftc3i2y-gmqd2b...@mail.gmail.com Backpatch-through: 15 Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/12f57681c79bd3e8283bfb223dadba0efbe0b37e Modified Files -- src/backend/utils/activity/pgstat.c | 10 ++ src/backend/utils/activity/pgstat_shmem.c | 28 +++- 2 files changed, 37 insertions(+), 1 deletion(-)
pgsql: Update parser README to include parse_jsontable.c
Update parser README to include parse_jsontable.c The README was missing parse_jsontable.c which handles JSON_TABLE. Oversight in de3600452b61. Author: Karthik S Discussion: https://postgr.es/m/CAK4gQD9gdcj+vq_FZGp=Rv-W+41v8_C7cmCUmDeu=cfrodf...@mail.gmail.com Backpatch-through: 17 Branch -- REL_18_STABLE Details --- https://git.postgresql.org/pg/commitdiff/fbb8ae2aec009491a72597d051b9f597266e1bd1 Modified Files -- src/backend/parser/README | 1 + 1 file changed, 1 insertion(+)
pgsql: Update parser README to include parse_jsontable.c
Update parser README to include parse_jsontable.c The README was missing parse_jsontable.c which handles JSON_TABLE. Oversight in de3600452b61. Author: Karthik S Discussion: https://postgr.es/m/CAK4gQD9gdcj+vq_FZGp=Rv-W+41v8_C7cmCUmDeu=cfrodf...@mail.gmail.com Backpatch-through: 17 Branch -- REL_17_STABLE Details --- https://git.postgresql.org/pg/commitdiff/1819ff4ed67780629268d47adaf3068ba167783a Modified Files -- src/backend/parser/README | 1 + 1 file changed, 1 insertion(+)
pgsql: Post-commit review fixes for 228c370868.
Post-commit review fixes for 228c370868. This commit fixes three issues: 1) When a disabled subscription is created with retain_dead_tuples set to true, the launcher is not woken up immediately, which may lead to delays in creating the conflict detection slot. Creating the conflict detection slot is essential even when the subscription is not enabled. This ensures that dead tuples are retained, which is necessary for accurately identifying the type of conflict during replication. 2) Conflict-related data was unnecessarily retained when the subscription does not have a table. 3) Conflict-relevant data could be prematurely removed before applying prepared transactions on the publisher that are in the commit critical section. This issue occurred because the backend executing COMMIT PREPARED was not accounted for during the computation of oldestXid in the commit phase on the publisher. As a result, the subscriber could advance the conflict slot's xmin without waiting for such COMMIT PREPARED transactions to complete. We fixed this issue by identifying prepared transactions that are in the commit critical section during computation of oldestXid in commit phase. Author: Zhijie Hou Reviewed-by: shveta malik Reviewed-by: Dilip Kumar Reviewed-by: Nisha Moond Reviewed-by: Amit Kapila Discussion: https://postgr.es/m/os9pr01mb16913dacb64e5721872aa5c0294...@os9pr01mb16913.jpnprd01.prod.outlook.com Discussion: https://postgr.es/m/os9pr01mb16913f67856b0da2a9097881294...@os9pr01mb16913.jpnprd01.prod.outlook.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/1f7e9ba3ac4eff13041abcc4c9c517ad835fa449 Modified Files -- src/backend/access/transam/twophase.c | 55 + src/backend/commands/subscriptioncmds.c | 12 ++- src/backend/replication/logical/tablesync.c | 26 ++ src/backend/replication/logical/worker.c| 25 ++--- src/backend/replication/walsender.c | 12 +++ src/include/access/twophase.h | 2 ++ src/include/replication/worker_internal.h | 1 + src/test/subscription/t/035_conflicts.pl| 29 +++ 8 files changed, 157 insertions(+), 5 deletions(-)
pgsql: Update parser README to include parse_jsontable.c
Update parser README to include parse_jsontable.c The README was missing parse_jsontable.c which handles JSON_TABLE. Oversight in de3600452b61. Author: Karthik S Discussion: https://postgr.es/m/CAK4gQD9gdcj+vq_FZGp=Rv-W+41v8_C7cmCUmDeu=cfrodf...@mail.gmail.com Backpatch-through: 17 Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/43eb2c541941479714c11de9cfb7c67b54f1810d Modified Files -- src/backend/parser/README | 1 + 1 file changed, 1 insertion(+)
