pgsql: Get rid of artificial restriction on hash table sizes on Windows
Get rid of artificial restriction on hash table sizes on Windows. The point of introducing the hash_mem_multiplier GUC was to let users reproduce the old behavior of hash aggregation, i.e. that it could use more than work_mem at need. However, the implementation failed to get the job done on Win64, where work_mem is clamped to 2GB to protect various places that calculate memory sizes using "long int". As written, the same clamp was applied to hash_mem. This resulted in severe performance regressions for queries requiring a bit more than 2GB for hash aggregation, as they now spill to disk and there's no way to stop that. Getting rid of the work_mem restriction seems like a good idea, but it's a big job and could not conceivably be back-patched. However, there's only a fairly small number of places that are concerned with the hash_mem value, and it turns out to be possible to remove the restriction there without too much code churn or any ABI breaks. So, let's do that for now to fix the regression, and leave the larger task for another day. This patch does introduce a bit more infrastructure that should help with the larger task, namely pg_bitutils.h support for working with size_t values. Per gripe from Laurent Hasson. Back-patch to v13 where the behavior change came in. Discussion: https://postgr.es/m/[email protected] Discussion: https://postgr.es/m/mn2pr15mb25601e80a9b6d1ba6f592b1985...@mn2pr15mb2560.namprd15.prod.outlook.com Branch -- REL_13_STABLE Details --- https://git.postgresql.org/pg/commitdiff/2b8f3f5a7c0ede24903782fcffe2553fec01bfbe Modified Files -- src/backend/executor/execGrouping.c| 6 +- src/backend/executor/nodeAgg.c | 42 +- src/backend/executor/nodeHash.c| 144 +++-- src/backend/optimizer/path/costsize.c | 5 +- src/backend/optimizer/plan/planner.c | 6 +- src/backend/optimizer/plan/subselect.c | 3 +- src/backend/optimizer/prep/prepunion.c | 8 +- src/backend/optimizer/util/pathnode.c | 3 +- src/include/miscadmin.h| 1 + src/include/port/pg_bitutils.h | 50 +++- 10 files changed, 169 insertions(+), 99 deletions(-)
pgsql: Get rid of artificial restriction on hash table sizes on Windows
Get rid of artificial restriction on hash table sizes on Windows. The point of introducing the hash_mem_multiplier GUC was to let users reproduce the old behavior of hash aggregation, i.e. that it could use more than work_mem at need. However, the implementation failed to get the job done on Win64, where work_mem is clamped to 2GB to protect various places that calculate memory sizes using "long int". As written, the same clamp was applied to hash_mem. This resulted in severe performance regressions for queries requiring a bit more than 2GB for hash aggregation, as they now spill to disk and there's no way to stop that. Getting rid of the work_mem restriction seems like a good idea, but it's a big job and could not conceivably be back-patched. However, there's only a fairly small number of places that are concerned with the hash_mem value, and it turns out to be possible to remove the restriction there without too much code churn or any ABI breaks. So, let's do that for now to fix the regression, and leave the larger task for another day. This patch does introduce a bit more infrastructure that should help with the larger task, namely pg_bitutils.h support for working with size_t values. Per gripe from Laurent Hasson. Back-patch to v13 where the behavior change came in. Discussion: https://postgr.es/m/[email protected] Discussion: https://postgr.es/m/mn2pr15mb25601e80a9b6d1ba6f592b1985...@mn2pr15mb2560.namprd15.prod.outlook.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/28d936031a86d94806c6604480ff3f3f169b371c Modified Files -- src/backend/executor/execGrouping.c| 6 +- src/backend/executor/nodeAgg.c | 42 ++- src/backend/executor/nodeHash.c| 126 + src/backend/executor/nodeMemoize.c | 2 +- src/backend/optimizer/path/costsize.c | 7 +- src/backend/optimizer/plan/planner.c | 6 +- src/backend/optimizer/plan/subselect.c | 6 +- src/backend/optimizer/prep/prepunion.c | 8 +-- src/backend/optimizer/util/pathnode.c | 3 +- src/backend/storage/ipc/shm_mq.c | 6 +- src/include/miscadmin.h| 2 +- src/include/port/pg_bitutils.h | 50 - 12 files changed, 153 insertions(+), 111 deletions(-)
pgsql: Get rid of artificial restriction on hash table sizes on Windows
Get rid of artificial restriction on hash table sizes on Windows. The point of introducing the hash_mem_multiplier GUC was to let users reproduce the old behavior of hash aggregation, i.e. that it could use more than work_mem at need. However, the implementation failed to get the job done on Win64, where work_mem is clamped to 2GB to protect various places that calculate memory sizes using "long int". As written, the same clamp was applied to hash_mem. This resulted in severe performance regressions for queries requiring a bit more than 2GB for hash aggregation, as they now spill to disk and there's no way to stop that. Getting rid of the work_mem restriction seems like a good idea, but it's a big job and could not conceivably be back-patched. However, there's only a fairly small number of places that are concerned with the hash_mem value, and it turns out to be possible to remove the restriction there without too much code churn or any ABI breaks. So, let's do that for now to fix the regression, and leave the larger task for another day. This patch does introduce a bit more infrastructure that should help with the larger task, namely pg_bitutils.h support for working with size_t values. Per gripe from Laurent Hasson. Back-patch to v13 where the behavior change came in. Discussion: https://postgr.es/m/[email protected] Discussion: https://postgr.es/m/mn2pr15mb25601e80a9b6d1ba6f592b1985...@mn2pr15mb2560.namprd15.prod.outlook.com Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/b154ee63bb659ce280d486db6bbbe77ddec105c5 Modified Files -- src/backend/executor/execGrouping.c| 6 +- src/backend/executor/nodeAgg.c | 42 +- src/backend/executor/nodeHash.c| 144 +++-- src/backend/executor/nodeMemoize.c | 2 +- src/backend/optimizer/path/costsize.c | 7 +- src/backend/optimizer/plan/planner.c | 6 +- src/backend/optimizer/plan/subselect.c | 6 +- src/backend/optimizer/prep/prepunion.c | 8 +- src/backend/optimizer/util/pathnode.c | 3 +- src/include/miscadmin.h| 1 + src/include/port/pg_bitutils.h | 50 +++- 11 files changed, 172 insertions(+), 103 deletions(-)
pgsql: Fix a couple of memory leaks in src/bin/pg_basebackup/
Fix a couple of memory leaks in src/bin/pg_basebackup/ These have been introduced by 7fbe0c8, and could happen for pg_basebackup and pg_receivewal. Per report from Coverity for the ones in walmethods.c, I have spotted the ones in receivelog.c after more review. Backpatch-through: 10 Branch -- REL_12_STABLE Details --- https://git.postgresql.org/pg/commitdiff/c4ef3b81b85434b3d2eac8d0cca3a0078898266a Modified Files -- src/bin/pg_basebackup/receivelog.c | 7 +++ src/bin/pg_basebackup/walmethods.c | 7 +++ src/bin/pg_basebackup/walmethods.h | 4 ++-- 3 files changed, 16 insertions(+), 2 deletions(-)
pgsql: Fix a couple of memory leaks in src/bin/pg_basebackup/
Fix a couple of memory leaks in src/bin/pg_basebackup/ These have been introduced by 7fbe0c8, and could happen for pg_basebackup and pg_receivewal. Per report from Coverity for the ones in walmethods.c, I have spotted the ones in receivelog.c after more review. Backpatch-through: 10 Branch -- REL_11_STABLE Details --- https://git.postgresql.org/pg/commitdiff/9c6fa3403ecdb856fd61c26bb11c5e6564973323 Modified Files -- src/bin/pg_basebackup/receivelog.c | 7 +++ src/bin/pg_basebackup/walmethods.c | 7 +++ src/bin/pg_basebackup/walmethods.h | 4 ++-- 3 files changed, 16 insertions(+), 2 deletions(-)
pgsql: Fix a couple of memory leaks in src/bin/pg_basebackup/
Fix a couple of memory leaks in src/bin/pg_basebackup/ These have been introduced by 7fbe0c8, and could happen for pg_basebackup and pg_receivewal. Per report from Coverity for the ones in walmethods.c, I have spotted the ones in receivelog.c after more review. Backpatch-through: 10 Branch -- REL_13_STABLE Details --- https://git.postgresql.org/pg/commitdiff/2c7395aad7e849235faac4ead5f69700cfd3c21b Modified Files -- src/bin/pg_basebackup/receivelog.c | 6 ++ src/bin/pg_basebackup/walmethods.c | 7 +++ src/bin/pg_basebackup/walmethods.h | 4 ++-- 3 files changed, 15 insertions(+), 2 deletions(-)
pgsql: Fix a couple of memory leaks in src/bin/pg_basebackup/
Fix a couple of memory leaks in src/bin/pg_basebackup/ These have been introduced by 7fbe0c8, and could happen for pg_basebackup and pg_receivewal. Per report from Coverity for the ones in walmethods.c, I have spotted the ones in receivelog.c after more review. Backpatch-through: 10 Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/4ef64c425dbcda151c9f163aadff982343808e09 Modified Files -- src/bin/pg_basebackup/receivelog.c | 6 ++ src/bin/pg_basebackup/walmethods.c | 7 +++ src/bin/pg_basebackup/walmethods.h | 4 ++-- 3 files changed, 15 insertions(+), 2 deletions(-)
pgsql: Fix a couple of memory leaks in src/bin/pg_basebackup/
Fix a couple of memory leaks in src/bin/pg_basebackup/ These have been introduced by 7fbe0c8, and could happen for pg_basebackup and pg_receivewal. Per report from Coverity for the ones in walmethods.c, I have spotted the ones in receivelog.c after more review. Backpatch-through: 10 Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/b0d28671964d61416248789e3cee4508ac6281b4 Modified Files -- src/bin/pg_basebackup/receivelog.c | 6 ++ src/bin/pg_basebackup/walmethods.c | 7 +++ src/bin/pg_basebackup/walmethods.h | 4 ++-- 3 files changed, 15 insertions(+), 2 deletions(-)
pgsql: Fix a couple of memory leaks in src/bin/pg_basebackup/
Fix a couple of memory leaks in src/bin/pg_basebackup/ These have been introduced by 7fbe0c8, and could happen for pg_basebackup and pg_receivewal. Per report from Coverity for the ones in walmethods.c, I have spotted the ones in receivelog.c after more review. Backpatch-through: 10 Branch -- REL_10_STABLE Details --- https://git.postgresql.org/pg/commitdiff/4372f0685de164e769e895dbada18cad792192b2 Modified Files -- src/bin/pg_basebackup/receivelog.c | 7 +++ src/bin/pg_basebackup/walmethods.c | 7 +++ src/bin/pg_basebackup/walmethods.h | 4 ++-- 3 files changed, 16 insertions(+), 2 deletions(-)
pgsql: Fix incorrect comment for get_agg_clause_costs
Fix incorrect comment for get_agg_clause_costs Adjust the header comment in get_agg_clause_costs so that it matches what the function currently does. No recursive searching has been done ever since 0a2bc5d61. It also does not determine the aggtranstype like the comment claimed. That's all done in preprocess_aggref(). preprocess_aggref also now determines the numOrderedAggs, so remove the mention that get_agg_clause_costs also calculates "counts". Normally, since this is just an adjustment of a comment it might not be worth back-patching, but since this code is new to PG14 and that version is still in beta, then it seems worth having the comments match. Discussion: https://postgr.es/m/caaphdvrrgrtjfpelrjx0cndtz9b7jy2xyw3z2bkifawlsaj...@mail.gmail.com Backpatch-though: 14 Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/2b58f894e56a1944e824e19c92337d6bf24d9c41 Modified Files -- src/backend/optimizer/prep/prepagg.c | 28 1 file changed, 12 insertions(+), 16 deletions(-)
pgsql: Fix incorrect comment for get_agg_clause_costs
Fix incorrect comment for get_agg_clause_costs Adjust the header comment in get_agg_clause_costs so that it matches what the function currently does. No recursive searching has been done ever since 0a2bc5d61. It also does not determine the aggtranstype like the comment claimed. That's all done in preprocess_aggref(). preprocess_aggref also now determines the numOrderedAggs, so remove the mention that get_agg_clause_costs also calculates "counts". Normally, since this is just an adjustment of a comment it might not be worth back-patching, but since this code is new to PG14 and that version is still in beta, then it seems worth having the comments match. Discussion: https://postgr.es/m/caaphdvrrgrtjfpelrjx0cndtz9b7jy2xyw3z2bkifawlsaj...@mail.gmail.com Backpatch-though: 14 Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/dece64a941457686184654968ac1ded58b7d4ee8 Modified Files -- src/backend/optimizer/prep/prepagg.c | 28 1 file changed, 12 insertions(+), 16 deletions(-)
pgsql: Harden pg_stat_statements tests against CLOBBER_CACHE_ALWAYS.
Harden pg_stat_statements tests against CLOBBER_CACHE_ALWAYS. Turns out the buildfarm hasn't been testing this, which will soon change. Julien Rouhaud, per report from me Discussion: https://postgr.es/m/[email protected] Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/0806d08d4626f0a71a4e83cb399ee965a8aac99b Modified Files -- .../expected/pg_stat_statements.out| 30 ++ .../pg_stat_statements/sql/pg_stat_statements.sql | 7 - 2 files changed, 26 insertions(+), 11 deletions(-)
pgsql: Harden pg_stat_statements tests against CLOBBER_CACHE_ALWAYS.
Harden pg_stat_statements tests against CLOBBER_CACHE_ALWAYS. Turns out the buildfarm hasn't been testing this, which will soon change. Julien Rouhaud, per report from me Discussion: https://postgr.es/m/[email protected] Branch -- REL_13_STABLE Details --- https://git.postgresql.org/pg/commitdiff/882d15144bbb21856676ff046faee1e9a580e3e4 Modified Files -- .../expected/pg_stat_statements.out| 30 ++ .../pg_stat_statements/sql/pg_stat_statements.sql | 7 - 2 files changed, 26 insertions(+), 11 deletions(-)
pgsql: Harden pg_stat_statements tests against CLOBBER_CACHE_ALWAYS.
Harden pg_stat_statements tests against CLOBBER_CACHE_ALWAYS. Turns out the buildfarm hasn't been testing this, which will soon change. Julien Rouhaud, per report from me Discussion: https://postgr.es/m/[email protected] Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/e6d9544681b3f5c66a16c72bfd8dca450ec7536a Modified Files -- .../expected/pg_stat_statements.out| 30 ++ .../pg_stat_statements/sql/pg_stat_statements.sql | 7 - 2 files changed, 26 insertions(+), 11 deletions(-)
