Hi, On 2022-11-09 14:44:42 -0500, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > A second question: Both autoconf and meson print the segment size as GB > > right > > now. Obviously that'll print out a size of 0 for a segsize < 1GB. > > > The easiest way to would be to just display the number of blocks, but that's > > not particularly nice. > > Well, it would be fine if you'd written --with-segsize-blocks, wouldn't > it? Can we make the printout format depend on which switch was used?
Not sure why I didn't think of that... Updated patch attached. I made one autoconf and one meson CI task use a small block size, but just to ensure it work on both. I'd probably leave it set on one, so we keep the coverage for cfbot? Greetings, Andres Freund
>From 7a76be232dd0587e9c84af4e4481d5a98f94bd22 Mon Sep 17 00:00:00 2001 From: Andres Freund <and...@anarazel.de> Date: Wed, 9 Nov 2022 12:01:54 -0800 Subject: [PATCH v2] Add option to specify segment size in blocks The tests don't have much coverage of segment related code, as we don't create large enough tables. To make it easier to test these paths, add a new option specifying the segment size in blocks. Set the new option to 6 blocks in one of the CI tasks. Smaller numbers currently fail one of the tests, for understandable reasons. While at it, fix some segment size related issues in the meson build. Author: Andres Freund <and...@anarazel.de> Discussion: https://postgr.es/m/20221107171355.c23fzwanfzq2p...@awork3.anarazel.de --- meson.build | 24 ++++++++++--- meson_options.txt | 3 ++ configure.ac | 36 ++++++++++++++----- doc/src/sgml/installation.sgml | 14 ++++++++ .cirrus.yml | 2 ++ configure | 63 ++++++++++++++++++++++++++++------ 6 files changed, 118 insertions(+), 24 deletions(-) diff --git a/meson.build b/meson.build index ce2f223a409..b0fcf82a9c2 100644 --- a/meson.build +++ b/meson.build @@ -418,7 +418,19 @@ meson_bin = find_program(meson_binpath, native: true) cdata.set('USE_ASSERT_CHECKING', get_option('cassert') ? 1 : false) -cdata.set('BLCKSZ', get_option('blocksize').to_int() * 1024, description: +blocksize = get_option('blocksize').to_int() * 1024 + +if get_option('segsize_blocks') != 0 + if get_option('segsize') != 1 + warning('both segsize and segsize_blocks specified, segsize_blocks wins') + endif + + segsize = get_option('segsize_blocks') +else + segsize = (get_option('segsize') * 1024 * 1024 * 1024) / blocksize +endif + +cdata.set('BLCKSZ', blocksize, description: '''Size of a disk block --- this also limits the size of a tuple. You can set it bigger if you need bigger tuples (although TOAST should reduce the need to have large tuples, since fields can be spread across multiple tuples). @@ -428,7 +440,7 @@ cdata.set('BLCKSZ', get_option('blocksize').to_int() * 1024, description: Changing BLCKSZ requires an initdb.''') cdata.set('XLOG_BLCKSZ', get_option('wal_blocksize').to_int() * 1024) -cdata.set('RELSEG_SIZE', get_option('segsize') * 131072) +cdata.set('RELSEG_SIZE', segsize) cdata.set('DEF_PGPORT', get_option('pgport')) cdata.set_quoted('DEF_PGPORT_STR', get_option('pgport').to_string()) cdata.set_quoted('PG_KRB_SRVNAM', get_option('krb_srvnam')) @@ -3053,9 +3065,11 @@ if meson.version().version_compare('>=0.57') summary( { - 'data block size': cdata.get('BLCKSZ'), - 'WAL block size': cdata.get('XLOG_BLCKSZ') / 1024, - 'segment size': cdata.get('RELSEG_SIZE') / 131072, + 'data block size': '@0@ kB'.format(cdata.get('BLCKSZ') / 1024), + 'WAL block size': '@0@ kB'.format(cdata.get('XLOG_BLCKSZ') / 1024), + 'segment size': get_option('segsize_blocks') != 0 ? + '@0@ blocks'.format(cdata.get('RELSEG_SIZE')) : + '@0@ GB'.format(get_option('segsize')), }, section: 'Data layout', ) diff --git a/meson_options.txt b/meson_options.txt index c7ea57994dc..0d2a34fd154 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -13,6 +13,9 @@ option('wal_blocksize', type : 'combo', option('segsize', type : 'integer', value : 1, description : '''Segment size, in gigabytes''') +option('segsize_blocks', type : 'integer', value: 0, + description : '''Segment size, in blocks''') + # Miscellaneous options diff --git a/configure.ac b/configure.ac index f76b7ee31fc..94542e862cf 100644 --- a/configure.ac +++ b/configure.ac @@ -285,15 +285,31 @@ AC_DEFINE_UNQUOTED([BLCKSZ], ${BLCKSZ}, [ # # Relation segment size # -AC_MSG_CHECKING([for segment size]) PGAC_ARG_REQ(with, segsize, [SEGSIZE], [set table segment size in GB [1]], [segsize=$withval], [segsize=1]) -# this expression is set up to avoid unnecessary integer overflow -# blocksize is already guaranteed to be a factor of 1024 -RELSEG_SIZE=`expr '(' 1024 / ${blocksize} ')' '*' ${segsize} '*' 1024` -test $? -eq 0 || exit 1 -AC_MSG_RESULT([${segsize}GB]) +PGAC_ARG_REQ(with, segsize-blocks, [SEGSIZE_BLOCKS], [set table segment size in blocks [0]], + [segsize_blocks=$withval], + [segsize_blocks=0]) + +# If --with-segsize-blocks is non-zero, it is used, --with-segsize +# otherwise. segsize-blocks is only really useful for developers wanting to +# test segment related code. Warn if both are used. +if test $segsize_blocks -ne 0 -a $segsize -ne 1; then + AC_MSG_WARN([both --with-segsize and --with-segsize-blocks specified, --with-segsize-blocks wins]) +fi + +AC_MSG_CHECKING([for segment size]) +if test $segsize_blocks -eq 0; then + # this expression is set up to avoid unnecessary integer overflow + # blocksize is already guaranteed to be a factor of 1024 + RELSEG_SIZE=`expr '(' 1024 / ${blocksize} ')' '*' ${segsize} '*' 1024` + test $? -eq 0 || exit 1 + AC_MSG_RESULT([${segsize}GB]) +else + RELSEG_SIZE=$segsize_blocks + AC_MSG_RESULT([${RELSEG_SIZE} blocks]) +fi AC_DEFINE_UNQUOTED([RELSEG_SIZE], ${RELSEG_SIZE}, [ RELSEG_SIZE is the maximum number of blocks allowed in one disk file. @@ -1736,9 +1752,11 @@ fi dnl Check for largefile support (must be after AC_SYS_LARGEFILE) AC_CHECK_SIZEOF([off_t]) -# If we don't have largefile support, can't handle segsize >= 2GB. -if test "$ac_cv_sizeof_off_t" -lt 8 -a "$segsize" != "1"; then - AC_MSG_ERROR([Large file support is not enabled. Segment size cannot be larger than 1GB.]) +# If we don't have largefile support, can't handle segment size >= 2GB. +if test "$ac_cv_sizeof_off_t" -lt 8; then + if expr $RELSEG_SIZE '*' $blocksize '>=' 2 '*' 1024 '*' 1024; then + AC_MSG_ERROR([Large file support is not enabled. Segment size cannot be larger than 1GB.]) + fi fi AC_CHECK_SIZEOF([bool], [], diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml index 319c7e69660..2b2f437d915 100644 --- a/doc/src/sgml/installation.sgml +++ b/doc/src/sgml/installation.sgml @@ -1668,6 +1668,20 @@ build-postgresql: </para> </listitem> </varlistentry> + + <varlistentry> + <term><option>--with-segsize-blocks=SEGSIZE_BLOCKS</option></term> + <listitem> + <para> + Specify the segment size in blocks. If both + <option>--with-segsize</option> and this option are specified, this + option wins. + + This option is only for developers, to test segment related code. + </para> + </listitem> + </varlistentry> + </variablelist> </sect3> diff --git a/.cirrus.yml b/.cirrus.yml index 9f2282471a9..07510c3c96f 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -211,6 +211,7 @@ task: ./configure \ --enable-cassert --enable-debug --enable-tap-tests \ --enable-nls \ + --with-segsize-blocks=8 \ \ ${LINUX_CONFIGURE_FEATURES} \ \ @@ -377,6 +378,7 @@ task: -Dextra_lib_dirs=${brewpath}/lib \ -Dcassert=true \ -Dssl=openssl -Duuid=e2fs -Ddtrace=auto \ + -Dsegsize_blocks=6 \ -DPG_TEST_EXTRA="$PG_TEST_EXTRA" \ build diff --git a/configure b/configure index 3966368b8d9..b1340ca3f58 100755 --- a/configure +++ b/configure @@ -844,6 +844,7 @@ enable_dtrace enable_tap_tests with_blocksize with_segsize +with_segsize_blocks with_wal_blocksize with_CC with_llvm @@ -1554,6 +1555,8 @@ Optional Packages: --with-blocksize=BLOCKSIZE set table block size in kB [8] --with-segsize=SEGSIZE set table segment size in GB [1] + --with-segsize-blocks=SEGSIZE_BLOCKS + set table segment size in blocks [0] --with-wal-blocksize=BLOCKSIZE set WAL block size in kB [8] --with-CC=CMD set compiler (deprecated) @@ -3735,8 +3738,6 @@ _ACEOF # # Relation segment size # -{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for segment size" >&5 -$as_echo_n "checking for segment size... " >&6; } @@ -3760,12 +3761,52 @@ else fi -# this expression is set up to avoid unnecessary integer overflow -# blocksize is already guaranteed to be a factor of 1024 -RELSEG_SIZE=`expr '(' 1024 / ${blocksize} ')' '*' ${segsize} '*' 1024` -test $? -eq 0 || exit 1 -{ $as_echo "$as_me:${as_lineno-$LINENO}: result: ${segsize}GB" >&5 + + + +# Check whether --with-segsize-blocks was given. +if test "${with_segsize_blocks+set}" = set; then : + withval=$with_segsize_blocks; + case $withval in + yes) + as_fn_error $? "argument required for --with-segsize-blocks option" "$LINENO" 5 + ;; + no) + as_fn_error $? "argument required for --with-segsize-blocks option" "$LINENO" 5 + ;; + *) + segsize_blocks=$withval + ;; + esac + +else + segsize_blocks=0 +fi + + + +# If --with-segsize-blocks is non-zero, it is used, --with-segsize +# otherwise. segsize-blocks is only really useful for developers wanting to +# test segment related code. Warn if both are used. +if test $segsize_blocks -ne 0 -a $segsize -ne 1; then + { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: both --with-segsize and --with-segsize-blocks specified, --with-segsize-blocks wins" >&5 +$as_echo "$as_me: WARNING: both --with-segsize and --with-segsize-blocks specified, --with-segsize-blocks wins" >&2;} +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for segment size" >&5 +$as_echo_n "checking for segment size... " >&6; } +if test $segsize_blocks -eq 0; then + # this expression is set up to avoid unnecessary integer overflow + # blocksize is already guaranteed to be a factor of 1024 + RELSEG_SIZE=`expr '(' 1024 / ${blocksize} ')' '*' ${segsize} '*' 1024` + test $? -eq 0 || exit 1 + { $as_echo "$as_me:${as_lineno-$LINENO}: result: ${segsize}GB" >&5 $as_echo "${segsize}GB" >&6; } +else + RELSEG_SIZE=$segsize_blocks + { $as_echo "$as_me:${as_lineno-$LINENO}: result: ${RELSEG_SIZE} blocks" >&5 +$as_echo "${RELSEG_SIZE} blocks" >&6; } +fi cat >>confdefs.h <<_ACEOF @@ -15554,9 +15595,11 @@ _ACEOF -# If we don't have largefile support, can't handle segsize >= 2GB. -if test "$ac_cv_sizeof_off_t" -lt 8 -a "$segsize" != "1"; then - as_fn_error $? "Large file support is not enabled. Segment size cannot be larger than 1GB." "$LINENO" 5 +# If we don't have largefile support, can't handle segment size >= 2GB. +if test "$ac_cv_sizeof_off_t" -lt 8; then + if expr $RELSEG_SIZE '*' $blocksize '>=' 2 '*' 1024 '*' 1024; then + as_fn_error $? "Large file support is not enabled. Segment size cannot be larger than 1GB." "$LINENO" 5 + fi fi # The cast to long int works around a bug in the HP C Compiler -- 2.38.0
