Re: Delta Materialized View Refreshes?
The idea of allowing a WHERE clause to be appended to REFRESH MATERIALIZED VIEW seems useful. It would enable those that know well the pattern of data modification in their underlying use case to schedule delta-updates (say, from crontab). And also it would be a useful as a foundation for more ambitious ideas. For example, some folk can/wish to do something entirely custom that is trigger-based. Others such as Nico's have toolkits that intend to cover quite a lot of the automagical refresh cases (see this other thread https://www.postgresql-archive.org/Query-Rewrite-for-Materialized-Views-FDW-Extension-td6015235.html), and it allows the data to remain in an MVIEW to boot, rather than forcing them to use normal tables instead. It could also be the foundation for something much more integrated such as Corey's CONTINUOUS MATERIALIZED VIEW concept. So I'm certainly supportive of the idea. -- Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html
Re: when set track_commit_timestamp on, database system abort startup
On Sat, Sep 15, 2018 at 12:29 AM, Alvaro Herrera
wrote:
> On 2018-Sep-15, Masahiko Sawada wrote:
>
>> On Fri, Sep 14, 2018 at 4:27 PM, 李海龙 wrote:
>
>> > When I enable the parameter track_commit_timestamp in postgresql.conf of a
>> > Base Backup (making a Base Backup from a standby and the
>> > track_commit_timestamp is off on it),
>>
>> In addition to the above operation, I've reproduced this issue by
>> replaying a commit WAL record that sets the timestamp to a new page
>> during the crash recovery (or from restart).
>>
>> It seems to me that the cause of this is that we could not extend
>> commitTs page since the COMMIT_TS_ZEROPAGE WAL wasn't generated at the
>> standby server whose track_commit_timestamp is off. So during
>> replaying the commit WAL record the startup process fails since the
>> corresponding commitTs page doesn't exist.
>
> Hmm, wow. I wonder if it's possible to detect the config difference
> early enough that the zeropage WAL records are emitted, instead. But
> even this might not work, since some transactions need to have their
> commitTS in pages that will not have been zeroed anyway, because the
> page threshold was crossed in the old primary.
>
>> To fix that maybe we can disable commitTs if
>> controlFile->track_commit_timestamp == false and the
>> track_commit_timestamp == true even in crash recovery.
>
> Hmm, so keep it off while crash recovery runs, and once it's out of that
> then enable it automatically?
Yes. The attached patch changes it to check
controlFile->track_commit_timestamp even the crash recover case. If
track_commit_timestamp is set to true in the config file, it's enabled
at end of the recovery.
> That might work -- by definition we don't
> care about the commit TSs of the transaction replayed during crash
> recovery, since they were executed in the primary that didn't have
> commitTS enable anyway.
>
> It seems like the first thing we need is TAP cases that reproduce these
> two crash scenarios.
I attached TAP test that reproduces this issue. We can reproduce it
even with single server; making postgres replay a commit WAL in the
crash recovery after consumed transactions and enabled
track_commit_timestamp.
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 493f1db..805bbe1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -6809,11 +6809,12 @@ StartupXLOG(void)
StartupMultiXact();
/*
- * Ditto commit timestamps. In a standby, we do it if setting is enabled
- * in ControlFile; in a master we base the decision on the GUC itself.
+ * Ditto commit timestamps. In both a standby and a master, we do it if
+ * setting is enabled in ControlFile since we don't care about the commit
+ * timestamps of the transaction that were executed when the commit
+ * timestamp is disabled.
*/
- if (ArchiveRecoveryRequested ?
- ControlFile->track_commit_timestamp : track_commit_timestamp)
+ if (ControlFile->track_commit_timestamp)
StartupCommitTs();
/*
diff --git a/src/test/modules/commit_ts/t/005_recovery.pl b/src/test/modules/commit_ts/t/005_recovery.pl
new file mode 100644
index 000..cdc1cf8
--- /dev/null
+++ b/src/test/modules/commit_ts/t/005_recovery.pl
@@ -0,0 +1,53 @@
+# Recovery test
+
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+
+my $node = get_new_node('test');
+$node->init;
+$node->append_conf('postgresql.conf',
+ "track_commit_timestamp = off");
+$node->start;
+
+# When we start firstly from the initdb the PARAMETER_CHANGES
+# is emitted at end of the recovery, which disables the
+# track_commit_timestamp if the crash recovery replay that
+# WAL. Therefore we restart the server so that we can recovery
+# from the point where doesn't contain that WAL.
+$node->restart;
+
+# Consume 2000 XIDs to beyond the commitTS page boundary.
+$node->safe_psql(
+ 'postgres',
+ qq(
+CREATE PROCEDURE comsume_xid(cnt int)
+AS \$\$
+DECLARE
+ i int;
+BEGIN
+ FOR i in 1..cnt LOOP
+ EXECUTE 'SELECT txid_current()';
+ COMMIT;
+ END LOOP;
+END;
+\$\$
+LANGUAGE plpgsql;
+));
+$node->safe_psql('postgres', 'CALL comsume_xid(2000)');
+
+$node->teardown_node;
+
+# Enable track_commit_tiemstamp
+$node->append_conf('postgresql.conf',
+ "track_commit_timestamp = on");
+
+# During the crash recovery we replay the commit WAL that sets
+# the commit timestamp to a new page.
+$node->start;
+
+# Check if the server launched.
+is($node->psql('postgres', qq(SELECT 1)), 0,
+ 'started from the crash recovery');
Re: Code of Conduct plan
Dear all, On Fri, Sep 14, 2018 at 5:18 PM Tom Lane wrote: > Robert Haas writes: > > It's not clear to me that there IS a general consensus here. It looks > > to me like the unelected core team got together and decided to impose > > a vaguely-worded code of conduct on a vaguely-defined group of people > > covering not only their work on PostgreSQL but also their entire life. > > There's been quite a lot of input, from quite a lot of people, dating > back at least as far as a well-attended session at PGCon 2016. I find > it quite upsetting to hear accusations that core is imposing this out > of nowhere. From my perspective, we're responding to a real need > voiced by other people, not so much by us. > > > However, I also don't think it matters very much. > > Yeah, this. The PG community is mostly nice people, AFAICT. I'll be > astonished (and worried) if the CoC committee finds much to do. We're > implementing this mostly to make newcomers to the project feel that > it's a safe space. > > It's also worth reminding people that this is v1.0 of the CoC document. > We plan to revisit it in a year or so, and thereafter as needed, to > improve anything that's causing problems or not working well. > > regards, tom lane > I must admit that I'm impressed by the huge amount of contributions to this thread and, to be honest, it is the only one I have witnessed that would have deserved a CoC. I had a quick look at the proposal and it sounds to me like the team is trying to handle excesses - as long as no one complains, I would bet that they won't even chime in. One thing to keep in mind is this simple definition: "One person's freedom ends where another's begins" and all the work should go in this direction. We are all different, have different sensitivities, come from different cultures where we interpret words in a different way - it's a given, no way to escape. But we have in common the love of a great piece of software provided by a very active and efficient community. Why don't we focus on what unites us, instead of what creates divisions? Have a peaceful week-end Olivier
Re: move PartitionDispatchData definition to execPartition.c
On 2018-Sep-14, Tom Lane wrote: > Alvaro Herrera writes: > > On 2018-Jul-04, Amit Langote wrote: > >> I think we may have simply forgotten to do $subject in the following > >> commit. > > > We did. Pushed now, thanks. > > Erm, shouldn't that have been done in HEAD as well as v11? Hmm, the commit was there, but it was not pushed. Workflow mistake. Pushed now. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: PATCH: pgbench - option to build using ppoll() for larger connection counts
The author hasn't replied, but the attached seems to have cured the
bitrot so that it at least applies. Let's see what the cfbot makes of
it and then possibly fix any Windows issues.
The patch was not applying cleanly anymore for me, so here is a rebase of
your latest version.
Morever, ISTM that Tom's "why?" question has been answered: there are very
large systems out there with many processors, which are tested against
many connections, exceeding select limit.
I have turned back this patch to ready.
--
Fabien.diff --git a/configure b/configure
index c6a44a9078..0456064456 100755
--- a/configure
+++ b/configure
@@ -15060,7 +15060,7 @@ fi
LIBS_including_readline="$LIBS"
LIBS=`echo "$LIBS" | sed -e 's/-ledit//g' -e 's/-lreadline//g'`
-for ac_func in cbrt clock_gettime fdatasync getifaddrs getpeerucred getrlimit mbstowcs_l memmove poll posix_fallocate pstat pthread_is_threaded_np readlink setproctitle setproctitle_fast setsid shm_open symlink sync_file_range utime utimes wcstombs_l
+for ac_func in cbrt clock_gettime fdatasync getifaddrs getpeerucred getrlimit mbstowcs_l memmove poll posix_fallocate ppoll pstat pthread_is_threaded_np readlink setproctitle setproctitle_fast setsid shm_open symlink sync_file_range utime utimes wcstombs_l
do :
as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh`
ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var"
diff --git a/configure.in b/configure.in
index 3ada48b5f9..fceba79023 100644
--- a/configure.in
+++ b/configure.in
@@ -1544,7 +1544,7 @@ PGAC_FUNC_WCSTOMBS_L
LIBS_including_readline="$LIBS"
LIBS=`echo "$LIBS" | sed -e 's/-ledit//g' -e 's/-lreadline//g'`
-AC_CHECK_FUNCS([cbrt clock_gettime fdatasync getifaddrs getpeerucred getrlimit mbstowcs_l memmove poll posix_fallocate pstat pthread_is_threaded_np readlink setproctitle setproctitle_fast setsid shm_open symlink sync_file_range utime utimes wcstombs_l])
+AC_CHECK_FUNCS([cbrt clock_gettime fdatasync getifaddrs getpeerucred getrlimit mbstowcs_l memmove poll posix_fallocate ppoll pstat pthread_is_threaded_np readlink setproctitle setproctitle_fast setsid shm_open symlink sync_file_range utime utimes wcstombs_l])
AC_REPLACE_FUNCS(fseeko)
case $host_os in
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 41b756c089..3d378db714 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -45,9 +45,18 @@
#include
#include
#include
+#ifndef PGBENCH_USE_SELECT /* force use of select(2)? */
+#ifdef HAVE_PPOLL
+#define POLL_USING_PPOLL
+#include
+#endif
+#endif
+#ifndef POLL_USING_PPOLL
+#define POLL_USING_SELECT
#ifdef HAVE_SYS_SELECT_H
#include
#endif
+#endif
#ifdef HAVE_SYS_RESOURCE_H
#include /* for getrlimit */
@@ -92,13 +101,19 @@ static int pthread_join(pthread_t th, void **thread_return);
/
* some configurable parameters */
-
-/* max number of clients allowed */
+#ifdef POLL_USING_SELECT /* using select(2) */
+#define SOCKET_WAIT_METHOD "select"
+typedef fd_set socket_set;
#ifdef FD_SETSIZE
-#define MAXCLIENTS (FD_SETSIZE - 10)
+#define MAXCLIENTS (FD_SETSIZE - 10) /* system limited max number of clients allowed */
#else
-#define MAXCLIENTS 1024
+#define MAXCLIENTS 1024 /* max number of clients allowed */
#endif
+#else /* using ppoll(2) */
+#define SOCKET_WAIT_METHOD "ppoll"
+typedef struct pollfd socket_set;
+#define MAXCLIENTS -1 /* unlimited number of clients */
+#endif /* POLL_USING_SELECT */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
@@ -525,6 +540,13 @@ static void addScript(ParsedScript script);
static void *threadRun(void *arg);
static void setalarm(int seconds);
static void finishCon(CState *st);
+static socket_set *alloc_socket_set(int count);
+static bool error_on_socket(socket_set *sa, int idx, PGconn *con);
+static void free_socket_set(socket_set *sa);
+static bool ignore_socket(socket_set *sa, int idx, PGconn *con);
+static void clear_socket_set(socket_set *sa, int count);
+static void set_socket(socket_set *sa, int fd, int idx);
+static int wait_on_socket_set(socket_set *sa, int nstate, int maxsock, int64 usec);
/* callback functions for our flex lexer */
@@ -1143,6 +1165,7 @@ doConnect(void)
!have_password)
{
PQfinish(conn);
+ conn = NULL;
simple_prompt("Password: ", password, sizeof(password), false);
have_password = true;
new_pass = true;
@@ -4903,7 +4926,7 @@ main(int argc, char **argv)
case 'c':
benchmarking_option_set = true;
nclients = atoi(optarg);
-if (nclients <= 0 || nclients > MAXCLIENTS)
+if (nclients <= 0 || (MAXCLIENTS != -1 && nclients > MAXCLIENTS))
{
fprintf(stderr, "invalid number of clients: \"%s\"\n",
optarg);
@@ -5614,6 +5637,7 @@ threadRun(void *arg)
int64 next_report = last_report + (int64) progress * 100;
StatsData last,
aggs;
+ socket_set *sockets = alloc_socket_set(nstate);
/*
* Initialize throttling rat
Re: Defaulting to jit=on/off for v11
On 09/14/2018 08:18 PM, Peter Geoghegan wrote: On Fri, Sep 14, 2018 at 3:32 PM, Tom Lane wrote: I'd go with #2, personally. It does seem that the costing needs work, but it's not clear to me that we know what to change, so it's kinda late to propose #3 for v11. +1. I also favor option #2. + about 0.8. I hope we do get some good field testing if it, though. cheers andrew -- Andrew Dunstanhttps://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: Postgres 11 release notes
Hi, On 9/12/18 1:34 PM, Michael Banck wrote: > Hi, > > On Fri, May 11, 2018 at 11:08:52AM -0400, Bruce Momjian wrote: >> I have committed the first draft of the Postgres 11 release notes. I >> will add more markup soon. You can view the most current version here: >> >> http://momjian.us/pgsql_docs/release-11.html > > The first item of section 'E.1.3.10. Server Applications' is mine and > has this TODO (though maybe that should be better marked up?) text: > > |IS IT CLEAR FROM THE DOCS THAT THE REPLICATION SLOT IS NOT TEMPORARY? > > So I guess we need to decide on that before release. The current doc is > AFAIK: > > | This option causes the replication slot specified by the > | option --slot to be created before starting the > | backup. In this case, an error is raised if the slot already exists. > > I guess it does not hurt to have something like "to be (permanently) > created before starting", but what do others think? Is it clear enough? Attached are two patches. The first modifies the major improvements section to align with what we have discovered during the beta period + the general advocacy push. It also modifies the "last updated date" + narrows down a prospective release month. The second modifies/removes the questions/placeholders in the release notes, such as Michael's comment above (which I agreed with and added a one sentence explanation after). Thanks, Jonathan From 2733f0df599aeda48408a3d27b796e36fbdd6fc3 Mon Sep 17 00:00:00 2001 From: "Jonathan S. Katz" Date: Sat, 15 Sep 2018 10:43:44 -0400 Subject: [PATCH 1/2] Updates to major improvements section of release notes. --- doc/src/sgml/release-11.sgml | 47 ++-- 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/release-11.sgml b/doc/src/sgml/release-11.sgml index 684d34c091..9ed34e72e8 100644 --- a/doc/src/sgml/release-11.sgml +++ b/doc/src/sgml/release-11.sgml @@ -6,7 +6,7 @@ Release date: - 2018-??-?? (CURRENT AS OF 2018-07-30) + 2018-10-?? (CURRENT AS OF 2018-09-20) @@ -22,11 +22,11 @@ - Major improvements to partitioning: + Improvements to partitioning functionality, including: - Partitioning by a hash key + Partitioning by a hash key (a.k.a. "hash partitioning") @@ -37,9 +37,8 @@ - Improved SELECT query performance due to - enhanced partition elimination during query processing and - execution + Improved SELECT performance from enhanced partition + elimination strategies during query processing and execution @@ -48,29 +47,37 @@ KEY, indexes, and triggers on partitioned tables + + + Having a "default" partition for storing data that does not match a + partition key + + - Improvements to parallelism: + Improvements to parallelism, including: - Parallelized hash joins + B-tree indexes can now be built in parallel with + CREATE INDEX - Parallelized CREATE INDEX for B-tree indexes + Parallelized CREATE TABLE .. AS, + CREATE MATERIALIZED VIEW, and certain + queries using UNION - Parallelized CREATE TABLE .. AS, - CREATE MATERIALIZED VIEW, and certain - queries using UNION + Performance improvements for parallelized hash joins and parallelized + sequential scans @@ -79,14 +86,17 @@ - SQL stored procedures, with support for embedded transactions + SQL stored procedures that support embedded transactions. Stored + procedures can be created with + CREATE PROCEDURE and executed with + CALL - JIT compilation of some SQL code, including support for fast evaluation - of expressions + Introduction of just-in-time (JIT) compilation + during query execution @@ -99,6 +109,13 @@ + + + Covering indexes, which can be utilized using the + INCLUDE clause of CREATE INDEX + + + Many other useful performance improvements, including making -- 2.14.3 (Apple Git-98) From feb08e3747e5dc043e21b60e9e681f44a0562896 Mon Sep 17 00:00:00 2001 From: "Jonathan S. Katz" Date: Sat, 15 Sep 2018 11:01:35 -0400 Subject: [PATCH 2/2] Remove or modify placeholders for v11 release notes. --- doc/src/sgml/release-11.sgml | 21 ++--- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/release-11.sgml b/doc/src/sgml/release-11.sgml index 9ed34e72e8..e8af023140 100644 --- a/doc/src/sgml/release-11.sgml +++ b/doc/src/sgml/release-11.sgml @@
Re: [HACKERS] proposal: schema variables
> The code is more cleaner now, there are more tests, and documentation is > mostly complete. I am sorry - my English is not good. > New features: > > o ON COMMIT DROP and ON TRANSACTION END RESET -- remove temp variable on > commit, reset variable on transaction end (commit, rollback) > o LET var = DEFAULT -- reset specified variable > > fix some forgotten warnings and dependency issue few more tests Regards Pavel > Regards > > Pavel > > >> Regards, >> Dean >> > schema-variables-20180915-01.patch.gz Description: application/gzip
Re: Defaulting to jit=on/off for v11
On September 15, 2018 8:26:17 AM MDT, Andrew Dunstan wrote: > > >On 09/14/2018 08:18 PM, Peter Geoghegan wrote: >> On Fri, Sep 14, 2018 at 3:32 PM, Tom Lane wrote: >>> I'd go with #2, personally. It does seem that the costing needs >work, >>> but it's not clear to me that we know what to change, so it's kinda >>> late to propose #3 for v11. >> +1. I also favor option #2. >> > >+ about 0.8. I hope we do get some good field testing if it, though. Cool, sounds we have agreement. I'll try to come up with a patch. I'm [un]fortunately hiking till Wednesday, so I won't have an easy time to push a patch. I probably could push something tomorrow, but I'd have a hard time cleaning up if needed. Does anybody feel we should have that in Mondays release? Andres Andres -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Defaulting to jit=on/off for v11
Andres Freund writes: > Cool, sounds we have agreement. I'll try to come up with a patch. I'm > [un]fortunately hiking till Wednesday, so I won't have an easy time to push a > patch. I probably could push something tomorrow, but I'd have a hard time > cleaning up if needed. Does anybody feel we should have that in Mondays > release? Recent discussions among the release team are pointing to the idea that Monday's wrap could be 11rc1, not 11beta4, if we get this done beforehand. So I'd kind of like to see it done. Is there more that has to be done than switching the GUC's default value and adjusting docs? I could probably make it happen if there's not any hidden stuff to worry about. regards, tom lane
Re: Defaulting to jit=on/off for v11
Hi, On September 15, 2018 12:14:07 PM MDT, Tom Lane wrote: >Andres Freund writes: >> Cool, sounds we have agreement. I'll try to come up with a patch. >I'm [un]fortunately hiking till Wednesday, so I won't have an easy time >to push a patch. I probably could push something tomorrow, but I'd >have a hard time cleaning up if needed. Does anybody feel we should >have that in Mondays release? > >Recent discussions among the release team are pointing to the idea >that Monday's wrap could be 11rc1, not 11beta4, if we get this done >beforehand. So I'd kind of like to see it done. Yes, that's why I'm asking... >Is there more that has to be done than switching the GUC's default >value and adjusting docs? I could probably make it happen if there's >not any hidden stuff to worry about. No, that should be all. I'll adapt the llvm animals to force the GUC to on. I think it'd be good to add a paragraph to the release notes, mentioning that it has to be turned on. Thanks! Andres -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Defaulting to jit=on/off for v11
Andres Freund writes: > On September 15, 2018 12:14:07 PM MDT, Tom Lane wrote: >> Is there more that has to be done than switching the GUC's default >> value and adjusting docs? I could probably make it happen if there's >> not any hidden stuff to worry about. > No, that should be all. I'll adapt the llvm animals to force the GUC to on. > I think it'd be good to add a paragraph to the release notes, mentioning that > it has to be turned on. > Thanks! Roger, I'll have a go at it. regards, tom lane
Re: Defaulting to jit=on/off for v11
On 2018-09-15 14:19:55 -0400, Tom Lane wrote: > Andres Freund writes: > > On September 15, 2018 12:14:07 PM MDT, Tom Lane wrote: > >> Is there more that has to be done than switching the GUC's default > >> value and adjusting docs? I could probably make it happen if there's > >> not any hidden stuff to worry about. > > > No, that should be all. I'll adapt the llvm animals to force the GUC to > > on. I think it'd be good to add a paragraph to the release notes, > > mentioning that it has to be turned on. > > > Thanks! > > Roger, I'll have a go at it. Thanks! Just managed to log into the machine hosting my llvm testing buildfarm animals and set jit=1 for those. So we shouldn't loose too much coverage. I'll email the folks that have LLVM enabled in the BF, to also do so. Greetings, Andres Freund
Re: Defaulting to jit=on/off for v11
Hi, On 2018-09-15 11:41:58 -0700, Andres Freund wrote: > I'll email the folks that have LLVM enabled in the BF, to also do so. Done so. I'll try to check-in again tonight, but while likely, it's not certain there'll be reception. - Andres
Re: Should contrib modules install .h files?
> "Andrew" == Andrew Gierth writes:
Andrew> Here's a patch that fixes (not necessarily in the best way) the
Andrew> PGXS builds of all the contrib/*_pl{perl,python} modules.
Andrew> Oh, obviously this patch doesn't fix the windows Install.pm
Andrew> yet, but that'd be easier to do after finalizing the list of
Andrew> include files to install.
So while looking into this, I found that to the best of my ability to
determine, the windows Install.pm has _never_ installed the plperl or
plpython header files. That's more than I'm in a position to fix, having
no windows box to test with.
So what I propose to do is to commit a cleaned-up version of the patch
posted above, with these changes:
- install all the plpy_*.h headers, not just a few; I know of no reason
to exclude any of them, and in the absence of feedback it seems
better to install them all;
- tidy up the makefile variables in the python transform modules to
remove some duplication
- fix Mkvcbuild.pm to account for the changes
This will allow hstore_plperl etc. to stand as examples of how to do
cross-module #includes both in and out of tree.
I'll post the patch for review shortly.
--
Andrew (irc:RhodiumToad)
Re: Index Skip Scan
> On Thu, 13 Sep 2018 at 21:36, Alexander Kuzmenkov > wrote: > > El 13/09/18 a las 18:39, Jesper Pedersen escribió: > >> I think we can improve this, >> and the skip scan can be strictly faster than index scan regardless of >> the data. As a first approximation, imagine that we somehow skipped >> equal tuples inside _bt_next instead of sending them to the parent >> Unique node. This would already be marginally faster than Unique + Index >> scan. A more practical implementation would be to remember our position >> in tree (that is, BTStack returned by _bt_search) and use it to skip >> pages in bulk. This looks straightforward to implement for a tree that >> does not change, but I'm not sure how to make it work with concurrent >> modifications. Still, this looks a worthwhile direction to me, because >> if we have a strictly faster skip scan, we can just use it always and >> not worry about our unreliable statistics. What do you think? >> > > This is something to look at -- maybe there is a way to use > btpo_next/btpo_prev instead/too in order to speed things up. Atm we just > have the scan key in BTScanOpaqueData. I'll take a look after my > upcoming vacation; feel free to contribute those changes in the meantime > of course. But having this logic inside _bt_next means that it will make a non-skip index only scan a bit slower, am I right? Probably it would be easier and more straightforward to go with the idea of dynamic fallback then. The first naive implementation that I came up with is to wrap an index scan node into a unique, and remember estimated number of groups into IndexOnlyScanState, so that we can check if we performed much more skips than expected. With this approach index skip scan will work a bit slower than in the original patch in case if ndistinct is correct (because a unique node will recheck rows we returned), and fallback to unique + index only scan in case if planner has underestimated ndistinct. index-skip-fallback.patch Description: Binary data
Re: Should contrib modules install .h files?
> "Andrew" == Andrew Gierth writes: Andrew> So what I propose to do is to commit a cleaned-up version of Andrew> the patch posted above, with these changes: Andrew> - install all the plpy_*.h headers, not just a few; I know of no reason Andrew>to exclude any of them, and in the absence of feedback it seems Andrew>better to install them all; Andrew> - tidy up the makefile variables in the python transform modules to Andrew>remove some duplication Andrew> - fix Mkvcbuild.pm to account for the changes Andrew> This will allow hstore_plperl etc. to stand as examples of how Andrew> to do cross-module #includes both in and out of tree. Andrew> I'll post the patch for review shortly. And here it is. -- Andrew (irc:RhodiumToad) diff --git a/contrib/hstore_plperl/Makefile b/contrib/hstore_plperl/Makefile index f63cba2745..32ecaa43cb 100644 --- a/contrib/hstore_plperl/Makefile +++ b/contrib/hstore_plperl/Makefile @@ -4,7 +4,6 @@ MODULE_big = hstore_plperl OBJS = hstore_plperl.o $(WIN32RES) PGFILEDESC = "hstore_plperl - hstore transform for plperl" -PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plperl -I$(top_srcdir)/contrib/hstore EXTENSION = hstore_plperl hstore_plperlu DATA = hstore_plperl--1.0.sql hstore_plperlu--1.0.sql @@ -13,10 +12,12 @@ REGRESS = hstore_plperl hstore_plperlu create_transform EXTRA_INSTALL = contrib/hstore ifdef USE_PGXS +PG_CPPFLAGS = -I$(includedir_server)/extension PG_CONFIG = pg_config PGXS := $(shell $(PG_CONFIG) --pgxs) include $(PGXS) else +PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plperl -I$(top_srcdir)/contrib subdir = contrib/hstore_plperl top_builddir = ../.. include $(top_builddir)/src/Makefile.global diff --git a/contrib/hstore_plperl/hstore_plperl.c b/contrib/hstore_plperl/hstore_plperl.c index c09bd38d09..61b5557421 100644 --- a/contrib/hstore_plperl/hstore_plperl.c +++ b/contrib/hstore_plperl/hstore_plperl.c @@ -5,7 +5,7 @@ #include "fmgr.h" #include "plperl.h" #include "plperl_helpers.h" -#include "hstore.h" +#include "hstore/hstore.h" PG_MODULE_MAGIC; diff --git a/contrib/hstore_plpython/Makefile b/contrib/hstore_plpython/Makefile index b81735ab91..6877e7a072 100644 --- a/contrib/hstore_plpython/Makefile +++ b/contrib/hstore_plpython/Makefile @@ -4,19 +4,21 @@ MODULE_big = hstore_plpython$(python_majorversion) OBJS = hstore_plpython.o $(WIN32RES) PGFILEDESC = "hstore_plpython - hstore transform for plpython" -PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plpython $(python_includespec) -I$(top_srcdir)/contrib/hstore -DPLPYTHON_LIBNAME='"plpython$(python_majorversion)"' - EXTENSION = hstore_plpythonu hstore_plpython2u hstore_plpython3u DATA = hstore_plpythonu--1.0.sql hstore_plpython2u--1.0.sql hstore_plpython3u--1.0.sql REGRESS = hstore_plpython REGRESS_PLPYTHON3_MANGLE := $(REGRESS) +PG_CPPFLAGS = $(python_includespec) -DPLPYTHON_LIBNAME='"plpython$(python_majorversion)"' + ifdef USE_PGXS +PG_CPPFLAGS += -I$(includedir_server)/extension PG_CONFIG = pg_config PGXS := $(shell $(PG_CONFIG) --pgxs) include $(PGXS) else +PG_CPPFLAGS += -I$(top_srcdir)/src/pl/plpython -I$(top_srcdir)/contrib subdir = contrib/hstore_plpython top_builddir = ../.. include $(top_builddir)/src/Makefile.global diff --git a/contrib/hstore_plpython/hstore_plpython.c b/contrib/hstore_plpython/hstore_plpython.c index 218e6612b1..2f24090ff3 100644 --- a/contrib/hstore_plpython/hstore_plpython.c +++ b/contrib/hstore_plpython/hstore_plpython.c @@ -3,7 +3,7 @@ #include "fmgr.h" #include "plpython.h" #include "plpy_typeio.h" -#include "hstore.h" +#include "hstore/hstore.h" PG_MODULE_MAGIC; diff --git a/contrib/ltree_plpython/Makefile b/contrib/ltree_plpython/Makefile index 7e988c7993..ce2c0cd2e2 100644 --- a/contrib/ltree_plpython/Makefile +++ b/contrib/ltree_plpython/Makefile @@ -4,19 +4,21 @@ MODULE_big = ltree_plpython$(python_majorversion) OBJS = ltree_plpython.o $(WIN32RES) PGFILEDESC = "ltree_plpython - ltree transform for plpython" -PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plpython $(python_includespec) -I$(top_srcdir)/contrib/ltree -DPLPYTHON_LIBNAME='"plpython$(python_majorversion)"' - EXTENSION = ltree_plpythonu ltree_plpython2u ltree_plpython3u DATA = ltree_plpythonu--1.0.sql ltree_plpython2u--1.0.sql ltree_plpython3u--1.0.sql REGRESS = ltree_plpython REGRESS_PLPYTHON3_MANGLE := $(REGRESS) +PG_CPPFLAGS = $(python_includespec) -DPLPYTHON_LIBNAME='"plpython$(python_majorversion)"' + ifdef USE_PGXS +PG_CPPFLAGS += -I$(includedir_server)/extension PG_CONFIG = pg_config PGXS := $(shell $(PG_CONFIG) --pgxs) include $(PGXS) else +PG_CPPFLAGS += -I$(top_srcdir)/src/pl/plpython -I$(top_srcdir)/contrib subdir = contrib/ltree_plpython top_builddir = ../.. include $(top_builddir)/src/Makefile.global diff --git a/contrib/ltree_plpython/ltree_plpython.c b/contrib/ltree_plpython/ltree_plpython.c index e88636a0a9..b254aa558d 100644 --- a/contrib/ltree_plpython/ltree_plpython.c +++ b/contrib/ltree_plpython/ltree_plpython.c @@ -2,7
Re: Something fishy happening on frogmouth
On Wed, Oct 30, 2013 at 09:07:43AM -0400, Robert Haas wrote:
> On Wed, Oct 30, 2013 at 8:47 AM, Andres Freund wrote:
> > On 2013-10-30 08:45:03 -0400, Robert Haas wrote:
> >> If I'm reading this correctly, the last three runs on frogmouth have
> >> all failed, and all of them have failed with a complaint about,
> >> specifically, Global/PostgreSQL.851401618. Now, that really shouldn't
> >> be happening, because the code to choose that number looks like this:
> >>
> >> dsm_control_handle = random();
> > Could it be that we haven't primed the random number generator with the
> > time or something like that yet?
>
> Yeah, I think that's probably what it is.
I experienced a variation of this, namely a RHEL 7 system where initdb always
says "selecting dynamic shared memory implementation ... sysv". Each initdb
is rejecting posix shm by probing the same ten segments:
$ strace initdb -D scratch 2>&1 | grep /dev/shm/P
open("/dev/shm/PostgreSQL.1804289383",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.846930886",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.1681692777",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.1714636915",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.1957747793",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.424238335",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.719885386",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.1649760492",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.596516649",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.1189641421",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
Regular postmaster runs choose a random segment, but initdb, bootstrap
postgres, and single-user postgres all start with the same segment. These
segments are months old. Perhaps I was testing something that caused a
bootstrap postgres to crash. After ten such crashes, future initdb runs
considered posix shm unusable.
> There's PostmasterRandom()
> to initialize the random-number generator on first use, but that
> doesn't help if some other module calls random(). I wonder if we
> ought to just get rid of PostmasterRandom() and instead have the
> postmaster run that initialization code very early in startup.
Usually, the first srandom() call happens early in PostmasterMain(). I plan
to add one to InitStandaloneProcess(), which substitutes for several tasks
otherwise done in PostmasterMain(). That seems like a good thing even if DSM
weren't in the picture. Also, initdb needs an srandom() somewhere;
choose_dsm_implementation() itself seems fine. Attached. With this, "make
-j20 check-world" selected posix shm and passed even when I forced DSM
creation to fail on unseeded random():
--- a/src/backend/storage/ipc/dsm_impl.c
+++ b/src/backend/storage/ipc/dsm_impl.c
@@ -249,2 +249,5 @@ dsm_impl_posix(dsm_op op, dsm_handle handle, Size
request_size,
+ if (handle == 1804289383)
+ elog(ERROR, "generated handle with no randomness");
+
snprintf(name, 64, "/PostgreSQL.%u", handle);
diff --git a/src/backend/utils/init/miscinit.c
b/src/backend/utils/init/miscinit.c
index 865119d..f003831 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -325,6 +325,8 @@ InitStandaloneProcess(const char *argv0)
MyStartTime = time(NULL); /* set our start time in case we call
elog */
+ srandom((unsigned int) (MyProcPid ^ MyStartTime));
+
/* Initialize process-local latch support */
InitializeLatchSupport();
MyLatch = &LocalLatchData;
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 32746c7..83f4b0b 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -922,6 +922,8 @@ choose_dsm_implementation(void)
#ifdef HAVE_SHM_OPEN
int ntries = 10;
+ srandom((unsigned int) (getpid() ^ time(NULL)));
+
while (ntries > 0)
{
uint32 handle;
Re: Something fishy happening on frogmouth
Noah Misch writes: > Usually, the first srandom() call happens early in PostmasterMain(). I plan > to add one to InitStandaloneProcess(), which substitutes for several tasks > otherwise done in PostmasterMain(). That seems like a good thing even if DSM > weren't in the picture. Also, initdb needs an srandom() somewhere; > choose_dsm_implementation() itself seems fine. Attached. +1, but some comments would be good. regards, tom lane
Re: Code of Conduct plan
On 9/14/18 11:13 AM, Robert Haas wrote: On Fri, Sep 14, 2018 at 11:10 AM, Dave Page wrote: That wording has been in the published draft for 18 months, and noone objected to it that I'm aware of. There will always be people who don't like some of the wording, much as there are often people who disagree with the way a patch to the code is written. Sooner or later though, the general consensus prevails and we have to move on, otherwise nothing will ever get completed. It's not clear to me that there IS a general consensus here. It looks to me like the unelected core team got together and decided to impose a vaguely-worded code of conduct on a vaguely-defined group of people covering not only their work on PostgreSQL but also their entire life. It is not difficult to imagine that someone's private life might include "behavior that may bring the PostgreSQL project into disrepute." However, I also don't think it matters very much. The Code of Conduct Committee is going to consist of small number of people -- at least four, perhaps a few more. But there are hundreds of people involved on the PostgreSQL mailing lists, maybe thousands. If the Code of Conduct Committee, or the core team, believes that it can impose on a very large group of people, all of whom are volunteers, some set of rules with which they don't agree, it's probably going to find out pretty quickly that it is mistaken. If people from that large group get banned for behavior which is perceived by other members of that large group to be legitimate, then there will be a ferocious backlash. Nobody wants to see people who are willing to contribute driven away from the project, and anyone we drive away without a really good reason will find some other project that welcomes their participation. So the only thing that the Code of Conduct Committee is likely to be able to do in practice is admonish people to be nicer (which is probably a good thing) and punish really egregious conduct, especially when committed by people who aren't involved enough that their absence will be keenly felt. In practice, therefore, democracy is going to win out. That's both good and bad. It's good because nobody wants a CoC witch-hunt, and it's bad because there's probably some behavior which legitimately deserves censure and will escape it. +1 -- Adrian Klaver adrian.kla...@aklaver.com
Re: Code of Conduct plan
On 15/09/18 08:17, Tom Lane wrote: Yeah, this. The PG community is mostly nice people, AFAICT. I'll be astonished (and worried) if the CoC committee finds much to do. We're implementing this mostly to make newcomers to the project feel that it's a safe space. Agreed. However I think the all-of-life clause gives an open door to potential less than well intentioned new members joining up to extend a SJW agenda. So in fact the unintended consequence of this may be a *less* safe place for some existing members - unless all of their social media utterances are agreeable to the angry militant left. It's also worth reminding people that this is v1.0 of the CoC document. We plan to revisit it in a year or so, and thereafter as needed, to improve anything that's causing problems or not working well. +1, At least this means we can address the above if it emerges as a problem regards Mark regards, tom lane
Re: Delta Materialized View Refreshes?
The idea of allowing a WHERE clause to be appended to REFRESH MATERIALIZED VIEW seems useful. It would enable those that know well the pattern of data modification in their underlying use case to schedule delta-updates (say, from crontab). And also it would be a useful as a foundation for more ambitious ideas. For example, some folk can/wish to do something entirely custom that is trigger-based. Others such as Nico's have toolkits that intend to cover quite a lot of the automagical refresh cases (see this other thread https://www.postgresql-archive.org/Query-Rewrite-for-Materialized-Views-FDW-Extension-td6015235.html), and it allows the data to remain in an MVIEW to boot, rather than forcing them to use normal tables instead. It could also be the foundation for something much more integrated such as Corey's CONTINUOUS MATERIALIZED VIEW concept. So I'm certainly supportive of the idea. -- Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html
Re: when set track_commit_timestamp on, database system abort startup
On Sat, Sep 15, 2018 at 12:29 AM, Alvaro Herrera
wrote:
> On 2018-Sep-15, Masahiko Sawada wrote:
>
>> On Fri, Sep 14, 2018 at 4:27 PM, 李海龙 wrote:
>
>> > When I enable the parameter track_commit_timestamp in postgresql.conf of a
>> > Base Backup (making a Base Backup from a standby and the
>> > track_commit_timestamp is off on it),
>>
>> In addition to the above operation, I've reproduced this issue by
>> replaying a commit WAL record that sets the timestamp to a new page
>> during the crash recovery (or from restart).
>>
>> It seems to me that the cause of this is that we could not extend
>> commitTs page since the COMMIT_TS_ZEROPAGE WAL wasn't generated at the
>> standby server whose track_commit_timestamp is off. So during
>> replaying the commit WAL record the startup process fails since the
>> corresponding commitTs page doesn't exist.
>
> Hmm, wow. I wonder if it's possible to detect the config difference
> early enough that the zeropage WAL records are emitted, instead. But
> even this might not work, since some transactions need to have their
> commitTS in pages that will not have been zeroed anyway, because the
> page threshold was crossed in the old primary.
>
>> To fix that maybe we can disable commitTs if
>> controlFile->track_commit_timestamp == false and the
>> track_commit_timestamp == true even in crash recovery.
>
> Hmm, so keep it off while crash recovery runs, and once it's out of that
> then enable it automatically?
Yes. The attached patch changes it to check
controlFile->track_commit_timestamp even the crash recover case. If
track_commit_timestamp is set to true in the config file, it's enabled
at end of the recovery.
> That might work -- by definition we don't
> care about the commit TSs of the transaction replayed during crash
> recovery, since they were executed in the primary that didn't have
> commitTS enable anyway.
>
> It seems like the first thing we need is TAP cases that reproduce these
> two crash scenarios.
I attached TAP test that reproduces this issue. We can reproduce it
even with single server; making postgres replay a commit WAL in the
crash recovery after consumed transactions and enabled
track_commit_timestamp.
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 493f1db..805bbe1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -6809,11 +6809,12 @@ StartupXLOG(void)
StartupMultiXact();
/*
- * Ditto commit timestamps. In a standby, we do it if setting is enabled
- * in ControlFile; in a master we base the decision on the GUC itself.
+ * Ditto commit timestamps. In both a standby and a master, we do it if
+ * setting is enabled in ControlFile since we don't care about the commit
+ * timestamps of the transaction that were executed when the commit
+ * timestamp is disabled.
*/
- if (ArchiveRecoveryRequested ?
- ControlFile->track_commit_timestamp : track_commit_timestamp)
+ if (ControlFile->track_commit_timestamp)
StartupCommitTs();
/*
diff --git a/src/test/modules/commit_ts/t/005_recovery.pl b/src/test/modules/commit_ts/t/005_recovery.pl
new file mode 100644
index 000..cdc1cf8
--- /dev/null
+++ b/src/test/modules/commit_ts/t/005_recovery.pl
@@ -0,0 +1,53 @@
+# Recovery test
+
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+
+my $node = get_new_node('test');
+$node->init;
+$node->append_conf('postgresql.conf',
+ "track_commit_timestamp = off");
+$node->start;
+
+# When we start firstly from the initdb the PARAMETER_CHANGES
+# is emitted at end of the recovery, which disables the
+# track_commit_timestamp if the crash recovery replay that
+# WAL. Therefore we restart the server so that we can recovery
+# from the point where doesn't contain that WAL.
+$node->restart;
+
+# Consume 2000 XIDs to beyond the commitTS page boundary.
+$node->safe_psql(
+ 'postgres',
+ qq(
+CREATE PROCEDURE comsume_xid(cnt int)
+AS \$\$
+DECLARE
+ i int;
+BEGIN
+ FOR i in 1..cnt LOOP
+ EXECUTE 'SELECT txid_current()';
+ COMMIT;
+ END LOOP;
+END;
+\$\$
+LANGUAGE plpgsql;
+));
+$node->safe_psql('postgres', 'CALL comsume_xid(2000)');
+
+$node->teardown_node;
+
+# Enable track_commit_tiemstamp
+$node->append_conf('postgresql.conf',
+ "track_commit_timestamp = on");
+
+# During the crash recovery we replay the commit WAL that sets
+# the commit timestamp to a new page.
+$node->start;
+
+# Check if the server launched.
+is($node->psql('postgres', qq(SELECT 1)), 0,
+ 'started from the crash recovery');
Re: Code of Conduct plan
Dear all, On Fri, Sep 14, 2018 at 5:18 PM Tom Lane wrote: > Robert Haas writes: > > It's not clear to me that there IS a general consensus here. It looks > > to me like the unelected core team got together and decided to impose > > a vaguely-worded code of conduct on a vaguely-defined group of people > > covering not only their work on PostgreSQL but also their entire life. > > There's been quite a lot of input, from quite a lot of people, dating > back at least as far as a well-attended session at PGCon 2016. I find > it quite upsetting to hear accusations that core is imposing this out > of nowhere. From my perspective, we're responding to a real need > voiced by other people, not so much by us. > > > However, I also don't think it matters very much. > > Yeah, this. The PG community is mostly nice people, AFAICT. I'll be > astonished (and worried) if the CoC committee finds much to do. We're > implementing this mostly to make newcomers to the project feel that > it's a safe space. > > It's also worth reminding people that this is v1.0 of the CoC document. > We plan to revisit it in a year or so, and thereafter as needed, to > improve anything that's causing problems or not working well. > > regards, tom lane > I must admit that I'm impressed by the huge amount of contributions to this thread and, to be honest, it is the only one I have witnessed that would have deserved a CoC. I had a quick look at the proposal and it sounds to me like the team is trying to handle excesses - as long as no one complains, I would bet that they won't even chime in. One thing to keep in mind is this simple definition: "One person's freedom ends where another's begins" and all the work should go in this direction. We are all different, have different sensitivities, come from different cultures where we interpret words in a different way - it's a given, no way to escape. But we have in common the love of a great piece of software provided by a very active and efficient community. Why don't we focus on what unites us, instead of what creates divisions? Have a peaceful week-end Olivier
Re: move PartitionDispatchData definition to execPartition.c
On 2018-Sep-14, Tom Lane wrote: > Alvaro Herrera writes: > > On 2018-Jul-04, Amit Langote wrote: > >> I think we may have simply forgotten to do $subject in the following > >> commit. > > > We did. Pushed now, thanks. > > Erm, shouldn't that have been done in HEAD as well as v11? Hmm, the commit was there, but it was not pushed. Workflow mistake. Pushed now. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: PATCH: pgbench - option to build using ppoll() for larger connection counts
The author hasn't replied, but the attached seems to have cured the
bitrot so that it at least applies. Let's see what the cfbot makes of
it and then possibly fix any Windows issues.
The patch was not applying cleanly anymore for me, so here is a rebase of
your latest version.
Morever, ISTM that Tom's "why?" question has been answered: there are very
large systems out there with many processors, which are tested against
many connections, exceeding select limit.
I have turned back this patch to ready.
--
Fabien.diff --git a/configure b/configure
index c6a44a9078..0456064456 100755
--- a/configure
+++ b/configure
@@ -15060,7 +15060,7 @@ fi
LIBS_including_readline="$LIBS"
LIBS=`echo "$LIBS" | sed -e 's/-ledit//g' -e 's/-lreadline//g'`
-for ac_func in cbrt clock_gettime fdatasync getifaddrs getpeerucred getrlimit mbstowcs_l memmove poll posix_fallocate pstat pthread_is_threaded_np readlink setproctitle setproctitle_fast setsid shm_open symlink sync_file_range utime utimes wcstombs_l
+for ac_func in cbrt clock_gettime fdatasync getifaddrs getpeerucred getrlimit mbstowcs_l memmove poll posix_fallocate ppoll pstat pthread_is_threaded_np readlink setproctitle setproctitle_fast setsid shm_open symlink sync_file_range utime utimes wcstombs_l
do :
as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh`
ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var"
diff --git a/configure.in b/configure.in
index 3ada48b5f9..fceba79023 100644
--- a/configure.in
+++ b/configure.in
@@ -1544,7 +1544,7 @@ PGAC_FUNC_WCSTOMBS_L
LIBS_including_readline="$LIBS"
LIBS=`echo "$LIBS" | sed -e 's/-ledit//g' -e 's/-lreadline//g'`
-AC_CHECK_FUNCS([cbrt clock_gettime fdatasync getifaddrs getpeerucred getrlimit mbstowcs_l memmove poll posix_fallocate pstat pthread_is_threaded_np readlink setproctitle setproctitle_fast setsid shm_open symlink sync_file_range utime utimes wcstombs_l])
+AC_CHECK_FUNCS([cbrt clock_gettime fdatasync getifaddrs getpeerucred getrlimit mbstowcs_l memmove poll posix_fallocate ppoll pstat pthread_is_threaded_np readlink setproctitle setproctitle_fast setsid shm_open symlink sync_file_range utime utimes wcstombs_l])
AC_REPLACE_FUNCS(fseeko)
case $host_os in
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 41b756c089..3d378db714 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -45,9 +45,18 @@
#include
#include
#include
+#ifndef PGBENCH_USE_SELECT /* force use of select(2)? */
+#ifdef HAVE_PPOLL
+#define POLL_USING_PPOLL
+#include
+#endif
+#endif
+#ifndef POLL_USING_PPOLL
+#define POLL_USING_SELECT
#ifdef HAVE_SYS_SELECT_H
#include
#endif
+#endif
#ifdef HAVE_SYS_RESOURCE_H
#include /* for getrlimit */
@@ -92,13 +101,19 @@ static int pthread_join(pthread_t th, void **thread_return);
/
* some configurable parameters */
-
-/* max number of clients allowed */
+#ifdef POLL_USING_SELECT /* using select(2) */
+#define SOCKET_WAIT_METHOD "select"
+typedef fd_set socket_set;
#ifdef FD_SETSIZE
-#define MAXCLIENTS (FD_SETSIZE - 10)
+#define MAXCLIENTS (FD_SETSIZE - 10) /* system limited max number of clients allowed */
#else
-#define MAXCLIENTS 1024
+#define MAXCLIENTS 1024 /* max number of clients allowed */
#endif
+#else /* using ppoll(2) */
+#define SOCKET_WAIT_METHOD "ppoll"
+typedef struct pollfd socket_set;
+#define MAXCLIENTS -1 /* unlimited number of clients */
+#endif /* POLL_USING_SELECT */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
@@ -525,6 +540,13 @@ static void addScript(ParsedScript script);
static void *threadRun(void *arg);
static void setalarm(int seconds);
static void finishCon(CState *st);
+static socket_set *alloc_socket_set(int count);
+static bool error_on_socket(socket_set *sa, int idx, PGconn *con);
+static void free_socket_set(socket_set *sa);
+static bool ignore_socket(socket_set *sa, int idx, PGconn *con);
+static void clear_socket_set(socket_set *sa, int count);
+static void set_socket(socket_set *sa, int fd, int idx);
+static int wait_on_socket_set(socket_set *sa, int nstate, int maxsock, int64 usec);
/* callback functions for our flex lexer */
@@ -1143,6 +1165,7 @@ doConnect(void)
!have_password)
{
PQfinish(conn);
+ conn = NULL;
simple_prompt("Password: ", password, sizeof(password), false);
have_password = true;
new_pass = true;
@@ -4903,7 +4926,7 @@ main(int argc, char **argv)
case 'c':
benchmarking_option_set = true;
nclients = atoi(optarg);
-if (nclients <= 0 || nclients > MAXCLIENTS)
+if (nclients <= 0 || (MAXCLIENTS != -1 && nclients > MAXCLIENTS))
{
fprintf(stderr, "invalid number of clients: \"%s\"\n",
optarg);
@@ -5614,6 +5637,7 @@ threadRun(void *arg)
int64 next_report = last_report + (int64) progress * 100;
StatsData last,
aggs;
+ socket_set *sockets = alloc_socket_set(nstate);
/*
* Initialize throttling rat
Re: Defaulting to jit=on/off for v11
On 09/14/2018 08:18 PM, Peter Geoghegan wrote: On Fri, Sep 14, 2018 at 3:32 PM, Tom Lane wrote: I'd go with #2, personally. It does seem that the costing needs work, but it's not clear to me that we know what to change, so it's kinda late to propose #3 for v11. +1. I also favor option #2. + about 0.8. I hope we do get some good field testing if it, though. cheers andrew -- Andrew Dunstanhttps://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: Postgres 11 release notes
Hi, On 9/12/18 1:34 PM, Michael Banck wrote: > Hi, > > On Fri, May 11, 2018 at 11:08:52AM -0400, Bruce Momjian wrote: >> I have committed the first draft of the Postgres 11 release notes. I >> will add more markup soon. You can view the most current version here: >> >> http://momjian.us/pgsql_docs/release-11.html > > The first item of section 'E.1.3.10. Server Applications' is mine and > has this TODO (though maybe that should be better marked up?) text: > > |IS IT CLEAR FROM THE DOCS THAT THE REPLICATION SLOT IS NOT TEMPORARY? > > So I guess we need to decide on that before release. The current doc is > AFAIK: > > | This option causes the replication slot specified by the > | option --slot to be created before starting the > | backup. In this case, an error is raised if the slot already exists. > > I guess it does not hurt to have something like "to be (permanently) > created before starting", but what do others think? Is it clear enough? Attached are two patches. The first modifies the major improvements section to align with what we have discovered during the beta period + the general advocacy push. It also modifies the "last updated date" + narrows down a prospective release month. The second modifies/removes the questions/placeholders in the release notes, such as Michael's comment above (which I agreed with and added a one sentence explanation after). Thanks, Jonathan From 2733f0df599aeda48408a3d27b796e36fbdd6fc3 Mon Sep 17 00:00:00 2001 From: "Jonathan S. Katz" Date: Sat, 15 Sep 2018 10:43:44 -0400 Subject: [PATCH 1/2] Updates to major improvements section of release notes. --- doc/src/sgml/release-11.sgml | 47 ++-- 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/release-11.sgml b/doc/src/sgml/release-11.sgml index 684d34c091..9ed34e72e8 100644 --- a/doc/src/sgml/release-11.sgml +++ b/doc/src/sgml/release-11.sgml @@ -6,7 +6,7 @@ Release date: - 2018-??-?? (CURRENT AS OF 2018-07-30) + 2018-10-?? (CURRENT AS OF 2018-09-20) @@ -22,11 +22,11 @@ - Major improvements to partitioning: + Improvements to partitioning functionality, including: - Partitioning by a hash key + Partitioning by a hash key (a.k.a. "hash partitioning") @@ -37,9 +37,8 @@ - Improved SELECT query performance due to - enhanced partition elimination during query processing and - execution + Improved SELECT performance from enhanced partition + elimination strategies during query processing and execution @@ -48,29 +47,37 @@ KEY, indexes, and triggers on partitioned tables + + + Having a "default" partition for storing data that does not match a + partition key + + - Improvements to parallelism: + Improvements to parallelism, including: - Parallelized hash joins + B-tree indexes can now be built in parallel with + CREATE INDEX - Parallelized CREATE INDEX for B-tree indexes + Parallelized CREATE TABLE .. AS, + CREATE MATERIALIZED VIEW, and certain + queries using UNION - Parallelized CREATE TABLE .. AS, - CREATE MATERIALIZED VIEW, and certain - queries using UNION + Performance improvements for parallelized hash joins and parallelized + sequential scans @@ -79,14 +86,17 @@ - SQL stored procedures, with support for embedded transactions + SQL stored procedures that support embedded transactions. Stored + procedures can be created with + CREATE PROCEDURE and executed with + CALL - JIT compilation of some SQL code, including support for fast evaluation - of expressions + Introduction of just-in-time (JIT) compilation + during query execution @@ -99,6 +109,13 @@ + + + Covering indexes, which can be utilized using the + INCLUDE clause of CREATE INDEX + + + Many other useful performance improvements, including making -- 2.14.3 (Apple Git-98) From feb08e3747e5dc043e21b60e9e681f44a0562896 Mon Sep 17 00:00:00 2001 From: "Jonathan S. Katz" Date: Sat, 15 Sep 2018 11:01:35 -0400 Subject: [PATCH 2/2] Remove or modify placeholders for v11 release notes. --- doc/src/sgml/release-11.sgml | 21 ++--- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/release-11.sgml b/doc/src/sgml/release-11.sgml index 9ed34e72e8..e8af023140 100644 --- a/doc/src/sgml/release-11.sgml +++ b/doc/src/sgml/release-11.sgml @@
Re: [HACKERS] proposal: schema variables
> The code is more cleaner now, there are more tests, and documentation is > mostly complete. I am sorry - my English is not good. > New features: > > o ON COMMIT DROP and ON TRANSACTION END RESET -- remove temp variable on > commit, reset variable on transaction end (commit, rollback) > o LET var = DEFAULT -- reset specified variable > > fix some forgotten warnings and dependency issue few more tests Regards Pavel > Regards > > Pavel > > >> Regards, >> Dean >> > schema-variables-20180915-01.patch.gz Description: application/gzip
Re: Defaulting to jit=on/off for v11
On September 15, 2018 8:26:17 AM MDT, Andrew Dunstan wrote: > > >On 09/14/2018 08:18 PM, Peter Geoghegan wrote: >> On Fri, Sep 14, 2018 at 3:32 PM, Tom Lane wrote: >>> I'd go with #2, personally. It does seem that the costing needs >work, >>> but it's not clear to me that we know what to change, so it's kinda >>> late to propose #3 for v11. >> +1. I also favor option #2. >> > >+ about 0.8. I hope we do get some good field testing if it, though. Cool, sounds we have agreement. I'll try to come up with a patch. I'm [un]fortunately hiking till Wednesday, so I won't have an easy time to push a patch. I probably could push something tomorrow, but I'd have a hard time cleaning up if needed. Does anybody feel we should have that in Mondays release? Andres Andres -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Defaulting to jit=on/off for v11
Andres Freund writes: > Cool, sounds we have agreement. I'll try to come up with a patch. I'm > [un]fortunately hiking till Wednesday, so I won't have an easy time to push a > patch. I probably could push something tomorrow, but I'd have a hard time > cleaning up if needed. Does anybody feel we should have that in Mondays > release? Recent discussions among the release team are pointing to the idea that Monday's wrap could be 11rc1, not 11beta4, if we get this done beforehand. So I'd kind of like to see it done. Is there more that has to be done than switching the GUC's default value and adjusting docs? I could probably make it happen if there's not any hidden stuff to worry about. regards, tom lane
Re: Defaulting to jit=on/off for v11
Hi, On September 15, 2018 12:14:07 PM MDT, Tom Lane wrote: >Andres Freund writes: >> Cool, sounds we have agreement. I'll try to come up with a patch. >I'm [un]fortunately hiking till Wednesday, so I won't have an easy time >to push a patch. I probably could push something tomorrow, but I'd >have a hard time cleaning up if needed. Does anybody feel we should >have that in Mondays release? > >Recent discussions among the release team are pointing to the idea >that Monday's wrap could be 11rc1, not 11beta4, if we get this done >beforehand. So I'd kind of like to see it done. Yes, that's why I'm asking... >Is there more that has to be done than switching the GUC's default >value and adjusting docs? I could probably make it happen if there's >not any hidden stuff to worry about. No, that should be all. I'll adapt the llvm animals to force the GUC to on. I think it'd be good to add a paragraph to the release notes, mentioning that it has to be turned on. Thanks! Andres -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Defaulting to jit=on/off for v11
Andres Freund writes: > On September 15, 2018 12:14:07 PM MDT, Tom Lane wrote: >> Is there more that has to be done than switching the GUC's default >> value and adjusting docs? I could probably make it happen if there's >> not any hidden stuff to worry about. > No, that should be all. I'll adapt the llvm animals to force the GUC to on. > I think it'd be good to add a paragraph to the release notes, mentioning that > it has to be turned on. > Thanks! Roger, I'll have a go at it. regards, tom lane
Re: Defaulting to jit=on/off for v11
On 2018-09-15 14:19:55 -0400, Tom Lane wrote: > Andres Freund writes: > > On September 15, 2018 12:14:07 PM MDT, Tom Lane wrote: > >> Is there more that has to be done than switching the GUC's default > >> value and adjusting docs? I could probably make it happen if there's > >> not any hidden stuff to worry about. > > > No, that should be all. I'll adapt the llvm animals to force the GUC to > > on. I think it'd be good to add a paragraph to the release notes, > > mentioning that it has to be turned on. > > > Thanks! > > Roger, I'll have a go at it. Thanks! Just managed to log into the machine hosting my llvm testing buildfarm animals and set jit=1 for those. So we shouldn't loose too much coverage. I'll email the folks that have LLVM enabled in the BF, to also do so. Greetings, Andres Freund
Re: Defaulting to jit=on/off for v11
Hi, On 2018-09-15 11:41:58 -0700, Andres Freund wrote: > I'll email the folks that have LLVM enabled in the BF, to also do so. Done so. I'll try to check-in again tonight, but while likely, it's not certain there'll be reception. - Andres
Re: Should contrib modules install .h files?
> "Andrew" == Andrew Gierth writes:
Andrew> Here's a patch that fixes (not necessarily in the best way) the
Andrew> PGXS builds of all the contrib/*_pl{perl,python} modules.
Andrew> Oh, obviously this patch doesn't fix the windows Install.pm
Andrew> yet, but that'd be easier to do after finalizing the list of
Andrew> include files to install.
So while looking into this, I found that to the best of my ability to
determine, the windows Install.pm has _never_ installed the plperl or
plpython header files. That's more than I'm in a position to fix, having
no windows box to test with.
So what I propose to do is to commit a cleaned-up version of the patch
posted above, with these changes:
- install all the plpy_*.h headers, not just a few; I know of no reason
to exclude any of them, and in the absence of feedback it seems
better to install them all;
- tidy up the makefile variables in the python transform modules to
remove some duplication
- fix Mkvcbuild.pm to account for the changes
This will allow hstore_plperl etc. to stand as examples of how to do
cross-module #includes both in and out of tree.
I'll post the patch for review shortly.
--
Andrew (irc:RhodiumToad)
Re: Index Skip Scan
> On Thu, 13 Sep 2018 at 21:36, Alexander Kuzmenkov > wrote: > > El 13/09/18 a las 18:39, Jesper Pedersen escribió: > >> I think we can improve this, >> and the skip scan can be strictly faster than index scan regardless of >> the data. As a first approximation, imagine that we somehow skipped >> equal tuples inside _bt_next instead of sending them to the parent >> Unique node. This would already be marginally faster than Unique + Index >> scan. A more practical implementation would be to remember our position >> in tree (that is, BTStack returned by _bt_search) and use it to skip >> pages in bulk. This looks straightforward to implement for a tree that >> does not change, but I'm not sure how to make it work with concurrent >> modifications. Still, this looks a worthwhile direction to me, because >> if we have a strictly faster skip scan, we can just use it always and >> not worry about our unreliable statistics. What do you think? >> > > This is something to look at -- maybe there is a way to use > btpo_next/btpo_prev instead/too in order to speed things up. Atm we just > have the scan key in BTScanOpaqueData. I'll take a look after my > upcoming vacation; feel free to contribute those changes in the meantime > of course. But having this logic inside _bt_next means that it will make a non-skip index only scan a bit slower, am I right? Probably it would be easier and more straightforward to go with the idea of dynamic fallback then. The first naive implementation that I came up with is to wrap an index scan node into a unique, and remember estimated number of groups into IndexOnlyScanState, so that we can check if we performed much more skips than expected. With this approach index skip scan will work a bit slower than in the original patch in case if ndistinct is correct (because a unique node will recheck rows we returned), and fallback to unique + index only scan in case if planner has underestimated ndistinct. index-skip-fallback.patch Description: Binary data
Re: Should contrib modules install .h files?
> "Andrew" == Andrew Gierth writes: Andrew> So what I propose to do is to commit a cleaned-up version of Andrew> the patch posted above, with these changes: Andrew> - install all the plpy_*.h headers, not just a few; I know of no reason Andrew>to exclude any of them, and in the absence of feedback it seems Andrew>better to install them all; Andrew> - tidy up the makefile variables in the python transform modules to Andrew>remove some duplication Andrew> - fix Mkvcbuild.pm to account for the changes Andrew> This will allow hstore_plperl etc. to stand as examples of how Andrew> to do cross-module #includes both in and out of tree. Andrew> I'll post the patch for review shortly. And here it is. -- Andrew (irc:RhodiumToad) diff --git a/contrib/hstore_plperl/Makefile b/contrib/hstore_plperl/Makefile index f63cba2745..32ecaa43cb 100644 --- a/contrib/hstore_plperl/Makefile +++ b/contrib/hstore_plperl/Makefile @@ -4,7 +4,6 @@ MODULE_big = hstore_plperl OBJS = hstore_plperl.o $(WIN32RES) PGFILEDESC = "hstore_plperl - hstore transform for plperl" -PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plperl -I$(top_srcdir)/contrib/hstore EXTENSION = hstore_plperl hstore_plperlu DATA = hstore_plperl--1.0.sql hstore_plperlu--1.0.sql @@ -13,10 +12,12 @@ REGRESS = hstore_plperl hstore_plperlu create_transform EXTRA_INSTALL = contrib/hstore ifdef USE_PGXS +PG_CPPFLAGS = -I$(includedir_server)/extension PG_CONFIG = pg_config PGXS := $(shell $(PG_CONFIG) --pgxs) include $(PGXS) else +PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plperl -I$(top_srcdir)/contrib subdir = contrib/hstore_plperl top_builddir = ../.. include $(top_builddir)/src/Makefile.global diff --git a/contrib/hstore_plperl/hstore_plperl.c b/contrib/hstore_plperl/hstore_plperl.c index c09bd38d09..61b5557421 100644 --- a/contrib/hstore_plperl/hstore_plperl.c +++ b/contrib/hstore_plperl/hstore_plperl.c @@ -5,7 +5,7 @@ #include "fmgr.h" #include "plperl.h" #include "plperl_helpers.h" -#include "hstore.h" +#include "hstore/hstore.h" PG_MODULE_MAGIC; diff --git a/contrib/hstore_plpython/Makefile b/contrib/hstore_plpython/Makefile index b81735ab91..6877e7a072 100644 --- a/contrib/hstore_plpython/Makefile +++ b/contrib/hstore_plpython/Makefile @@ -4,19 +4,21 @@ MODULE_big = hstore_plpython$(python_majorversion) OBJS = hstore_plpython.o $(WIN32RES) PGFILEDESC = "hstore_plpython - hstore transform for plpython" -PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plpython $(python_includespec) -I$(top_srcdir)/contrib/hstore -DPLPYTHON_LIBNAME='"plpython$(python_majorversion)"' - EXTENSION = hstore_plpythonu hstore_plpython2u hstore_plpython3u DATA = hstore_plpythonu--1.0.sql hstore_plpython2u--1.0.sql hstore_plpython3u--1.0.sql REGRESS = hstore_plpython REGRESS_PLPYTHON3_MANGLE := $(REGRESS) +PG_CPPFLAGS = $(python_includespec) -DPLPYTHON_LIBNAME='"plpython$(python_majorversion)"' + ifdef USE_PGXS +PG_CPPFLAGS += -I$(includedir_server)/extension PG_CONFIG = pg_config PGXS := $(shell $(PG_CONFIG) --pgxs) include $(PGXS) else +PG_CPPFLAGS += -I$(top_srcdir)/src/pl/plpython -I$(top_srcdir)/contrib subdir = contrib/hstore_plpython top_builddir = ../.. include $(top_builddir)/src/Makefile.global diff --git a/contrib/hstore_plpython/hstore_plpython.c b/contrib/hstore_plpython/hstore_plpython.c index 218e6612b1..2f24090ff3 100644 --- a/contrib/hstore_plpython/hstore_plpython.c +++ b/contrib/hstore_plpython/hstore_plpython.c @@ -3,7 +3,7 @@ #include "fmgr.h" #include "plpython.h" #include "plpy_typeio.h" -#include "hstore.h" +#include "hstore/hstore.h" PG_MODULE_MAGIC; diff --git a/contrib/ltree_plpython/Makefile b/contrib/ltree_plpython/Makefile index 7e988c7993..ce2c0cd2e2 100644 --- a/contrib/ltree_plpython/Makefile +++ b/contrib/ltree_plpython/Makefile @@ -4,19 +4,21 @@ MODULE_big = ltree_plpython$(python_majorversion) OBJS = ltree_plpython.o $(WIN32RES) PGFILEDESC = "ltree_plpython - ltree transform for plpython" -PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plpython $(python_includespec) -I$(top_srcdir)/contrib/ltree -DPLPYTHON_LIBNAME='"plpython$(python_majorversion)"' - EXTENSION = ltree_plpythonu ltree_plpython2u ltree_plpython3u DATA = ltree_plpythonu--1.0.sql ltree_plpython2u--1.0.sql ltree_plpython3u--1.0.sql REGRESS = ltree_plpython REGRESS_PLPYTHON3_MANGLE := $(REGRESS) +PG_CPPFLAGS = $(python_includespec) -DPLPYTHON_LIBNAME='"plpython$(python_majorversion)"' + ifdef USE_PGXS +PG_CPPFLAGS += -I$(includedir_server)/extension PG_CONFIG = pg_config PGXS := $(shell $(PG_CONFIG) --pgxs) include $(PGXS) else +PG_CPPFLAGS += -I$(top_srcdir)/src/pl/plpython -I$(top_srcdir)/contrib subdir = contrib/ltree_plpython top_builddir = ../.. include $(top_builddir)/src/Makefile.global diff --git a/contrib/ltree_plpython/ltree_plpython.c b/contrib/ltree_plpython/ltree_plpython.c index e88636a0a9..b254aa558d 100644 --- a/contrib/ltree_plpython/ltree_plpython.c +++ b/contrib/ltree_plpython/ltree_plpython.c @@ -2,7
Re: Something fishy happening on frogmouth
On Wed, Oct 30, 2013 at 09:07:43AM -0400, Robert Haas wrote:
> On Wed, Oct 30, 2013 at 8:47 AM, Andres Freund wrote:
> > On 2013-10-30 08:45:03 -0400, Robert Haas wrote:
> >> If I'm reading this correctly, the last three runs on frogmouth have
> >> all failed, and all of them have failed with a complaint about,
> >> specifically, Global/PostgreSQL.851401618. Now, that really shouldn't
> >> be happening, because the code to choose that number looks like this:
> >>
> >> dsm_control_handle = random();
> > Could it be that we haven't primed the random number generator with the
> > time or something like that yet?
>
> Yeah, I think that's probably what it is.
I experienced a variation of this, namely a RHEL 7 system where initdb always
says "selecting dynamic shared memory implementation ... sysv". Each initdb
is rejecting posix shm by probing the same ten segments:
$ strace initdb -D scratch 2>&1 | grep /dev/shm/P
open("/dev/shm/PostgreSQL.1804289383",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.846930886",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.1681692777",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.1714636915",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.1957747793",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.424238335",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.719885386",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.1649760492",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.596516649",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
open("/dev/shm/PostgreSQL.1189641421",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EEXIST (File exists)
Regular postmaster runs choose a random segment, but initdb, bootstrap
postgres, and single-user postgres all start with the same segment. These
segments are months old. Perhaps I was testing something that caused a
bootstrap postgres to crash. After ten such crashes, future initdb runs
considered posix shm unusable.
> There's PostmasterRandom()
> to initialize the random-number generator on first use, but that
> doesn't help if some other module calls random(). I wonder if we
> ought to just get rid of PostmasterRandom() and instead have the
> postmaster run that initialization code very early in startup.
Usually, the first srandom() call happens early in PostmasterMain(). I plan
to add one to InitStandaloneProcess(), which substitutes for several tasks
otherwise done in PostmasterMain(). That seems like a good thing even if DSM
weren't in the picture. Also, initdb needs an srandom() somewhere;
choose_dsm_implementation() itself seems fine. Attached. With this, "make
-j20 check-world" selected posix shm and passed even when I forced DSM
creation to fail on unseeded random():
--- a/src/backend/storage/ipc/dsm_impl.c
+++ b/src/backend/storage/ipc/dsm_impl.c
@@ -249,2 +249,5 @@ dsm_impl_posix(dsm_op op, dsm_handle handle, Size
request_size,
+ if (handle == 1804289383)
+ elog(ERROR, "generated handle with no randomness");
+
snprintf(name, 64, "/PostgreSQL.%u", handle);
diff --git a/src/backend/utils/init/miscinit.c
b/src/backend/utils/init/miscinit.c
index 865119d..f003831 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -325,6 +325,8 @@ InitStandaloneProcess(const char *argv0)
MyStartTime = time(NULL); /* set our start time in case we call
elog */
+ srandom((unsigned int) (MyProcPid ^ MyStartTime));
+
/* Initialize process-local latch support */
InitializeLatchSupport();
MyLatch = &LocalLatchData;
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 32746c7..83f4b0b 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -922,6 +922,8 @@ choose_dsm_implementation(void)
#ifdef HAVE_SHM_OPEN
int ntries = 10;
+ srandom((unsigned int) (getpid() ^ time(NULL)));
+
while (ntries > 0)
{
uint32 handle;
Re: Something fishy happening on frogmouth
Noah Misch writes: > Usually, the first srandom() call happens early in PostmasterMain(). I plan > to add one to InitStandaloneProcess(), which substitutes for several tasks > otherwise done in PostmasterMain(). That seems like a good thing even if DSM > weren't in the picture. Also, initdb needs an srandom() somewhere; > choose_dsm_implementation() itself seems fine. Attached. +1, but some comments would be good. regards, tom lane
Re: Code of Conduct plan
On 9/14/18 11:13 AM, Robert Haas wrote: On Fri, Sep 14, 2018 at 11:10 AM, Dave Page wrote: That wording has been in the published draft for 18 months, and noone objected to it that I'm aware of. There will always be people who don't like some of the wording, much as there are often people who disagree with the way a patch to the code is written. Sooner or later though, the general consensus prevails and we have to move on, otherwise nothing will ever get completed. It's not clear to me that there IS a general consensus here. It looks to me like the unelected core team got together and decided to impose a vaguely-worded code of conduct on a vaguely-defined group of people covering not only their work on PostgreSQL but also their entire life. It is not difficult to imagine that someone's private life might include "behavior that may bring the PostgreSQL project into disrepute." However, I also don't think it matters very much. The Code of Conduct Committee is going to consist of small number of people -- at least four, perhaps a few more. But there are hundreds of people involved on the PostgreSQL mailing lists, maybe thousands. If the Code of Conduct Committee, or the core team, believes that it can impose on a very large group of people, all of whom are volunteers, some set of rules with which they don't agree, it's probably going to find out pretty quickly that it is mistaken. If people from that large group get banned for behavior which is perceived by other members of that large group to be legitimate, then there will be a ferocious backlash. Nobody wants to see people who are willing to contribute driven away from the project, and anyone we drive away without a really good reason will find some other project that welcomes their participation. So the only thing that the Code of Conduct Committee is likely to be able to do in practice is admonish people to be nicer (which is probably a good thing) and punish really egregious conduct, especially when committed by people who aren't involved enough that their absence will be keenly felt. In practice, therefore, democracy is going to win out. That's both good and bad. It's good because nobody wants a CoC witch-hunt, and it's bad because there's probably some behavior which legitimately deserves censure and will escape it. +1 -- Adrian Klaver adrian.kla...@aklaver.com
Re: Code of Conduct plan
On 15/09/18 08:17, Tom Lane wrote: Yeah, this. The PG community is mostly nice people, AFAICT. I'll be astonished (and worried) if the CoC committee finds much to do. We're implementing this mostly to make newcomers to the project feel that it's a safe space. Agreed. However I think the all-of-life clause gives an open door to potential less than well intentioned new members joining up to extend a SJW agenda. So in fact the unintended consequence of this may be a *less* safe place for some existing members - unless all of their social media utterances are agreeable to the angry militant left. It's also worth reminding people that this is v1.0 of the CoC document. We plan to revisit it in a year or so, and thereafter as needed, to improve anything that's causing problems or not working well. +1, At least this means we can address the above if it emerges as a problem regards Mark regards, tom lane
