Re: pg_validatebackup -> pg_verifybackup?

2020-04-12 Thread Kyotaro Horiguchi
I recklessly join the discussion about naming. At Sun, 12 Apr 2020 17:29:55 +0200, Magnus Hagander wrote in > Agreed, pgsql would certainly be better than pg_ctl. I like pgsql. And if we are going to join to the THREE-LETTERS CONTROLLER COMMANDS ALLIANCE, pgc (pg controller) might be a candida

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-12 Thread Kyotaro Horiguchi
At Mon, 13 Apr 2020 15:31:01 +0900 (JST), Kyotaro Horiguchi wrote in > At Sat, 11 Apr 2020 18:30:30 -0400, Tom Lane wrote in > > The point that I was trying to make originally is that it seems quite > > insane to imagine that a walsender's sync_standby_priority value is > > somehow more stable

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-12 Thread Kyotaro Horiguchi
At Sat, 11 Apr 2020 18:30:30 -0400, Tom Lane wrote in > Masahiko Sawada writes: > > On Tue, 31 Mar 2020 at 23:16, Masahiko Sawada > > wrote: > >>> Therefore, the band-aid fix seems to be to set the lowest priority to > >>> very large number at the beginning of SyncRepGetSyncStandbysPriority().

Re: Corruption during WAL replay

2020-04-12 Thread Masahiko Sawada
On Sat, 11 Apr 2020 at 09:00, Teja Mupparti wrote: > > Thanks Andres and Kyotaro for the quick review. I have fixed the typos and > also included the critical section (emulated it with try-catch block since > palloc()s are causing issues in the truncate code). This time I used git > format-pat

Re: WAL usage calculation patch

2020-04-12 Thread Amit Kapila
On Sat, Apr 11, 2020 at 6:55 PM Julien Rouhaud wrote: > > On Fri, Apr 10, 2020 at 9:37 PM Julien Rouhaud wrote: > > > > On Fri, Apr 10, 2020 at 8:17 AM Amit Kapila wrote: > > > Would you like to send a consolidated patch that includes Euler's > > > suggestion and Justin's patch (by making change

Re: Implementing Incremental View Maintenance

2020-04-12 Thread Yugo NAGATA
On Fri, 10 Apr 2020 23:26:58 +0900 Yugo NAGATA wrote: > Hi, > > Attached is the latest patch (v15) to add support for Incremental Materialized > View Maintenance (IVM). It is possible to apply to current latest master > branch. I found a mistake of splitting patch, so I attached the fixed pat

Re: doc review for parallel vacuum

2020-04-12 Thread Amit Kapila
On Fri, Apr 10, 2020 at 7:16 PM Justin Pryzby wrote: > > Also, this part still doesn't read well: > > -* amvacuumcleanup to the DSM segment if it's the first time to get > it? > -* from them? because they? allocate it locally and it's possible > that an > -* index will be

Re: snapshot too old issues, first around wraparound and then more.

2020-04-12 Thread Andres Freund
Hi, On 2020-04-13 14:58:34 +1200, Thomas Munro wrote: > On Fri, Apr 3, 2020 at 2:22 PM Peter Geoghegan wrote: > > I think that it's worth considering whether or not there are a > > significant number of "snapshot too old" users that rarely or never > > rely on old snapshots used by new queries. K

Re: pg_basebackup, manifests and backends older than ~12

2020-04-12 Thread Michael Paquier
On Mon, Apr 13, 2020 at 11:52:51AM +0900, Kyotaro Horiguchi wrote: > Since I'm not sure about the work flow that contains taking a > basebackup from a server of a different version, I'm not sure which is > better between silently disabling and erroring out. However, it seems > to me, the option fo

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-12 Thread Amit Kapila
On Sun, Apr 12, 2020 at 4:03 AM Justin Pryzby wrote: > > On Sat, Mar 28, 2020 at 04:17:21PM +0100, Julien Rouhaud wrote: > > On Sat, Mar 28, 2020 at 02:38:27PM +0100, Julien Rouhaud wrote: > > > On Sat, Mar 28, 2020 at 04:14:04PM +0530, Amit Kapila wrote: > > > > > > > > I see some basic problems

Re: backup manifests

2020-04-12 Thread Michael Paquier
On Mon, Apr 13, 2020 at 11:09:34AM +0900, Fujii Masao wrote: > - while ((c = getopt_long(argc, argv, > "CD:F:r:RS:T:X:l:nNzZ:d:c:h:p:U:s:wWkvP", > + while ((c = getopt_long(argc, argv, > "CD:F:r:RS:T:X:l:nNzZ:d:c:h:p:U:s:wWkvPm:", > > "m:" seems unnecessary, so should be removed? > Patch

Re: snapshot too old issues, first around wraparound and then more.

2020-04-12 Thread Thomas Munro
On Fri, Apr 3, 2020 at 2:22 PM Peter Geoghegan wrote: > I think that it's worth considering whether or not there are a > significant number of "snapshot too old" users that rarely or never > rely on old snapshots used by new queries. Kevin said that this > happens "in some cases", but how many cas

Re: pg_basebackup, manifests and backends older than ~12

2020-04-12 Thread Kyotaro Horiguchi
At Mon, 13 Apr 2020 09:56:02 +0900, Michael Paquier wrote in > On Sun, Apr 12, 2020 at 08:08:17AM +0900, Michael Paquier wrote: > > Exactly. My point is exactly that. The current code would force > > users maintaining scripts with pg_basebackup to use --no-manifest if > > such a script runs wi

RE: pg_validatebackup -> pg_verifybackup?

2020-04-12 Thread Shinoda, Noriyoshi (PN Japan A&PS Delivery)
Hi, Sorry this email is not a discussion about word selection. Since part of the manual had left pg_validatebackup in commit dbc60c5593f26dc777a3be032bff4fb4eab1ddd1. I've attached a patch to fix this. Regards, Noriyoshi Shinoda -Original Message- From: Tom Lane [mailto:t...@sss.pgh.pa.

Re: backup manifests

2020-04-12 Thread Fujii Masao
On 2020/04/09 23:06, Fujii Masao wrote: On 2020/04/09 2:35, Robert Haas wrote: On Wed, Apr 8, 2020 at 1:15 AM Fujii Masao wrote: When there is a backup_manifest in the database cluster, it's included in the backup even when --no-manifest is specified. ISTM that this is problematic because

Re: where should I stick that backup?

2020-04-12 Thread Stephen Frost
Greetings, Answering both in one since they're largely the same. * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Apr 10, 2020 at 10:54:10AM -0400, Stephen Frost wrote: > > * Robert Haas (robertmh...@gmail.com) wrote: > > > On Thu, Apr 9, 2020 at 6:44 PM Bruce Momjian wrote: > > > > Good poin

Re: pg_basebackup, manifests and backends older than ~12

2020-04-12 Thread Michael Paquier
On Sun, Apr 12, 2020 at 08:08:17AM +0900, Michael Paquier wrote: > Exactly. My point is exactly that. The current code would force > users maintaining scripts with pg_basebackup to use --no-manifest if > such a script runs with older versions of Postgres, but we should > encourage users not do to

Re: sqlsmith crash incremental sort

2020-04-12 Thread Justin Pryzby
On Mon, Apr 13, 2020 at 02:09:43AM +0200, Tomas Vondra wrote: > Justin, can you try if this resolves the crashes or if there's something > else going on? With your patch, this no longer crashes: |explain SELECT * FROM information_schema.transforms AS ref_1 RIGHT JOIN (SELECT 1 FROM pg_catalog.pg_

Re: where should I stick that backup?

2020-04-12 Thread Andres Freund
Hi, On 2020-04-12 20:02:50 -0400, Robert Haas wrote: > On Sun, Apr 12, 2020 at 3:17 PM Andres Freund wrote: > > A huge advantage of a scheme like this would be that it wouldn't have to > > be specific to pg_basebackup. It could just as well work directly on the > > server, avoiding an unnecesary

Re: where should I stick that backup?

2020-04-12 Thread Stephen Frost
Greetings, * David Steele (da...@pgmasters.net) wrote: > On 4/12/20 6:37 PM, Andres Freund wrote: > >On 2020-04-12 17:57:05 -0400, David Steele wrote: > >>On 4/12/20 3:17 PM, Andres Freund wrote: > >>>There's various ways we could address the issue for how the subcommand > >>>can access the file d

Re: sqlsmith crash incremental sort

2020-04-12 Thread Tomas Vondra
On Sun, Apr 12, 2020 at 12:44:45AM +0200, Tomas Vondra wrote: Hi, I've looked into this a bit, and at first I thought that maybe the issue is in how cost_incremental_sort picks the EC members. It simply does this: EquivalenceMember *member = (EquivalenceMember *) linitial(key->pk_e

Re: where should I stick that backup?

2020-04-12 Thread Robert Haas
On Sun, Apr 12, 2020 at 3:17 PM Andres Freund wrote: > A huge advantage of a scheme like this would be that it wouldn't have to > be specific to pg_basebackup. It could just as well work directly on the > server, avoiding an unnecesary loop through the network. Which > e.g. could integrate with fi

Re: where should I stick that backup?

2020-04-12 Thread David Steele
e advantages though. We use JSON, but each protocol request/response is linefeed-delimited. So for example here's what it looks like when the main process requests a local process to backup a specific file: {"{"cmd":"backupFile","param":["b

Re: cleaning perl code

2020-04-12 Thread David Steele
On 4/12/20 6:24 PM, Andrew Dunstan wrote: On 4/12/20 4:12 PM, David Steele wrote: Just in case it is useful, I have attached our old policy file with exceptions and excuses (when we had one). That's a pretty short list for --brutal, well done. I agree there is value in keeping documented the

Re: where should I stick that backup?

2020-04-12 Thread Andres Freund
es, we'd have to deal with those files vanishing, etc. > > [2] yes, I already hear json. A line deliminated format would have some > > advantages though. > > We use JSON, but each protocol request/response is linefeed-delimited. So > for example here's what it looks lik

Re: cleaning perl code

2020-04-12 Thread Andrew Dunstan
On 4/12/20 4:12 PM, David Steele wrote: > On 4/12/20 3:22 PM, Robert Haas wrote: >> On Sat, Apr 11, 2020 at 11:15 AM Tom Lane wrote: >>> Noah Misch writes: In summary, among those warnings, I see non-negative value in "Code before warnings are enabled" only.  While we're changing

Re: where should I stick that backup?

2020-04-12 Thread David Steele
quest/response is linefeed-delimited. So for example here's what it looks like when the main process requests a local process to backup a specific file: {"{"cmd":"backupFile","param":["base/32768/33001",true,65536,null,true,0,"pg_data/base/32

Re: doc review for v13

2020-04-12 Thread Justin Pryzby
Added a few more. And rebased on top of dbc60c5593f26dc777a3be032bff4fb4eab1ddd1 -- Justin >From 37b796862eb8c08dbe0d1a6947a8212d1c515491 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sun, 29 Mar 2020 19:51:32 -0500 Subject: [PATCH v3 01/20] doc: psql opclass/opfamily commit b0b5e20cd8d1a5

Re: where should I stick that backup?

2020-04-12 Thread David Steele
On 4/12/20 11:04 AM, Robert Haas wrote: On Sun, Apr 12, 2020 at 10:09 AM Magnus Hagander wrote: There are certainly cases for it. It might not be they have to be the same connection, but still be the same session, meaning before the first time you perform some step of authentication, get a to

Re: cleaning perl code

2020-04-12 Thread David Steele
On 4/12/20 3:22 PM, Robert Haas wrote: On Sat, Apr 11, 2020 at 11:15 AM Tom Lane wrote: Noah Misch writes: In summary, among those warnings, I see non-negative value in "Code before warnings are enabled" only. While we're changing this, I propose removing Subroutines::RequireFinalReturn. I

Re: pg_validatebackup -> pg_verifybackup?

2020-04-12 Thread Tom Lane
Andres Freund writes: > On 2020-04-12 11:21:50 -0400, Robert Haas wrote: >> We could also have an alternate name, like pgsql, and make 'pg' a >> symlink to it that packagers can choose to omit. > We could even name the non-abbreviated binary postgres :). I shudder to imagine the confusion that w

Re: pg_validatebackup -> pg_verifybackup?

2020-04-12 Thread Andres Freund
On 2020-04-12 11:21:50 -0400, Robert Haas wrote: > We could also have an alternate name, like pgsql, and make 'pg' a > symlink to it that packagers can choose to omit. We could even name the non-abbreviated binary postgres :). Sure, that'd cause a bit more trouble upgrading for people that script

Re: pg_validatebackup -> pg_verifybackup?

2020-04-12 Thread Andres Freund
Hi, On 2020-04-12 10:57:59 -0400, Tom Lane wrote: > Yeah, I'd be happier with "pg" than "pg_ctl" as well. But it's so > short that I wonder if some other software has already adopted it. FWIW, Debian unstable does not have a 'pg' binary. There's a few modules in various languages called 'pg', bu

Re: where should I stick that backup?

2020-04-12 Thread Andres Freund
Hi, On 2020-04-12 11:04:46 -0400, Robert Haas wrote: > I would expect that we would want to provide a flexible way for a > target or filter to be passed options from the pg_basebackup command > line. So one might for example write this: > > pg_basebackup --filter='lz4 -9' --filter='encrypt-with-r

Re: cleaning perl code

2020-04-12 Thread Robert Haas
On Sat, Apr 11, 2020 at 11:15 AM Tom Lane wrote: > Noah Misch writes: > > In summary, among those warnings, I see non-negative value in "Code before > > warnings are enabled" only. While we're changing this, I propose removing > > Subroutines::RequireFinalReturn. > > If it's possible to turn off

Re: where should I stick that backup?

2020-04-12 Thread Andres Freund
Hi, On 2020-04-11 16:22:09 -0400, Robert Haas wrote: > On Fri, Apr 10, 2020 at 3:38 PM Andres Freund wrote: > > Wouldn't there be state like a S3/ssh/https/... connection? And perhaps > > a 'backup_id' in the backup metadata DB that'd one would want to update > > at the end? > > Good question. I

Re: execExprInterp() questions / How to improve scalar array op expr eval?

2020-04-12 Thread Andres Freund
Hi, On 2020-04-12 08:55:44 -0400, James Coleman wrote: > On Sat, Apr 11, 2020 at 5:32 PM Andres Freund wrote: > > On 2020-04-11 15:53:11 -0400, James Coleman wrote: > > > On Sat, Apr 11, 2020 at 2:01 PM Andres Freund wrote: > > > > > - If not, is there a way in that framework to know if the arra

Re: Add "-Wimplicit-fallthrough" to default flags (was Re: pgsql: Support FETCH FIRST WITH TIES)

2020-04-12 Thread Julien Rouhaud
On Sun, Apr 12, 2020 at 5:25 PM Mark Dilger wrote: > > > On Apr 12, 2020, at 7:55 AM, Tom Lane wrote: > > > > Poking around in the archives, it seems like the only previous formal > > proposal to add -Wimplicit-fallthrough was in the context of a much > > more aggressive proposal to make a lot of

Re: pg_validatebackup -> pg_verifybackup?

2020-04-12 Thread Magnus Hagander
On Sun, Apr 12, 2020 at 5:22 PM Robert Haas wrote: > On Sun, Apr 12, 2020 at 11:02 AM Magnus Hagander > wrote: > > There's https://en.wikipedia.org/wiki/Pg_(Unix). > > > > So it's been removed from posix, but not unlikely to be around. For > example, I see it on a server with Debian 9 (Stretch)

Re: Add "-Wimplicit-fallthrough" to default flags (was Re: pgsql: Support FETCH FIRST WITH TIES)

2020-04-12 Thread Mark Dilger
> On Apr 12, 2020, at 7:55 AM, Tom Lane wrote: > > Poking around in the archives, it seems like the only previous formal > proposal to add -Wimplicit-fallthrough was in the context of a much > more aggressive proposal to make a lot of non-Wall warnings into > errors [1], which people did not l

Re: pg_validatebackup -> pg_verifybackup?

2020-04-12 Thread Robert Haas
On Sun, Apr 12, 2020 at 11:02 AM Magnus Hagander wrote: > There's https://en.wikipedia.org/wiki/Pg_(Unix). > > So it's been removed from posix, but not unlikely to be around. For example, > I see it on a server with Debian 9 (Stretch) or Ubuntu 16.04 which is still > well in support (but not on

Re: where should I stick that backup?

2020-04-12 Thread Robert Haas
On Sun, Apr 12, 2020 at 10:09 AM Magnus Hagander wrote: > There are certainly cases for it. It might not be they have to be the same > connection, but still be the same session, meaning before the first time you > perform some step of authentication, get a token, and then use that for all > the

Re: pg_validatebackup -> pg_verifybackup?

2020-04-12 Thread Magnus Hagander
On Sun, Apr 12, 2020 at 4:58 PM Tom Lane wrote: > Robert Haas writes: > > On Sat, Apr 11, 2020 at 5:51 PM Alvaro Herrera > wrote: > >> I like this idea so much that I already proposed it in the past[1], so > +1. > >> > >> [1] https://postgr.es/m/20160826202911.GA320593@alvherre.pgsql > > > Hey,

Re: pg_validatebackup -> pg_verifybackup?

2020-04-12 Thread Tom Lane
Robert Haas writes: > On Sat, Apr 11, 2020 at 5:51 PM Alvaro Herrera > wrote: >> I like this idea so much that I already proposed it in the past[1], so +1. >> >> [1] https://postgr.es/m/20160826202911.GA320593@alvherre.pgsql > Hey, look at that. I think I had some vague recollection of a prior

Re: Add "-Wimplicit-fallthrough" to default flags (was Re: pgsql: Support FETCH FIRST WITH TIES)

2020-04-12 Thread Tom Lane
Julien Rouhaud writes: > On Sat, Apr 11, 2020 at 02:47:34PM -0400, Tom Lane wrote: >> Yeah, assorted buildfarm animals are mentioning that too. I wonder >> if we should add that to the default warning options selected by >> configure? I don't remember if that's been discussed before. > I'm all

Re: Support for DATETIMEOFFSET

2020-04-12 Thread Tom Lane
Jeremy Morton writes: > At just about every development shop I've worked for, I've seen > developers use methods to get a local DateTime - both in the DB and in > the code - such as DateTime.Now, and throw it at a DateTime field. > Heck, even I've occasionally forgotten to use .UtcNow. With >

Re: pg_validatebackup -> pg_verifybackup?

2020-04-12 Thread Robert Haas
On Sat, Apr 11, 2020 at 5:51 PM Alvaro Herrera wrote: > I like this idea so much that I already proposed it in the past[1], so +1. > > [1] https://postgr.es/m/20160826202911.GA320593@alvherre.pgsql Hey, look at that. I think I had some vague recollection of a prior proposal, but I couldn't rememb

Re: where should I stick that backup?

2020-04-12 Thread Magnus Hagander
On Sat, Apr 11, 2020 at 10:22 PM Robert Haas wrote: > On Fri, Apr 10, 2020 at 3:38 PM Andres Freund wrote: > > Wouldn't there be state like a S3/ssh/https/... connection? And perhaps > > a 'backup_id' in the backup metadata DB that'd one would want to update > > at the end? > > Good question. I

Re: execExprInterp() questions / How to improve scalar array op expr eval?

2020-04-12 Thread James Coleman
On Sat, Apr 11, 2020 at 5:32 PM Andres Freund wrote: > > Hi, > > On 2020-04-11 15:53:11 -0400, James Coleman wrote: > > On Sat, Apr 11, 2020 at 2:01 PM Andres Freund wrote: > > > > - If not, is there a way in that framework to know if the array expr > > > > has stayed the same through multiple ev

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2020-04-12 Thread Fabien COELHO
Hello Justin, About v15, seen as one patch. Patches serie applies cleanly, compiles, "make check" ok. Documentation: - indent documentation text around 80 cols, as done around? - indent SQL example for readability and capitalize keywords (pg_ls_dir_metadata) - "For each file in a direct

Re: Support for DATETIMEOFFSET

2020-04-12 Thread Jeremy Morton
Neil wrote: On Apr 10, 2020, at 8:19 AM, Jeremy Morton wrote: Oh well. Guess I keep using SQL Server then. datetimeoffset makes it impossible for developers to make the mistake of forgetting to use UTC instead of local datetime, and for that reason alone it makes it invaluable in my opin

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-12 Thread Julien Rouhaud
Le dim. 12 avr. 2020 à 00:33, Justin Pryzby a écrit : > On Sat, Mar 28, 2020 at 04:17:21PM +0100, Julien Rouhaud wrote: > > > > Just to be sure I did a quick test with pg_stat_statements behavior using > > parallel/non-parallel CREATE INDEX and VACUUM, and unsurprisingly buffer > usage > > doesn'

Add "-Wimplicit-fallthrough" to default flags (was Re: pgsql: Support FETCH FIRST WITH TIES)

2020-04-12 Thread Julien Rouhaud
On Sat, Apr 11, 2020 at 02:47:34PM -0400, Tom Lane wrote: > Julien Rouhaud writes: > > On Tue, Apr 7, 2020 at 10:28 PM Alvaro Herrera > > wrote: > >> Support FETCH FIRST WITH TIES > > > FTR I now get the following when compiling with -Wimplicit-fallthrough > > -Werror: > > Yeah, assorted buil

Re: cleaning perl code

2020-04-12 Thread Noah Misch
On Sat, Apr 11, 2020 at 12:13:08PM -0400, Andrew Dunstan wrote: > --- a/src/tools/msvc/Project.pm > +++ b/src/tools/msvc/Project.pm > @@ -420,13 +420,10 @@ sub read_file > { > my $filename = shift; > my $F; > - my $t = $/; > - > - undef $/; > + local $/ = undef; > ope

Re: Add A Glossary

2020-04-12 Thread Jürgen Purtz
On 11.04.20 21:47, Corey Huinker wrote: Term 'relation': A sequence is internally a table with one row - right? Shall we extend the list of concrete relations by 'sequence'? Or is this not necessary because 'table' is already there? I wrote one for sequence, it was a bit

Re: cleaning perl code

2020-04-12 Thread Noah Misch
On Sat, Apr 11, 2020 at 11:14:52AM -0400, Tom Lane wrote: > Noah Misch writes: > > In summary, among those warnings, I see non-negative value in "Code before > > warnings are enabled" only. While we're changing this, I propose removing > > Subroutines::RequireFinalReturn. > > If it's possible to