Re: [PATHC] Fix minor memory leak in pg_basebackup

2020-04-15 Thread Michael Paquier
On Wed, Apr 15, 2020 at 10:06:52AM +, Zhang, Jie wrote: > In some cases , PGresult is not cleared. > > File: src\bin\pg_basebackup\streamutil.c > > bool > RetrieveWalSegSize(PGconn *conn) > { > PGresult *res; RetrieveWalSegSize() gets called only once at the beginning of pg_basebacku

Re: Poll: are people okay with function/operator table redesign?

2020-04-15 Thread Pierre Giraud
Le 16/04/2020 à 00:18, Tom Lane a écrit : > As I threatened to do earlier, I made a pass at converting table 9.10 > to a couple of the styles under discussion. (This is just a > draft-quality patch, so it might have some minor bugs --- the point > is just to see what these styles look like.) >

Re: Vacuum o/p with (full 1, parallel 0) option throwing an error

2020-04-15 Thread Amit Kapila
On Wed, Apr 15, 2020 at 9:12 AM Amit Kapila wrote: > > On Wed, Apr 15, 2020 at 9:03 AM Justin Pryzby wrote: > > > > On Wed, Apr 15, 2020 at 08:54:17AM +0530, Amit Kapila wrote: > > > On Tue, Apr 14, 2020 at 8:55 AM Amit Kapila > > > wrote: > > > > > > > > On Tue, Apr 14, 2020 at 7:52 AM Michael

Re: pg_restore: could not close data file: Success

2020-04-15 Thread Michael Paquier
On Thu, Apr 16, 2020 at 12:08:09PM +0900, Kyotaro Horiguchi wrote: > I'm surprised to find an old thread about the same issue. > > https://www.postgresql.org/message-id/20160307.174354.251049100.horiguchi.kyotaro%40lab.ntt.co.jp > > But I don't think it's not acceptable that use fake errno for gz

Re: Do we need to handle orphaned prepared transactions in the server?

2020-04-15 Thread Craig Ringer
On Thu, 16 Apr 2020 at 13:23, Craig Ringer wrote: > > Just discarding the prepared xacts is not the answer though. > ... however, I have wondered a few times about making vacuum smarter about cases where the xmin is held down by prepared xacts or by replication slots. If we could record the olde

Re: Do we need to handle orphaned prepared transactions in the server?

2020-04-15 Thread Craig Ringer
On Wed, 15 Apr 2020 at 03:19, Robert Haas wrote: > On Wed, Feb 19, 2020 at 10:05 AM Hamid Akhtar > wrote: > > Attached is version 1 of POC patch for notifying of orphaned > > prepared transactions via warnings emitted to a client > > application and/or log file. It applies to PostgreSQL branch >

Re: pg_basebackup, manifests and backends older than ~12

2020-04-15 Thread Michael Paquier
On Tue, Apr 14, 2020 at 08:09:22PM -0400, Tom Lane wrote: > Michael Paquier writes: >> On Mon, Apr 13, 2020 at 07:04:20PM -0400, Alvaro Herrera wrote: >>> I agree, I think forcing users to specify --no-manifest when run on old >>> servers will cause users to write bad scripts; I vote for silently

Re: Poll: are people okay with function/operator table redesign?

2020-04-15 Thread Kyotaro Horiguchi
At Wed, 15 Apr 2020 12:04:34 -0400, Robert Haas wrote in > On Wed, Apr 15, 2020 at 11:54 AM Pavel Stehule > wrote: > > st 15. 4. 2020 v 17:43 odesílatel Isaac Morland > > napsal: > >> On Wed, 15 Apr 2020 at 11:26, Pierre Giraud > >> wrote: > >>> The best way to achieve this is to use some

Re: Include sequence relation support in logical replication

2020-04-15 Thread Craig Ringer
On Thu, 16 Apr 2020 at 07:44, Andres Freund wrote: > > > I would like to ask if you have some suggestions or ideas that can make > subscriber receives the current value without the need to > > > > disabling the 32 increment behavior? > > It simply shouldn't expect that to be the case. What do yo

Re: Allow pg_read_all_stats to read pg_stat_progress_*

2020-04-15 Thread Kyotaro Horiguchi
At Wed, 15 Apr 2020 15:58:05 +0500, "Andrey M. Borodin" wrote in > > 15 апр. 2020 г., в 15:25, Magnus Hagander написал(а): > > I think that makes perfect sense. The documentation explicitly says "can > > read all pg_stat_* views", which is clearly wrong -- so either the code or > > the docs s

Re: pg_restore: could not close data file: Success

2020-04-15 Thread Kyotaro Horiguchi
Hello. Added -hackers. At Wed, 15 Apr 2020 12:14:25 +0200, "Peter J. Holzer" wrote in > On 2020-04-15 12:01:46 +0200, Peter J. Holzer wrote: > > I'm trying to restore a backup on a different machine and it terminates > > with the not really helpful messages: > > > > pg_restore: [directory arc

Re: cleaning perl code

2020-04-15 Thread Noah Misch
On Wed, Apr 15, 2020 at 03:43:36PM -0400, Andrew Dunstan wrote: > On 4/14/20 4:44 PM, Alvaro Herrera wrote: > > On 2020-Apr-14, Andrew Dunstan wrote: > >> One of the things that's a bit sad is that perlcritic doesn't generally > >> let you apply policies to a given set of files or files matching so

Re: remove_useless_groupby_columns does not need to record constraint dependencies

2020-04-15 Thread David Rowley
On Thu, 16 Apr 2020 at 03:24, Tom Lane wrote: > > David Rowley writes: > > Over in [1], Tom and I had a discussion in response to some confusion > > about why remove_useless_groupby_columns() goes to the trouble of > > recording a dependency on the PRIMARY KEY constraint when removing > > surplus

Re: [DOC] Document concurrent index builds waiting on each other

2020-04-15 Thread James Coleman
On Wed, Apr 15, 2020 at 6:31 PM Andres Freund wrote: > > Hi, > > On 2020-04-15 09:31:58 -0400, James Coleman wrote: > > On Wed, Mar 25, 2020 at 3:58 PM Andres Freund wrote: > > > On 2020-03-25 16:30:10 -0300, Alvaro Herrera wrote: > > > > I posted this in November > > > > https://postgr.es/m/2019

Re: xid wraparound danger due to INDEX_CLEANUP false

2020-04-15 Thread Peter Geoghegan
On Wed, Apr 15, 2020 at 4:57 PM Robert Haas wrote: > I seem to recall Simon raising this issue at the time that the patch > was being discussed, and I thought that we had eventually decided that > it was OK for some reason. But I don't remember the details, and it is > possible that we got it wron

Re: tadarida vs REL_12_STABLE

2020-04-15 Thread Tom Lane
"Tom Turelinckx" writes: > It turns out I'd missed one failure: > https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=tadarida&br=REL_12_STABLE > Only tadarida fails the sequence regression test, and only > for REL_12_STABLE. It fails with -O2 and -O1 but succeeds with -O0. Yeah, I saw t

Re: xid wraparound danger due to INDEX_CLEANUP false

2020-04-15 Thread Robert Haas
On Wed, Apr 15, 2020 at 7:38 PM Andres Freund wrote: > It's possible that something protects against dangers in the case of > INDEX CLEANUP false, or that the consequences aren't too bad. But I > didn't see any comments about the danagers in the patch. I seem to recall Simon raising this issue at

Re: where should I stick that backup?

2020-04-15 Thread Robert Haas
On Wed, Apr 15, 2020 at 6:13 PM Andres Freund wrote: > I guess what I perceived to be the fundamental difference, before this > email, between our positions is that I (still) think that exposing > detailed postprocessing shell fragment style arguments to pg_basebackup, > especially as the only opt

Re: Include sequence relation support in logical replication

2020-04-15 Thread Andres Freund
Hi, On 2020-03-26 15:33:33 -0700, Cary Huang wrote: > >> For the replication to make sense, the patch actually disables the WAL > > >> update at every 32 nextval() calls, so every call to nextval() will > > >> emit a WAL update for proper replication. This is done by setting > > >> SEQ_LOG_

xid wraparound danger due to INDEX_CLEANUP false

2020-04-15 Thread Andres Freund
Hi, commit a96c41feec6b6616eb9d5baee9a9e08c20533c38 Author: Robert Haas Date: 2019-04-04 14:58:53 -0400 Allow VACUUM to be run with index cleanup disabled. This commit adds a new reloption, vacuum_index_cleanup, which controls whether index cleanup is performed for a particular

Re: Comment explaining why vacuum needs to push snapshot seems insufficient.

2020-04-15 Thread Andres Freund
Hi, On 2020-04-15 17:56:58 -0400, Alvaro Herrera wrote: > On 2020-Apr-05, Andres Freund wrote: > > > vacuum_rel() has the following comment: > > /* > > * Functions in indexes may want a snapshot set. Also, setting a > > snapshot > > * ensures that RecentGlobalXmin is kept truly re

Re: documenting the backup manifest file format

2020-04-15 Thread David Steele
On 4/15/20 6:43 PM, Jehan-Guillaume de Rorthais wrote: On Wed, 15 Apr 2020 12:03:28 -0400 Robert Haas wrote: On Wed, Apr 15, 2020 at 11:23 AM Jehan-Guillaume de Rorthais wrote: But for backup_manifest, it's kind of shame we have to check the checksum against an transformed version of the fil

Re: documenting the backup manifest file format

2020-04-15 Thread Jehan-Guillaume de Rorthais
On Wed, 15 Apr 2020 12:03:28 -0400 Robert Haas wrote: > On Wed, Apr 15, 2020 at 11:23 AM Jehan-Guillaume de Rorthais > wrote: > > But for backup_manifest, it's kind of shame we have to check the checksum > > against an transformed version of the file. Did you consider creating eg. a > > separate

Re: [DOC] Document concurrent index builds waiting on each other

2020-04-15 Thread Andres Freund
Hi, On 2020-04-15 09:31:58 -0400, James Coleman wrote: > On Wed, Mar 25, 2020 at 3:58 PM Andres Freund wrote: > > On 2020-03-25 16:30:10 -0300, Alvaro Herrera wrote: > > > I posted this in November > > > https://postgr.es/m/20191101203310.GA12239@alvherre.pgsql but I didn't > > > put time to go t

Re: where should I stick that backup?

2020-04-15 Thread Andres Freund
Hi, On 2020-04-15 09:23:30 -0400, Robert Haas wrote: > On Tue, Apr 14, 2020 at 9:50 PM Andres Freund wrote: > > On 2020-04-14 11:38:03 -0400, Robert Haas wrote: > > > I'm fairly deeply uncomfortable with what Andres is proposing. I see > > > that it's very powerful, and can do a lot of things, an

Re: Comment explaining why vacuum needs to push snapshot seems insufficient.

2020-04-15 Thread Alvaro Herrera
On 2020-Apr-05, Andres Freund wrote: > vacuum_rel() has the following comment: > /* >* Functions in indexes may want a snapshot set. Also, setting a > snapshot >* ensures that RecentGlobalXmin is kept truly recent. >*/ > PushActiveSnapshot(GetTransactionSnapsh

Re: pgsql: Rationalize GetWalRcv{Write,Flush}RecPtr().

2020-04-15 Thread Alvaro Herrera
On 2020-Apr-09, Alvaro Herrera wrote: > It seems worth pointing out that the new GetWalRcvWriteRecPtr function > has a different signature from the original one -- so any third-party > code using the original function will now get a compile failure that > should alert them that they need to change

tadarida vs REL_12_STABLE

2020-04-15 Thread Tom Turelinckx
Hi, > On 2020-Apr-13, I wrote to buildfarm-admins: > > As skate and snapper are increasingly difficult to keep alive and > > building on debian sparc, and as there aren't many sparc animals > > in general, I've set up four new debian sparc64 animals, two on > > stretch and two on buster. > > >

Re: Parallel copy

2020-04-15 Thread Kuntal Ghosh
On Wed, Apr 15, 2020 at 10:45 PM Andres Freund wrote: > > Hi, > > On 2020-04-15 20:36:39 +0530, Kuntal Ghosh wrote: > > I was thinking from this point of view - the sooner we introduce > > parallelism in the process, the greater the benefits. > > I don't really agree. Sure, that's true from a theo

Re: cleaning perl code

2020-04-15 Thread Andrew Dunstan
On 4/14/20 4:44 PM, Alvaro Herrera wrote: > On 2020-Apr-14, Andrew Dunstan wrote: > >> One of the things that's a bit sad is that perlcritic doesn't generally >> let you apply policies to a given set of files or files matching some >> pattern. It would be nice, for instance, to be able to apply so

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-04-15 Thread Ranier Vilela
Em qua., 15 de abr. de 2020 às 15:28, Juan José Santamaría Flecha < juanjo.santama...@gmail.com> escreveu: > > > On Wed, Apr 15, 2020 at 4:46 PM Ranier Vilela wrote: > >> Em qua., 15 de abr. de 2020 às 03:08, davinder singh < >> davindersingh2...@gmail.com> escreveu: >> >>> >>> 5. Why call _creat

Re: Poll: are people okay with function/operator table redesign?

2020-04-15 Thread David G. Johnston
On Mon, Apr 13, 2020 at 10:13 AM Tom Lane wrote: > As discussed in the thread at [1], I've been working on redesigning > the tables we use to present SQL functions and operators. The > first installment of that is now up; see tables 9.30 and 9.31 at > > https://www.postgresql.org/docs/devel/func

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-04-15 Thread Juan José Santamaría Flecha
On Wed, Apr 15, 2020 at 4:46 PM Ranier Vilela wrote: > Em qua., 15 de abr. de 2020 às 03:08, davinder singh < > davindersingh2...@gmail.com> escreveu: > >> >> 5. Why call _create_locale if _WIN32_WINNT >= 0x0600 is true and loct is >>> not used? >>> >> _create_locale can take bigger input than Ge

Re: Parallel copy

2020-04-15 Thread Andres Freund
Hi, On 2020-04-15 12:05:47 +0300, Ants Aasma wrote: > I see the benefit of having one process responsible for splitting as > being able to run ahead of the workers to queue up work when many of > them need new data at the same time. Yea, I agree. > I don't think the locking benefits of a ring a

Re: Incremental sorts and EXEC_FLAG_REWIND

2020-04-15 Thread James Coleman
On Wed, Apr 15, 2020 at 11:02 AM James Coleman wrote: > > On Tue, Apr 14, 2020 at 2:53 AM Michael Paquier wrote: > > > > Hi, > > > > When initializing an incremental sort node, we have the following as > > of ExecInitIncrementalSort(): > > /* > > * Incremental sort can't be used with eit

Re: Poll: are people okay with function/operator table redesign?

2020-04-15 Thread Tom Lane
Steven Pousty writes: > Is there a way to get a heavier line between each function? It would be > helpful to have a clearer demarcation of what belongs to each function. The first alternative I posted at https://www.postgresql.org/message-id/31833.1586817876%40sss.pgh.pa.us seems like it would

Re: Parallel copy

2020-04-15 Thread Andres Freund
Hi, On 2020-04-15 20:36:39 +0530, Kuntal Ghosh wrote: > I was thinking from this point of view - the sooner we introduce > parallelism in the process, the greater the benefits. I don't really agree. Sure, that's true from a theoretical perspective, but the incremental gains may be very small, and

Re: Poll: are people okay with function/operator table redesign?

2020-04-15 Thread Steven Pousty
Is there a way to get a heavier line between each function? It would be helpful to have a clearer demarcation of what belongs to each function. On Wed, Apr 15, 2020 at 9:04 AM Robert Haas wrote: > On Wed, Apr 15, 2020 at 11:54 AM Pavel Stehule > wrote: > > st 15. 4. 2020 v 17:43 odesílatel Isaa

Re: Parallel copy

2020-04-15 Thread Andres Freund
On 2020-04-15 10:12:14 -0400, Robert Haas wrote: > On Wed, Apr 15, 2020 at 7:15 AM Amit Kapila wrote: > > As I understand this, it needs to parse the lines twice (second time > > in phase-3) and till the first two phases are over, we can't start the > > tuple processing work which is done in phase

Re: Poll: are people okay with function/operator table redesign?

2020-04-15 Thread Robert Haas
On Wed, Apr 15, 2020 at 11:54 AM Pavel Stehule wrote: > st 15. 4. 2020 v 17:43 odesílatel Isaac Morland > napsal: >> On Wed, 15 Apr 2020 at 11:26, Pierre Giraud wrote: >>> The best way to achieve this is to use some styling (font style and color). >>> >>> Attached you will find two different op

Re: documenting the backup manifest file format

2020-04-15 Thread Robert Haas
On Wed, Apr 15, 2020 at 11:23 AM Jehan-Guillaume de Rorthais wrote: > But for backup_manifest, it's kind of shame we have to check the checksum > against an transformed version of the file. Did you consider creating eg. a > separate backup_manifest.sha256 file? > > I'm very sorry in advance if thi

design for parallel backup

2020-04-15 Thread Robert Haas
Hi, Over at http://postgr.es/m/CADM=JehKgobEknb+_nab9179HzGj=9eitzwmod2mpqr_rif...@mail.gmail.com there's a proposal for a parallel backup patch which works in the way that I have always thought parallel backup would work: instead of having a monolithic command that returns a series of tarballs,

Re: Poll: are people okay with function/operator table redesign?

2020-04-15 Thread Pavel Stehule
st 15. 4. 2020 v 17:43 odesílatel Isaac Morland napsal: > On Wed, 15 Apr 2020 at 11:26, Pierre Giraud > wrote: > > >> The best way to achieve this is to use some styling (font style and >> color). >> >> Attached you will find two different options I worked on very quickly. >> > > I really like t

Re: Poll: are people okay with function/operator table redesign?

2020-04-15 Thread Isaac Morland
On Wed, 15 Apr 2020 at 11:26, Pierre Giraud wrote: > The best way to achieve this is to use some styling (font style and color). > > Attached you will find two different options I worked on very quickly. > I really like the first. Just a couple of suggestions I would make: - leave a space betw

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-15 Thread Tom Lane
Kyotaro Horiguchi writes: > At Tue, 14 Apr 2020 16:32:40 -0400, Tom Lane wrote in > + stby->is_sync_standby = true; /* might change below */ > I'm uneasy with that. In quorum mode all running standbys are marked > as "sync" and that's bogus. I don't follow that? The existing co

Re: sqlsmith crash incremental sort

2020-04-15 Thread James Coleman
On Wed, Apr 15, 2020 at 10:47 AM Tomas Vondra wrote: > > On Tue, Apr 14, 2020 at 01:16:33PM -0400, James Coleman wrote: > >On Sun, Apr 12, 2020 at 8:09 PM Tomas Vondra > > wrote: > >> > >> On Sun, Apr 12, 2020 at 12:44:45AM +0200, Tomas Vondra wrote: > >> >Hi, > >> > > >> >I've looked into this a

Re: remove_useless_groupby_columns does not need to record constraint dependencies

2020-04-15 Thread Tom Lane
David Rowley writes: > Over in [1], Tom and I had a discussion in response to some confusion > about why remove_useless_groupby_columns() goes to the trouble of > recording a dependency on the PRIMARY KEY constraint when removing > surplus columns from the GROUP BY clause. > The outcome was that

Re: documenting the backup manifest file format

2020-04-15 Thread Jehan-Guillaume de Rorthais
On Tue, 14 Apr 2020 12:56:49 -0400 Robert Haas wrote: > On Mon, Apr 13, 2020 at 5:43 PM Alvaro Herrera > wrote: > > Yeah, I guess I'm just saying that it feels brittle to have a file > > format that's supposed to be good for data exchange and then make it > > itself depend on representation deta

Re: Parallel copy

2020-04-15 Thread Kuntal Ghosh
On Wed, Apr 15, 2020 at 2:15 PM Ants Aasma wrote: > > On Tue, 14 Apr 2020 at 22:40, Kuntal Ghosh wrote: > > 1. Each worker scans a distinct fixed sized chunk of the CSV file and > > collects the following three stats from the chunk: > > a) number of quotes > > b) position of the first new line af

Re: wrong relkind error messages

2020-04-15 Thread Alvaro Herrera
On 2020-Apr-15, Robert Haas wrote: > [good arguments] I don't disagree with anything you said, and I don't have anything to add for now. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Incremental sorts and EXEC_FLAG_REWIND

2020-04-15 Thread James Coleman
On Tue, Apr 14, 2020 at 2:53 AM Michael Paquier wrote: > > Hi, > > When initializing an incremental sort node, we have the following as > of ExecInitIncrementalSort(): > /* > * Incremental sort can't be used with either EXEC_FLAG_REWIND, > * EXEC_FLAG_BACKWARD or EXEC_FLAG_MARK, beca

Re: sqlsmith crash incremental sort

2020-04-15 Thread Tomas Vondra
On Tue, Apr 14, 2020 at 01:16:33PM -0400, James Coleman wrote: On Sun, Apr 12, 2020 at 8:09 PM Tomas Vondra wrote: On Sun, Apr 12, 2020 at 12:44:45AM +0200, Tomas Vondra wrote: >Hi, > >I've looked into this a bit, and at first I thought that maybe the >issue is in how cost_incremental_sort pic

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-04-15 Thread Ranier Vilela
Em qua., 15 de abr. de 2020 às 03:08, davinder singh < davindersingh2...@gmail.com> escreveu: > > Thanks for the review comments. > > On Tue, Apr 14, 2020 at 9:12 PM Ranier Vilela wrote: > >> >>I m still working on testing this patch. If anyone has Idea please >> suggest. >> I still see problems

Re: wrong relkind error messages

2020-04-15 Thread Robert Haas
On Tue, Apr 14, 2020 at 7:02 PM Alvaro Herrera wrote: > On 2020-Apr-13, Robert Haas wrote: > > + ereport(ERROR, > > + (errcode(ERRCODE_WRONG_OBJECT_TYPE), > > + errmsg("action cannot be performed on relation \"%s\"", > > + RelationGetRelationName(rel)), > > > > Super-vague. > > Maybe, but note tha

Re: Perl modules for testing/viewing/corrupting/repairing your heap files

2020-04-15 Thread Mark Dilger
> On Apr 14, 2020, at 6:17 PM, Peter Geoghegan wrote: > > On Wed, Apr 8, 2020 at 3:51 PM Mark Dilger > wrote: >> Recently, as part of testing something else, I had need of a tool to create >> surgically precise corruption within heap pages. I wanted to make the >> corruption from within TAP

Re: Parallel copy

2020-04-15 Thread Robert Haas
On Wed, Apr 15, 2020 at 7:15 AM Amit Kapila wrote: > As I understand this, it needs to parse the lines twice (second time > in phase-3) and till the first two phases are over, we can't start the > tuple processing work which is done in phase-3. So even if the > tokenization is done a bit faster b

Re: [DOC] Document concurrent index builds waiting on each other

2020-04-15 Thread James Coleman
On Wed, Mar 25, 2020 at 3:58 PM Andres Freund wrote: > > Hi, > > On 2020-03-25 16:30:10 -0300, Alvaro Herrera wrote: > > I posted this in November > > https://postgr.es/m/20191101203310.GA12239@alvherre.pgsql but I didn't > > put time to go through the issues there. > > Oh, missed that. > > > > I

Re: WIP/PoC for parallel backup

2020-04-15 Thread Robert Haas
On Wed, Apr 15, 2020 at 4:49 AM Ahsan Hadi wrote: > Fair enough. Some of this is also due to backup related features i.e backup > manifest, progress reporting that got committed to master towards the tail > end of PG-13. Rushing to get parallel backup feature compatible with these > features al

Re: documenting the backup manifest file format

2020-04-15 Thread Robert Haas
On Tue, Apr 14, 2020 at 11:49 PM Fujii Masao wrote: > While reading the document that you pushed, I thought that it's better > to define index term for backup manifest, so that we can easily reach > this document from the index page. Thought? Patch attached. Fine with me. I tend not to think abou

Re: where should I stick that backup?

2020-04-15 Thread Robert Haas
On Tue, Apr 14, 2020 at 9:50 PM Andres Freund wrote: > On 2020-04-14 11:38:03 -0400, Robert Haas wrote: > > I'm fairly deeply uncomfortable with what Andres is proposing. I see > > that it's very powerful, and can do a lot of things, and that if > > you're building something that does sophisticate

Re: Parallel copy

2020-04-15 Thread Amit Kapila
On Wed, Apr 15, 2020 at 1:10 AM Kuntal Ghosh wrote: > > Hence, I was trying to think whether we can leverage this idea for > implementing parallel COPY in PG. We can design an algorithm similar > to parallel hash-join where the workers pass through different phases. > 1. Phase 1 - Read fixed size

Re: Allow pg_read_all_stats to read pg_stat_progress_*

2020-04-15 Thread Andrey M. Borodin
> 15 апр. 2020 г., в 15:25, Magnus Hagander написал(а): > > > I think that makes perfect sense. The documentation explicitly says "can read > all pg_stat_* views", which is clearly wrong -- so either the code or the > docs should be fixed, and it looks like it's the code that should be fixe

Re: Allow pg_read_all_stats to read pg_stat_progress_*

2020-04-15 Thread Magnus Hagander
On Wed, Apr 15, 2020 at 9:14 AM Andrey M. Borodin wrote: > Hi! > > One of our users asked me why they cannot read details of > pg_stat_progress_vacuum while they have pg_read_all_stats role. > Maybe I'm missing something, but I think they should be able to read > stats... > > PFA fix. > This affe

[PATHC] Fix minor memory leak in pg_basebackup

2020-04-15 Thread Zhang, Jie
Hi all In some cases , PGresult is not cleared. File: src\bin\pg_basebackup\streamutil.c bool RetrieveWalSegSize(PGconn *conn) { PGresult *res; .. res = PQexec(conn, "SHOW wal_segment_size"); if (PQresultStatus(res) != PGRES_TUPLES_OK) { pg_l

Re: WIP/PoC for parallel backup

2020-04-15 Thread Rajkumar Raghuwanshi
Hi Asif, In below scenarios backup verification failed for tablespace, when backup taken with parallel option. without parallel for the same scenario pg_verifybackup is passed without any error. [edb@localhost bin]$ mkdir /tmp/test_bkp/tblsp1 [edb@localhost bin]$ ./psql postgres -p 5432 -c "creat

Re: Parallel copy

2020-04-15 Thread Ants Aasma
On Mon, 13 Apr 2020 at 23:16, Andres Freund wrote: > > Still, if the reader does the splitting, then you don't need as much > > IPC, right? The shared memory data structure is just a ring of bytes, > > and whoever reads from it is responsible for the rest. > > I don't think so. If only one process

Re: WIP/PoC for parallel backup

2020-04-15 Thread Ahsan Hadi
On Wed, 15 Apr 2020 at 1:49 AM, Robert Haas wrote: > On Tue, Apr 14, 2020 at 10:37 AM Asif Rehman > wrote: > > I forgot to make a check for no-manifest. Fixed. Attached is the updated > patch. > > +typedef struct > +{ > ... > +} BackupFile; > + > +typedef struct > +{ > ... > +} BackupState; > >

Re: Parallel copy

2020-04-15 Thread Ants Aasma
On Tue, 14 Apr 2020 at 22:40, Kuntal Ghosh wrote: > 1. Each worker scans a distinct fixed sized chunk of the CSV file and > collects the following three stats from the chunk: > a) number of quotes > b) position of the first new line after even number of quotes > c) position of the first new line a

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-15 Thread Kyotaro Horiguchi
At Wed, 15 Apr 2020 13:01:02 +0900, Masahiko Sawada wrote in > On Tue, 14 Apr 2020 at 18:35, Kyotaro Horiguchi > wrote: > > > > At Tue, 14 Apr 2020 13:06:14 +0900, Masahiko Sawada > > wrote in > > > On Tue, 14 Apr 2020 at 10:34, Tom Lane wrote: > > > > > > > > Kyotaro Horiguchi writes: > >

Re: Race condition in SyncRepGetSyncStandbysPriority

2020-04-15 Thread Kyotaro Horiguchi
At Wed, 15 Apr 2020 11:35:58 +0900 (JST), Kyotaro Horiguchi wrote in > I'm looking this more closer. It looks to be in the right direction to me. As mentioned in the previous mail, I removed is_sync_standby from SycnStandbyData. But just doing that breaks pg_stat_get_wal_senders. It is an exst

Parallel Append can break run-time partition pruning

2020-04-15 Thread David Rowley
I had a report from the wilds that run-time partition pruning was not working in certain cases. After some investigation and obtaining the mockup of the actual case, I discovered that the problem was down to accumulate_append_subpath() hitting the case where it does not pullup a Parallel Append wh

Allow pg_read_all_stats to read pg_stat_progress_*

2020-04-15 Thread Andrey M. Borodin
Hi! One of our users asked me why they cannot read details of pg_stat_progress_vacuum while they have pg_read_all_stats role. Maybe I'm missing something, but I think they should be able to read stats... PFA fix. This affects pg_stat_progress_analyze, pg_stat_progress_basebackup, pg_stat_progre

Re: [patch] some PQExpBuffer are not destroyed in pg_dump

2020-04-15 Thread Michael Paquier
On Tue, Apr 14, 2020 at 10:11:56AM +0900, Michael Paquier wrote: > Indeed. Any code path of pg_dump calling buildACLQueries() clears up > things, and I think that it is a better practice to clean up properly > PQExpBuffer stuff even if there is always the argument that pg_dump > is a tool running