Re: [HACKERS] POC: Cache data in GetSnapshotData()

2017-09-20 Thread Michael Paquier
On Thu, Sep 21, 2017 at 3:15 AM, Robert Haas wrote: > On Tue, Aug 29, 2017 at 1:57 AM, Mithun Cy wrote: >> All TPS are median of 3 runs >> Clients TPS-With Patch 05 TPS-Base%Diff >> 1 752.461117755.186777 -0.3% >> 64 32171.296537

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-20 Thread Michael Paquier
On Thu, Sep 21, 2017 at 9:51 AM, Tom Lane wrote: > Michael Paquier writes: >> On Thu, Sep 21, 2017 at 9:08 AM, Tom Lane wrote: >>> Um ... so? With Nathan's proposed behavior, there are two cases depending >>> on just when the unexpected schema change happens

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-20 Thread Michael Paquier
On Thu, Sep 21, 2017 at 9:08 AM, Tom Lane wrote: > Michael Paquier writes: >> On Thu, Sep 21, 2017 at 8:18 AM, Tom Lane wrote: >>> "Bossart, Nathan" writes: >>>> I agree that the patch might be simpler without this, but the user-visible >>>>

Re: [HACKERS] Windows warnings from VS 2017

2017-09-20 Thread Michael Paquier
On Thu, Sep 21, 2017 at 7:49 AM, Andrew Dunstan wrote: > On 09/20/2017 06:13 PM, Michael Paquier wrote: >> On Thu, Sep 21, 2017 at 7:08 AM, Andrew Dunstan >> wrote: >>> First, it warns about a couple of unused variables at lines 4553 and >>> 4673 of src/backend/o

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-20 Thread Michael Paquier
On Thu, Sep 21, 2017 at 8:18 AM, Tom Lane wrote: > "Bossart, Nathan" writes: >> On 9/20/17, 2:29 PM, "Tom Lane" wrote: >>> The idea of "fast failing" because some later VacuumRelation struct might >>> contain an error seems like useless complication, both in terms of the >>> implementation and t

Re: [HACKERS] Windows warnings from VS 2017

2017-09-20 Thread Michael Paquier
On Thu, Sep 21, 2017 at 7:08 AM, Andrew Dunstan wrote: > First, it warns about a couple of unused variables at lines 4553 and > 4673 of src/backend/optimizer/path/costsize.c. I think we can do a > little rearrangement to keep it happy there. Those are around for some time, see here: https://www.p

Re: [HACKERS] [COMMITTERS] pgsql: Make WAL segment size configurable at initdb time.

2017-09-19 Thread Michael Paquier
On Wed, Sep 20, 2017 at 2:04 PM, Andres Freund wrote: > Make WAL segment size configurable at initdb time. > > For performance reasons a larger segment size than the default 16MB > can be useful. A larger segment size has two main benefits: Firstly, > in setups using archiving, it makes it easier

Re: [HACKERS] "inconsistent page found" with checksum and wal_consistency_checking enabled

2017-09-19 Thread Michael Paquier
On Wed, Sep 20, 2017 at 2:26 PM, Kuntal Ghosh wrote: > On Wed, Sep 20, 2017 at 10:22 AM, Michael Paquier > wrote: >> On Wed, Sep 20, 2017 at 5:23 AM, Ashwin Agrawal wrote: >>> Currently, page checksum is not masked by Page masking routines used by >>> wal_consisten

Re: [HACKERS] src/test/subscription/t/005_encoding.pl is broken

2017-09-19 Thread Michael Paquier
On Wed, Sep 20, 2017 at 12:33 AM, Tom Lane wrote: > That would indicate that something isn't ever retrying the worker > start; but if that's the case, how is it that we get through the > other subscription tests with my random-failure patch in place? I have been able to dig into this issue furthe

Re: [HACKERS] "inconsistent page found" with checksum and wal_consistency_checking enabled

2017-09-19 Thread Michael Paquier
On Wed, Sep 20, 2017 at 5:23 AM, Ashwin Agrawal wrote: > Currently, page checksum is not masked by Page masking routines used by > wal_consistency_checking facility. So, when running `make installcheck` with > data checksum enabled and wal_consistency_checking='all', it easily and > consistently F

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-19 Thread Michael Paquier
On Wed, Sep 20, 2017 at 12:49 PM, Amit Kapila wrote: > On Wed, Sep 20, 2017 at 4:25 AM, Michael Paquier > wrote: >> Also, _hash_init() would need some extra work to >> generate FPWs, but I don't think that it is necessary per its handling >> of a per-record meta da

Re: [HACKERS] sync process names between ps and pg_stat_activity

2017-09-19 Thread Michael Paquier
On Fri, Sep 1, 2017 at 5:33 AM, Tom Lane wrote: > Peter Eisentraut writes: >> As an aside, is there a reason why the archiver process is not included >> in pg_stat_activity? > > It's not connected to shared memory. Do you think that monitoring would be a reason sufficient to do so? My personal o

Re: [HACKERS] SCRAM in the PG 10 release notes

2017-09-19 Thread Michael Paquier
On Wed, Sep 20, 2017 at 9:45 AM, Peter Eisentraut wrote: > On 9/19/17 17:55, Jeff Janes wrote: >> I guess I'm late to the party, but I don't see why this is needed at >> all. We encourage people to use any and all new features which are >> appropriate to them--that is why we implement new feature

Re: [HACKERS] SCRAM in the PG 10 release notes

2017-09-19 Thread Michael Paquier
On Wed, Sep 20, 2017 at 6:55 AM, Jeff Janes wrote: > On Tue, Sep 19, 2017 at 1:32 PM, Heikki Linnakangas wrote: >> I'm not sure what exactly to do here. Where should we stick that notice? >> We could put it in the release notes, where the bullet point about SCRAM is, >> but it would be well hidde

Re: [HACKERS] Rewriting the test of pg_upgrade as a TAP test

2017-09-19 Thread Michael Paquier
On Wed, Sep 20, 2017 at 7:57 AM, Peter Eisentraut wrote: > To get things rolling, I have committed just the basic TAP tests, to > give it a spin on the build farm. I'll work through the rest in the > coming days. Thanks! -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgr

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-19 Thread Michael Paquier
On Wed, Sep 20, 2017 at 12:47 AM, Tom Lane wrote: > Amit Kapila writes: >> On Tue, Sep 19, 2017 at 9:27 AM, Michael Paquier >> wrote: >>> I am not saying that no index AMs take advantage FPW compressibility >>> for their meta pages. There are cases like this one

Re: [HACKERS] src/test/subscription/t/005_encoding.pl is broken

2017-09-19 Thread Michael Paquier
On Tue, Sep 19, 2017 at 8:51 PM, Robert Haas wrote: > On Mon, Sep 18, 2017 at 1:58 PM, Andres Freund wrote: >> To my knowledge here's not really any difference between the two in >> logical replication. Received changes are immediately applied, there's >> no equivalent to a walreceiver queing up

Re: [HACKERS] Rewriting the test of pg_upgrade as a TAP test

2017-09-19 Thread Michael Paquier
On Tue, Sep 19, 2017 at 6:15 PM, Alvaro Herrera wrote: > Michael Paquier wrote: >> On Thu, Sep 7, 2017 at 11:14 AM, Michael Paquier >> wrote: >> > Or we could make upgradecheck a noop, then remove it once all the MSVC >> > animals have upgraded to a newer vers

Re: [HACKERS] Simplify ACL handling for large objects and removal of superuser() checks

2017-09-19 Thread Michael Paquier
On Tue, Sep 19, 2017 at 1:24 PM, Vaishnavi Prabakaran wrote: > On Mon, Aug 14, 2017, Michael Paquier wrote: >>Attached is a set of 3 patches: > > I tried to review the patch and firstly patch applies cleanly without any > noise. Thanks for the review, Vaishnavi.

Re: [HACKERS] Commits don't block for synchronous replication

2017-09-18 Thread Michael Paquier
On Tue, Sep 19, 2017 at 7:50 AM, Xin Zhang wrote: > If primary crashed at that moment, and failover to standby, the foo table is > lost, even though the replication is synced according to > `pg_stat_replication` view. GUC parameters are reloaded each time a query is run, and so SyncRepConfig is f

Re: [HACKERS] Rewriting the test of pg_upgrade as a TAP test

2017-09-18 Thread Michael Paquier
On Thu, Sep 7, 2017 at 11:14 AM, Michael Paquier wrote: > Or we could make upgradecheck a noop, then remove it once all the MSVC > animals have upgraded to a newer version of the buildfarm client which > does not use upgradecheck anymore (I am fine to send a patch or a pull > reque

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-18 Thread Michael Paquier
On Tue, Sep 19, 2017 at 12:57 PM, Michael Paquier wrote: > I'd think about adjusting the comments the proper way for each AM so > as one can read those comments and catch any limitation immediately. > The fact this has not been pointed out on this thread before any > checks a

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-18 Thread Michael Paquier
On Tue, Sep 19, 2017 at 12:40 PM, Amit Kapila wrote: > On Mon, Sep 18, 2017 at 4:03 PM, Michael Paquier > wrote: >> On Mon, Sep 18, 2017 at 11:16 AM, Amit Kapila >> wrote: >>> On Sat, Sep 16, 2017 at 7:15 PM, Michael Paquier >>> wrote: >>>> >

Re: [HACKERS] Bug with pg_basebackup and 'shared' tablespace

2017-09-18 Thread Michael Paquier
On Thu, Sep 14, 2017 at 2:17 AM, Pierre Ducroquet wrote: > All my apologies for the schockingly long time with no answer on this topic. No problem. That's the concept called life I suppose. > I will do my best to help review some patches in the current CF. Thanks for the potential help! > Atta

Re: [HACKERS] Creating backup history files for backups taken from standbys

2017-09-18 Thread Michael Paquier
On Tue, Sep 19, 2017 at 8:14 AM, David Steele wrote: > On 8/31/17 11:56 PM, Michael Paquier wrote: >> Here is an updated patch with refreshed documentation, as a result of >> 449338c which was discussed in thread >> https://www.postgresql.org/message-id/d4d951b9-89c

Re: [HACKERS] Is it time to kill support for very old servers?

2017-09-18 Thread Michael Paquier
On Mon, Sep 18, 2017 at 8:17 PM, Andres Freund wrote: > And now you missed the "same infrastructure" part. I am sort of aware of that per the list of parameters at the top fe-connect.c, thanks for the reminer :) -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] Is it time to kill support for very old servers?

2017-09-18 Thread Michael Paquier
On Mon, Sep 18, 2017 at 8:09 PM, Andres Freund wrote: > > > On September 18, 2017 4:08:21 AM PDT, Michael Paquier > wrote: >>On Mon, Sep 18, 2017 at 7:54 PM, Andres Freund >>wrote: >>>>It seems to me that you are looking more for a connection paramete

Re: [HACKERS] Is it time to kill support for very old servers?

2017-09-18 Thread Michael Paquier
On Mon, Sep 18, 2017 at 7:54 PM, Andres Freund wrote: >>It seems to me that you are looking more for a connection parameter >>here. > > I'm not seeing a meaningful distinction here? Env vars and connection > parameters are handled using the same framework in libpq. And using the env > var in th

Re: [HACKERS] Is it time to kill support for very old servers?

2017-09-18 Thread Michael Paquier
On Mon, Sep 18, 2017 at 6:53 PM, Andres Freund wrote: > On 2017-09-13 23:39:21 -0400, Tom Lane wrote: >> The real problem in this area, to my mind, is that we're not testing that >> code --- either end of it --- in any systematic way. If it's broken it >> could take us quite a while to notice. >

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-18 Thread Michael Paquier
On Mon, Sep 18, 2017 at 11:16 AM, Amit Kapila wrote: > On Sat, Sep 16, 2017 at 7:15 PM, Michael Paquier > wrote: >> On Fri, Sep 15, 2017 at 4:22 PM, Amit Langote >> wrote: >>> On 2017/09/14 16:00, Michael Paquier wrote: >>>> On Wed, Sep 13, 2017 at 4:43 P

Re: [HACKERS] pg_control_recovery() return value when not in recovery

2017-09-17 Thread Michael Paquier
On Mon, Sep 18, 2017 at 3:29 PM, Andres Freund wrote: > On 2017-09-18 07:24:43 +0100, Simon Riggs wrote: >> On 18 September 2017 at 05:50, Andres Freund wrote: >> > Hi, >> > >> > Just noticed that we're returning the underlying values for >> > pg_control_recovery() without any checks: >> > postgr

Re: [HACKERS] [PATCH]make pg_rewind to not copy useless WAL files

2017-09-16 Thread Michael Paquier
On Sun, Sep 17, 2017 at 3:19 AM, Alexander Korotkov wrote: > Hi! > > On Sat, Sep 16, 2017 at 5:56 PM, chenhj wrote: >> >> This patch optimizes the above mentioned issues, as follows: >> 1. In the target data directory, do not delete the WAL files before the >> divergence. >> 2. When copying files

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-16 Thread Michael Paquier
On Fri, Sep 15, 2017 at 4:22 PM, Amit Langote wrote: > On 2017/09/14 16:00, Michael Paquier wrote: >> On Wed, Sep 13, 2017 at 4:43 PM, Amit Langote >> wrote: >>> Sure, no problem. >> >> OK, I took a closer look at all patches, but did not run any manual >

Re: [HACKERS] additional contrib test suites

2017-09-15 Thread Michael Paquier
On Sat, Sep 16, 2017 at 5:15 AM, Tom Lane wrote: > I wrote: >> Peter Eisentraut writes: >>> So, we have one failure for chkpass on OpenBSD, because OpenBSD crypt() >>> doesn't support the traditional two-character salt format. > >>> Option: > >>> - Use the resultmap features to make this an expec

Re: [JDBC] [HACKERS] Channel binding support for SCRAM-SHA-256

2017-09-15 Thread Michael Paquier
On Sat, Sep 16, 2017 at 12:42 AM, Robert Haas wrote: > On Thu, Sep 14, 2017 at 12:58 PM, Peter Eisentraut > wrote: >> Second thoughts, to make things simpler. All we need for channel >> binding is a connection flag that says "I require channel binding". It >> could be modeled after the sslmode

Re: [HACKERS] [COMMITTERS] pgsql: passwordcheck: Add test suite

2017-09-14 Thread Michael Paquier
On Fri, Sep 15, 2017 at 12:02 PM, Michael Paquier wrote: > On Fri, Sep 15, 2017 at 11:46 AM, Peter Eisentraut wrote: >> passwordcheck: Add test suite >> >> Also improve one error message. >> >> Reviewed-by: David Steele > > Sorry for showing up late f

Re: [HACKERS] [COMMITTERS] pgsql: passwordcheck: Add test suite

2017-09-14 Thread Michael Paquier
On Fri, Sep 15, 2017 at 11:46 AM, Peter Eisentraut wrote: > passwordcheck: Add test suite > > Also improve one error message. > > Reviewed-by: David Steele Sorry for showing up late for this topic. +REGRESS_OPTS = --temp-config $(srcdir)/passwordcheck.conf +REGRESS = passwordcheck +# disabled be

Re: [HACKERS] Trouble with amcheck

2017-09-14 Thread Michael Paquier
On Fri, Sep 15, 2017 at 10:31 AM, Stephen Frost wrote: > Yes, I was working with someone earlier today who ran into exactly the > same issue. If you don't 'make world' or make the individual contrib > modules, then 'make installcheck-world' isn't going to work. Or should installcheck-world imply

Re: [HACKERS] Clarification in pg10's pgupgrade.html step 10 (upgrading standby servers)

2017-09-14 Thread Michael Paquier
On Fri, Sep 15, 2017 at 10:21 AM, Stephen Frost wrote: > No, one of the baseline requirements of pg_upgrade is to *not* screw > with the existing cluster. Removing its WAL or "cleaning it up" > definitely seems like it's violating that principle. Not necessarily. Using --link it is game over for

Re: [HACKERS] SCRAM in the PG 10 release notes

2017-09-14 Thread Michael Paquier
On Fri, Sep 15, 2017 at 12:10 AM, Alvaro Hernandez wrote: >> On the JDBC driver, strictly speaking, code has not been released yet. >> It is scheduled for v 42.2.0, and maybe the wiki should also mention from >> what version of the driver it is supported (I guess for all cases, unless >> their

Re: [HACKERS] Clarification in pg10's pgupgrade.html step 10 (upgrading standby servers)

2017-09-14 Thread Michael Paquier
On Fri, Sep 15, 2017 at 8:23 AM, Andreas Joseph Krogh wrote: > I tested upgrading from 9.6 to 10 now, using pg_upgrade, and pg_upgrade > creates the new data-dir with pg_wal "in it" (just like regular initdb), so > pg_upgrade seems not to care about where the old version's pg_xlog was. You > have

Re: [HACKERS] Log LDAP "diagnostic messages"?

2017-09-14 Thread Michael Paquier
On Fri, Sep 15, 2017 at 1:38 AM, Robert Haas wrote: > On Thu, Sep 14, 2017 at 10:21 AM, Alvaro Herrera > wrote: >> BTW I added --with-ldap and --with-pam to the configure line for the >> reports in coverage.postgresql.org and the % covered in auth.c went from >> 24% to 18.9% (from very bad to ter

Re: [HACKERS] Warnings "unrecognized node type" for some DDLs with log_statement = 'ddl'

2017-09-14 Thread Michael Paquier
On Fri, Sep 15, 2017 at 6:19 AM, Robert Haas wrote: > On Thu, Sep 14, 2017 at 6:02 AM, Ashutosh Sharma > wrote: >> Hmm...I am also able to reproduce the failures reported by you. Your >> patch does solve the problem. Just to confirm if we are still missing >> TAGS for any other Statement nodes,

Re: [JDBC] [HACKERS] Channel binding support for SCRAM-SHA-256

2017-09-14 Thread Michael Paquier
On Fri, Sep 15, 2017 at 1:58 AM, Peter Eisentraut wrote: > Second thoughts, to make things simpler. All we need for channel > binding is a connection flag that says "I require channel binding". It > could be modeled after the sslmode parameter, e.g., cbind=disable (maybe > for debugging), cbind=

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-14 Thread Michael Paquier
On Thu, Sep 14, 2017 at 6:36 PM, Amit Kapila wrote: > Why do we need to change metapage at every place for btree ... I have been hunting for some time places where meta buffers were marked as dirtied and logged. So in the effort, I think that my hands and mind got hotter, forgetting that pd_lower

Re: [HACKERS] SCRAM in the PG 10 release notes

2017-09-14 Thread Michael Paquier
On Thu, Sep 14, 2017 at 3:57 PM, Heikki Linnakangas wrote: > I updated the list of drivers on the wiki > (https://wiki.postgresql.org/wiki/List_of_drivers), adding a column for > whether the driver supports SCRAM authentication. Currently, the only > non-libpq driver that has implemented SCRAM is

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-14 Thread Michael Paquier
On Wed, Sep 13, 2017 at 4:43 PM, Amit Langote wrote: > Sure, no problem. OK, I took a closer look at all patches, but did not run any manual tests to test the compression except some stuff with wal_consistency_checking. + if (opaque->flags & GIN_DELETED) + mask_page_content(page); + el

[HACKERS] Warnings "unrecognized node type" for some DDLs with log_statement = 'ddl'

2017-09-13 Thread Michael Paquier
Hi all, While reviewing another patch, I have bumped into a couple of failures when running installcheck if log_statement = 'ddl'. This pops regression failures for 4 tests: object_address, alter_generic, alter_operator and stats_ext involving commands CREATE STATISTICS and ALTER OPERATOR. You ca

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-13 Thread Michael Paquier
On Wed, Sep 13, 2017 at 2:48 PM, Amit Langote wrote: > I updated the patches so that the metapage's pd_lower is set to the > correct value just before *every* point where we are about to insert a > full page image of the metapage into WAL. That's in addition to doing the > same in various metapag

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-13 Thread Michael Paquier
On Wed, Sep 13, 2017 at 1:12 PM, Bossart, Nathan wrote: > On 9/12/17, 9:47 PM, "Michael Paquier" wrote: >> Those are minor points. The patch seems to be in good shape, and >> passes all my tests, including some pgbench'ing to make sure that >> nothing goe

Re: [HACKERS] pg_rewind proposed scope and interface changes

2017-09-12 Thread Michael Paquier
On Tue, Sep 12, 2017 at 11:52 PM, Chris Travers wrote: > Additionally the wal, xact, timestamp and logical directories must be > processed in some way. To what does the term "logical directories" refer to? > * if --wal=sync the directories are processed the way they are today > * if --wal=cl

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-12 Thread Michael Paquier
On Wed, Sep 13, 2017 at 1:13 PM, Kyotaro HORIGUCHI wrote: > This patch creates a new memory context "Vacuum" under > PortalContext in vacuum.c, but AFAICS the current context there > is PortalHeapMemory, which has the same expected lifetime with > the new context (that is, a child of PotalContext

Re: [HACKERS] [PATCH] Call RelationDropStorage() for broader range of object drops.

2017-09-12 Thread Michael Paquier
On Wed, Sep 13, 2017 at 2:40 AM, Hadi Moshayedi wrote: > Motivation for this patch is that some FDWs (notably, cstore_fdw) try > utilizing PostgreSQL internal storage. PostgreSQL assigns relfilenode's to > foreign tables, but doesn't clean up storage for foreign tables when > dropping tables. Ther

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-12 Thread Michael Paquier
On Wed, Sep 13, 2017 at 12:31 AM, Bossart, Nathan wrote: > Sorry for the spam. I am re-sending these patches with modified names so that > the apply order is obvious to the new automated testing framework (and to > everybody else). - * relid, if not InvalidOid, indicate the relation to process;

Re: [HACKERS] More flexible LDAP auth search filters?

2017-09-12 Thread Michael Paquier
On Wed, Sep 13, 2017 at 8:04 AM, Thomas Munro wrote: > I wonder if there is a reasonable way to indicate or determine whether > you have slapd installed so that check-world could run this test... Module::Install's requires_external_bin is one: http://search.cpan.org/~ether/Module-Install-1.18/lib

Re: [HACKERS] Patches that don't apply or don't compile: 2017-09-12

2017-09-12 Thread Michael Paquier
On Wed, Sep 13, 2017 at 7:39 AM, Daniel Gustafsson wrote: >> On 12 Sep 2017, at 23:54, Tomas Vondra wrote: >> With all due respect, it's hard not to see this as a disruption of the >> current CF. I agree automating the patch processing is a worthwhile >> goal, but we're not there yet and it seems

Re: [JDBC] [HACKERS] Channel binding support for SCRAM-SHA-256

2017-09-12 Thread Michael Paquier
On Tue, Sep 12, 2017 at 11:38 PM, Peter Eisentraut wrote: > It seems we should start by sorting out the mechanism by which the > client can control what authentication mechanisms it accepts. In your > patch set you introduce a connection parameter saslname. I think we > should expand that to non

Re: [HACKERS] Coverage improvements of src/bin/pg_basebackup and pg_receivewal --endpos

2017-09-12 Thread Michael Paquier
On Tue, Sep 12, 2017 at 9:19 PM, Peter Eisentraut wrote: > On 9/11/17 18:21, Michael Paquier wrote: >> On Tue, Sep 12, 2017 at 5:57 AM, Peter Eisentraut >> wrote: >>> I think a more robust way would be to parse >>> Makefile.global, perhaps by a function in Te

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-11 Thread Michael Paquier
On Mon, Sep 11, 2017 at 2:05 PM, Bossart, Nathan wrote: >> + if (i == InvalidAttrNumber) >> + ereport(ERROR, >> + (errcode(ERRCODE_UNDEFINED_COLUMN), >> +errmsg("column \"%s\" of relation \"%s\" does not >> exist", >> +

[HACKERS] Removing wal_keep_segments as default configuration in PostgresNode.pm

2017-09-11 Thread Michael Paquier
Hi all, Right now, PostgresNode.pm uses this set of parameters when initializing a node: print $conf "\n# Added by PostgresNode.pm\n"; print $conf "fsync = off\n"; print $conf "restart_after_crash = off\n"; print $conf "log_line_prefix = '%m [%p] %q%a '\n"; print $conf "log_sta

Re: [HACKERS] Still another race condition in recovery TAP tests

2017-09-11 Thread Michael Paquier
On Mon, Sep 11, 2017 at 11:55 PM, Tom Lane wrote: > Hm, I don't much like having it silently ignore files that are present; > that seems like a foot-gun in the long run. What do you think of the > attached? That looks good to me. I have tried pretty hard to break it, but could not. -- Michael

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2017-09-11 Thread Michael Paquier
On Mon, Sep 11, 2017 at 11:43 PM, Robert Haas wrote: > So I think this is just an excuse for turning --no-security-labels > into --no-object-property=security-label. To me, that's just plain > worse. It does not seem that my thoughts here have been correctly transmitted to your brain. I do not m

Re: [HACKERS] Coverage improvements of src/bin/pg_basebackup and pg_receivewal --endpos

2017-09-11 Thread Michael Paquier
On Tue, Sep 12, 2017 at 5:57 AM, Peter Eisentraut wrote: > I have committed this patch, after a perltidy run, but the way the libz > detection was implemented was a bit too hackish for me, so I have > omitted that for now. Thanks. > I think a more robust way would be to parse > Makefile.global,

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-11 Thread Michael Paquier
On Mon, Sep 11, 2017 at 5:40 PM, Amit Langote wrote: > On 2017/09/10 15:22, Michael Paquier wrote: >> On Sun, Sep 10, 2017 at 3:15 PM, Amit Kapila wrote: >>> On Sat, Sep 9, 2017 at 9:00 PM, Tom Lane wrote: >>>> Michael Paquier writes: >>>>> On

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-11 Thread Michael Paquier
On Mon, Sep 11, 2017 at 4:07 PM, Amit Kapila wrote: > I have prepared separate patches for hash and btree index. I think > for another type of indexes, it is better to first fix the pd_lower > issue. Just wondering (sorry I have not looked at your patch in details)... Have you tested the compres

Re: [JDBC] [HACKERS] Channel binding support for SCRAM-SHA-256

2017-09-10 Thread Michael Paquier
On Mon, Aug 21, 2017 at 9:51 PM, Michael Paquier wrote: > On Tue, Jun 20, 2017 at 1:11 PM, Michael Paquier > wrote: >> With the tests directly in the patch, things are easy to run. WIth >> PG10 stabilization work, of course I don't expect much feedback :) >> But t

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-10 Thread Michael Paquier
On Mon, Sep 11, 2017 at 1:26 AM, Tom Lane wrote: > FWIW, now that we've noticed the discrepancy, I'm for using > REGBUF_STANDARD or equivalent for all metapage calls. Even if it > saves no space, inconsistency is bad because it's confusing. OK, I don't mind having a more aggressive approach, but

Re: [HACKERS] Still another race condition in recovery TAP tests

2017-09-10 Thread Michael Paquier
On Sun, Sep 10, 2017 at 12:38 AM, Tom Lane wrote: > Michael Paquier writes: >> On Sat, Sep 9, 2017 at 1:43 PM, Tom Lane wrote: >>> Yeah, even if we fixed this particular call site, I'm sure the issue >>> would come up again. Certainly we expect hot backups to

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-10 Thread Michael Paquier
On Mon, Sep 11, 2017 at 9:38 AM, Bossart, Nathan wrote: > I agree that it is nice to see when relations are skipped, but I do not > know if the WARNING messages would provide much value for this > particular use case (i.e. 'VACUUM;'). If a user does not provide a list > of tables to VACUUM, they

[HACKERS] Automatic testing of patches in commit fest

2017-09-10 Thread Michael Paquier
Hi all, Thomas Munro has hacked up a prototype of application testing automatically if patches submitted apply and build: http://commitfest.cputube.org/ I would recommend have a look at it from time to time if you are a patch author (or a reviewer) as any failure may say that your patch has rotte

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-10 Thread Michael Paquier
On Sun, Sep 10, 2017 at 3:28 PM, Amit Kapila wrote: > I think here we should first agree on what we want to do. Based on > Tom's comment, I was thinking of changing comments in btree/hash part > and additionally for hash indexes, I can see if we can pass > REGBUF_STANDARD for all usages of metapa

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-09 Thread Michael Paquier
On Sun, Sep 10, 2017 at 3:15 PM, Amit Kapila wrote: > On Sat, Sep 9, 2017 at 9:00 PM, Tom Lane wrote: >> Michael Paquier writes: >>> On Fri, Sep 8, 2017 at 5:24 AM, Tom Lane wrote: >>>> In short, this patch needs a significant rewrite, and more analysis than >

Re: [HACKERS] pg_basebackup fails on Windows when using tablespace mapping

2017-09-09 Thread Michael Paquier
On Sun, Sep 10, 2017 at 12:28 PM, Ashutosh Sharma wrote: > On Tue, Jun 27, 2017 at 6:36 PM, Michael Paquier > wrote: >> On Tue, Jun 27, 2017 at 7:46 PM, Ashutosh Sharma >> wrote: >>> I am still seeing the issue with the attached patch. I had a quick >>> look

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-09 Thread Michael Paquier
On Sat, Sep 9, 2017 at 2:05 AM, Bossart, Nathan wrote: > On 9/8/17, 1:27 AM, "Michael Paquier" wrote: >> Thanks. This looks now correct to me. Except that: >> + ereport(ERROR, >> + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), >> +

Re: [HACKERS] Still another race condition in recovery TAP tests

2017-09-09 Thread Michael Paquier
On Sat, Sep 9, 2017 at 1:43 PM, Tom Lane wrote: > Michael Paquier writes: >>> I'm not real sure if the appropriate answer to this is "we need to fix >>> RecursiveCopy" or "we need to fix the callers to not call RecursiveCopy >>> until the s

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-09 Thread Michael Paquier
On Fri, Sep 8, 2017 at 5:24 AM, Tom Lane wrote: > This means that the premise of this patch is wrong. Adjusting pd_lower, > by itself, would accomplish precisely zip for WAL compression, because > we'd still not be telling the WAL code to compress out the hole. > > To get any benefit, I think we'

Re: [HACKERS] Still another race condition in recovery TAP tests

2017-09-08 Thread Michael Paquier
On Sat, Sep 9, 2017 at 11:32 AM, Tom Lane wrote: > In a moment of idleness I tried to run the TAP tests on pademelon, > which is a mighty old and slow machine. How long did it take? Just wondering if that's actually the slowest one or not to run the full set of recovery tests. This would be might

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-07 Thread Michael Paquier
On Fri, Sep 8, 2017 at 7:27 AM, Bossart, Nathan wrote: > On 9/7/17, 2:33 AM, "Michael Paquier" wrote: >> Using the patch checking for duplicate columns: >> =# create table aa (a int); >> CREATE TABLE >> =# vacuum ANALYZE aa(z, z); >> ERROR: 0A000:

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-07 Thread Michael Paquier
On Thu, Sep 7, 2017 at 9:46 AM, Bossart, Nathan wrote: > I've attached v1 of this patch. I think we might want to refactor the > code for retrieving the relation name from a RangeVar, but it would > probably be better to do that in a separate patch. Using the patch checking for duplicate columns

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-06 Thread Michael Paquier
On Thu, Sep 7, 2017 at 2:40 PM, Amit Langote wrote: > On 2017/09/07 13:09, Michael Paquier wrote: >> pd_lower should remain at 0 on pre-10 servers. > > Doesn't PageInit(), which is where any page gets initialized, has always > set pd_lower to SizeOfPageHeaderData? Y

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-06 Thread Michael Paquier
On Thu, Sep 7, 2017 at 12:59 PM, Tom Lane wrote: > The idea I'd had was to apply the masking only if pd_lower >= > SizeOfPageHeaderData, or if you wanted to be stricter, only if > pd_lower != 0. If putting a check, it seems to me that the stricter one makes the most sense. pd_lower should remain

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-06 Thread Michael Paquier
On Thu, Sep 7, 2017 at 10:42 AM, Amit Langote wrote: > I too tend to think that any users who use this masking facility would > know to expect to get these failures on upgraded clusters with invalid > pd_lower in meta pages. Yes, I don't think that an optimization reducing WAL that impacts all us

Re: [HACKERS] Rewriting the test of pg_upgrade as a TAP test

2017-09-06 Thread Michael Paquier
On Wed, Sep 6, 2017 at 10:05 PM, Peter Eisentraut wrote: > Please send a rebased patch. > > I propose splitting the single Perl script into three separate test > files: one for basic command-line option handling and so on (I would > like to expand that later), one for the main upgrade test, and on

Re: [HACKERS] Rewriting the test of pg_upgrade as a TAP test

2017-09-06 Thread Michael Paquier
On Wed, Sep 6, 2017 at 11:44 PM, Tom Lane wrote: > Alvaro Herrera writes: >> Peter Eisentraut wrote: >>> We also need to have a plan for handling the build farm. Maybe keep the >>> vcregress.pl upgradecheck target as a thin wrapper for the time being? > >> The buildfarm already runs "make check"

Re: [HACKERS] Re: [COMMITTERS] pgsql: pg_rewind: Fix some problems when copying files >2GB.

2017-09-06 Thread Michael Paquier
On Thu, Sep 7, 2017 at 10:11 AM, Peter Eisentraut wrote: > On 9/4/17 06:03, Alvaro Herrera wrote: >> Tom Lane wrote: >>> Michael Paquier writes: >>>> I don't like breaking the abstraction of pg_log() with the existing >>>> flags with some kind o

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-06 Thread Michael Paquier
On Thu, Sep 7, 2017 at 5:49 AM, Tom Lane wrote: > Amit Langote writes: >> On 2017/08/22 9:39, Michael Paquier wrote: >>> On Tue, Jun 27, 2017 at 4:56 PM, Amit Langote >>> wrote: >>>> I updated brin_mask() and spg_mask() in the attached updated patches s

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2017-09-06 Thread Michael Paquier
On Thu, Sep 7, 2017 at 1:43 AM, Robert Haas wrote: > On Wed, Sep 6, 2017 at 12:26 PM, Simon Riggs wrote: I'd personally be fine with --no-whatever for any whatever that might be a subsidiary property of database objects. We've got --no-security-labels, --no-tablespaces, --no-owner

Re: [HACKERS] document and use SPI_result_code_string()

2017-09-06 Thread Michael Paquier
On Thu, Aug 31, 2017 at 11:23 AM, Peter Eisentraut wrote: > A lot of semi-internal code just prints out numeric SPI error codes, > which is not very helpful. We already have an API function > SPI_result_code_string() to convert the codes to a string, so here is a > patch to make more use of that

Re: [HACKERS] CLUSTER command progress monitor

2017-09-06 Thread Michael Paquier
On Wed, Sep 6, 2017 at 3:58 PM, Tatsuro Yamada wrote: > I revised the patch like this: You should avoid top-posting. > I didn't change the name of view (pg_stat_progress_cluster) because I'm not > sure > whether the new name (pg_stat_progress_reorg) is suitable or not. Here are some ideas: rewr

Re: [HACKERS] cache lookup errors for missing replication origins

2017-09-05 Thread Michael Paquier
On Tue, Sep 5, 2017 at 12:59 PM, Michael Paquier wrote: > ERROR: 42704: replication slot "%s" does not exist s/slot/origin/ > As far as I can see, replorigin_by_oid makes no use of its missing_ok > = false in the backend code, so letting it untouched would have no > impa

Re: [HACKERS] assorted code cleanup

2017-09-05 Thread Michael Paquier
On Wed, Sep 6, 2017 at 4:12 AM, Peter Eisentraut wrote: > On 8/21/17 01:11, Michael Paquier wrote: >>> - Drop excessive dereferencing of function pointers >> >> - (*next_ProcessUtility_hook) (pstmt, queryString, >> + next_ProcessU

Re: [HACKERS] [PATCH] Assert that the correct locks are held when calling PageGetLSN()

2017-09-05 Thread Michael Paquier
On Wed, Sep 6, 2017 at 1:15 AM, Jacob Champion wrote: > On Sun, Sep 3, 2017 at 11:14 PM, Michael Paquier > wrote: >> +#if defined(USE_ASSERT_CHECKING) && !defined(FRONTEND) >> +void >> +AssertPageIsLockedForLSN(Page page) >> +{ >> From a design poi

Re: [HACKERS] Coverage improvements of src/bin/pg_basebackup and pg_receivewal --endpos

2017-09-05 Thread Michael Paquier
On Tue, Sep 5, 2017 at 10:19 PM, Peter Eisentraut wrote: > On 6/9/17 02:08, Michael Paquier wrote: >> I have just played with that, and attached is a patch to implement the >> so-said option with a basic set of tests, increasing code coverage of >> pg_receivewal.c from 15% t

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-05 Thread Michael Paquier
On Wed, Sep 6, 2017 at 2:36 AM, Bossart, Nathan wrote: > On 9/4/17, 8:16 PM, "Michael Paquier" wrote: >> So I would tend to think that the same column specified multiple times >> should cause an error, and that we could let VACUUM run work N times >> on a relati

Re: [HACKERS] Proposal for changes to recovery.conf API

2017-09-05 Thread Michael Paquier
On Wed, Sep 6, 2017 at 12:31 AM, Simon Riggs wrote: > I've not worked on this at all, so it isn't ready for re-review and commit. > > I get the "lets do it early" thing and will try to extract a subset in > October. Nice to see that you are still planning to work on that. I would suggest to move

Re: [HACKERS] Proposal: pg_rewind to skip config files

2017-09-05 Thread Michael Paquier
On Tue, Sep 5, 2017 at 9:40 PM, Vladimir Borodin wrote: > We do compress WALs and send them over network. Doing it via archive_command > in single thread is sometimes slower than new WALs are written under heavy > load. Ah, yeah, true. I do use pg_receivexlog --compress for that locally and do a

Re: [HACKERS] Proposal: pg_rewind to skip config files

2017-09-05 Thread Michael Paquier
On Mon, Sep 4, 2017 at 10:38 PM, Alvaro Herrera wrote: > I wonder how portable fnmatch() is in practice (which we don't currently > use anywhere). A shell glob seems a more natural interface to me for > this than a regular expression. On Windows you could use roughly PathMatchSpecEx, but it does

Re: [HACKERS] Proposal: pg_rewind to skip config files

2017-09-05 Thread Michael Paquier
On Tue, Sep 5, 2017 at 7:54 PM, Vladimir Borodin wrote: > 5 сент. 2017 г., в 12:31, Chris Travers > написал(а): > > I think the simplest solution for now is to skip any files ending in .conf, > .log, and serverlog. This is not a portable solution. Users can include configuration files with the n

[HACKERS] cache lookup errors for missing replication origins

2017-09-04 Thread Michael Paquier
Hi all, Cache lookup errors with elog() can be triggered easily by users at SQL level using replication origin functions: =# select pg_replication_origin_advance('popo', '0/1'); ERROR: XX000: cache lookup failed for replication origin 'popo' LOCATION: replorigin_by_name, origin.c:229 =# select p

Re: [HACKERS] [Proposal] Allow users to specify multiple tables in VACUUM commands

2017-09-04 Thread Michael Paquier
On Tue, Sep 5, 2017 at 12:05 PM, Masahiko Sawada wrote: > In get_rel_oids() we often switch the memory context to vac_context > and switch back. As a result almost code in get_rel_oids() is working > in vac_context. Maybe we can switch memory context before and after > the calling get_rel_oids? I

<    1   2   3   4   5   6   7   8   9   10   >