RE: Single transaction in the tablesync worker?

2021-02-05 Thread osumi.takami...@fujitsu.com
Hi On Friday, February 5, 2021 5:51 PM Amit Kapila wrote: > On Fri, Feb 5, 2021 at 12:36 PM osumi.takami...@fujitsu.com > wrote: > > > > We need to add some tests to prove the new checks of AlterSubscription() > work. > > I chose TAP tests as we need to set connect = true for the subscription.

Re: pg_replication_origin_drop API potential race condition

2021-02-05 Thread Amit Kapila
On Fri, Feb 5, 2021 at 6:45 PM Euler Taveira wrote: > > On Fri, Feb 5, 2021, at 4:01 AM, Amit Kapila wrote: > > I am not completely whether we should retire replorigin_drop or just > keep it for backward compatibility? What do you think? Anybody else > has any opinion? > > We could certainly keep

Re: Single transaction in the tablesync worker?

2021-02-05 Thread Amit Kapila
On Sat, Feb 6, 2021 at 6:22 AM Peter Smith wrote: > > On Sat, Feb 6, 2021 at 2:10 AM Petr Jelinek > wrote: > > > > > +ReplicationSlotNameForTablesync(Oid suboid, Oid relid, char > > > syncslotname[NAMEDATALEN]) > > > +{ > > > + if (syncslotname) > > > + sprintf(syncslotname, "pg_

Re: Windows regress fails (latest HEAD)

2021-02-05 Thread Alexander Lakhin
Hello hackers, 11.11.2020 04:04, Michael Paquier wrote: > And this configuration matches exactly what you have with the host > where the test passed. > > Now I do see a difference in the Windows 10 build involved, 10.0.19041 > fails but 10.0.18363 passes. I find rather hard to buy that this is > d

Re: [PATCH 1/1] Fix detection of pwritev support for OSX.

2021-02-05 Thread James Hilliard
On Fri, Jan 22, 2021 at 1:53 PM James Hilliard wrote: > > On Thu, Jan 21, 2021 at 11:38 AM Tom Lane wrote: > > > > James Hilliard writes: > > > On Wed, Jan 20, 2021 at 4:07 PM Tom Lane wrote: > > >> I'm not sure that the case of not having the "command line tools" > > >> installed is interestin

Re: Fuzz testing COPY FROM parsing

2021-02-05 Thread David Fetter
On Fri, Feb 05, 2021 at 12:45:30PM +0200, Heikki Linnakangas wrote: > Hi, > > I've been mucking around with COPY FROM lately, and to test it, I wrote some > tools to generate input files and load them with COPY FROM: > > https://github.com/hlinnaka/pgcopyfuzz Neat! The way it's already produced

Re: Correct comment in StartupXLOG().

2021-02-05 Thread Michael Paquier
On Fri, Feb 05, 2021 at 02:42:57PM +0530, Amul Sul wrote: > Alright, that too looks good. Thank you ! Thanks, Amul. I have applied this one. -- Michael signature.asc Description: PGP signature

Re: Key management with tests

2021-02-05 Thread Bruce Momjian
On Fri, Feb 5, 2021 at 05:21:22PM -0500, Stephen Frost wrote: > > I disagree. If we only warn about some parts, attackers will just > > attack other parts. It will also give users a false sense of security. > > If you can get the keys, it doesn't matter if there is one or ten ways > > of gettin

Re: Single transaction in the tablesync worker?

2021-02-05 Thread Peter Smith
On Sat, Feb 6, 2021 at 2:10 AM Petr Jelinek wrote: > > > +ReplicationSlotNameForTablesync(Oid suboid, Oid relid, char > > syncslotname[NAMEDATALEN]) > > +{ > > + if (syncslotname) > > + sprintf(syncslotname, "pg_%u_sync_%u", suboid, relid); > > + else > > + syncslo

Re: Key management with tests

2021-02-05 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Feb 5, 2021 at 01:14:35PM -0500, Stephen Frost wrote: > > > I looked further. First, I don't think we are going to be able to > > > protect at all against users who have _write_ access on the OS running > > > Postgres. It would be t

Re: More test/kerberos tweaks

2021-02-05 Thread Jacob Champion
On Fri, 2021-02-05 at 15:55 -0500, Tom Lane wrote: > We have, almost invariably, regretted it when we tried to use short > timeouts in test cases. That's what I was afraid of. I can work around it easily enough on my local machine, so it's not really a blocker in any sense. That just leaves the f

Re: making update/delete of inheritance trees scale better

2021-02-05 Thread Tom Lane
Robert Haas writes: > As to why it causes us pain, I don't have a full picture of that. > Target list construction is one problem: we build all these target > lists for intermediate notes during planning and they're long enough > -- if the user has a bunch of columns -- and planning is cheap enoug

Re: automatic analyze: readahead - add "IO read time" log message

2021-02-05 Thread Stephen Frost
Greetings, * Heikki Linnakangas (hlinn...@iki.fi) wrote: > On 13/01/2021 23:17, Stephen Frost wrote: > >Would be great to get a review / comments from others as to if there's > >any concerns. I'll admit that it seems reasonably straight-forward to > >me, but hey, I wrote most of it, so that's not

Re: new heapcheck contrib module

2021-02-05 Thread Robert Haas
On Thu, Feb 4, 2021 at 11:10 AM Mark Dilger wrote: > Numbered 0001 in this next patch set. Hi, I committed 0001 as you had it and 0002 with some more cleanups. Things I did: - Adjusted some comments. - Changed processQueryResult so that it didn't do foo(bar) with foo being a pointer. Generally

Re: making update/delete of inheritance trees scale better

2021-02-05 Thread Robert Haas
On Fri, Feb 5, 2021 at 12:06 PM Tom Lane wrote: > You do realize that we're just copying Datums from one level to the > next? For pass-by-ref data, the Datums generally all point at the > same physical data in some disk buffer ... or if they don't, it's > because the join method had a good reason

Re: More test/kerberos tweaks

2021-02-05 Thread Tom Lane
Jacob Champion writes: > The second patch is more of a quality-of-life improvement for devs. On > a failed log match, the test will spin for three minutes before giving > up on the match. I think this is excessive, especially since > interrupting the test with Ctrl-C leaves behind a running KDC da

Re: Key management with tests

2021-02-05 Thread Bruce Momjian
On Fri, Feb 5, 2021 at 01:14:35PM -0500, Stephen Frost wrote: > > I looked further. First, I don't think we are going to be able to > > protect at all against users who have _write_ access on the OS running > > Postgres. It would be too easy to just read process memory, or modify > > ~/.profile.

More test/kerberos tweaks

2021-02-05 Thread Jacob Champion
While working on [1] I ended up running into a couple issues with the Kerberos test suite. Attached are two patches with possible improvements: Some tests check for expected log messages. They currently search through the entire log file, from the beginning, for every match. So if two tests share

Re: Fuzz testing COPY FROM parsing

2021-02-05 Thread Tom Lane
Heikki Linnakangas writes: > That said, I don't think it's important to run the fuzzer in the > buildfarm. It should be enough to do that every once in a while, when > you modify the COPY FROM code (or something else that you want to fuzz > test). But we could easily include the test inputs gen

First-draft release notes for back branches are up

2021-02-05 Thread Tom Lane
See https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=805093113df3f09979cb0e55e857976aad77b8af Please send any corrections by Sunday. regards, tom lane

Re: Fuzz testing COPY FROM parsing

2021-02-05 Thread Stephen Frost
Greetings, * Heikki Linnakangas (hlinn...@iki.fi) wrote: > On 05/02/2021 21:16, Andrew Dunstan wrote: > >On 2/5/21 10:54 AM, Stephen Frost wrote: > >>* Heikki Linnakangas (hlinn...@iki.fi) wrote: > >>>I ran it for about 2 h on my laptop with the patch I was working on [2]. It > >>>didn't find any

Re: New IndexAM API controlling index vacuum strategies

2021-02-05 Thread Peter Geoghegan
On Wed, Feb 3, 2021 at 8:18 PM Masahiko Sawada wrote: > > I'm starting to think that the right short term goal should not > > directly involve bottom-up index deletion. We should instead return to > > the idea of "unifying" the vacuum_cleanup_index_scale_factor feature > > with the INDEX_CLEANUP f

Re: Fuzz testing COPY FROM parsing

2021-02-05 Thread Heikki Linnakangas
On 05/02/2021 21:16, Andrew Dunstan wrote: On 2/5/21 10:54 AM, Stephen Frost wrote: * Heikki Linnakangas (hlinn...@iki.fi) wrote: I ran it for about 2 h on my laptop with the patch I was working on [2]. It didn't find any crashes, but it generated about 1300 input files that it considered "int

Re: Fuzz testing COPY FROM parsing

2021-02-05 Thread Andrew Dunstan
On 2/5/21 10:54 AM, Stephen Frost wrote: > Greetings, > > * Heikki Linnakangas (hlinn...@iki.fi) wrote: >> I've been mucking around with COPY FROM lately, and to test it, I wrote some >> tools to generate input files and load them with COPY FROM: >> >> https://github.com/hlinnaka/pgcopyfuzz > Nea

Re: Key management with tests

2021-02-05 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Wed, Feb 3, 2021 at 01:16:32PM -0500, Bruce Momjian wrote: > > On Wed, Feb 3, 2021 at 10:33:57AM -0500, Stephen Frost wrote: > > > I doubt anyone would actually stipulate that they *guarantee* detection > > > of malicious writes, and I do

Re: [HACKERS] Custom compression methods

2021-02-05 Thread Justin Pryzby
This fails make-check world for me, and CFBOT will say the same. Be sure to compile with --enable-tap-tests cd . && TESTDIR='/home/pryzbyj/src/postgres/src/bin/pg_dump' PATH="/home/pryzbyj/src/postgres/tmp_install/usr/local/pgsql/bin:$PATH" LD_LIBRARY_PATH="/home/pryzbyj/src/postgres/tmp_install

Re: making update/delete of inheritance trees scale better

2021-02-05 Thread Tom Lane
Robert Haas writes: > It's a bit annoying that we percolate things up the tree the way we do > at all. I realize this is far afield from the topic of this thread. > But suppose that I join 5 tables and select a subset of the table > columns in the output. Suppose WLOG the join order is A-B-C-D-E.

Re: Key management with tests

2021-02-05 Thread Bruce Momjian
On Wed, Feb 3, 2021 at 01:16:32PM -0500, Bruce Momjian wrote: > On Wed, Feb 3, 2021 at 10:33:57AM -0500, Stephen Frost wrote: > > I doubt anyone would actually stipulate that they *guarantee* detection > > of malicious writes, and I don't think we should either, but certainly > > the other system

Re: making update/delete of inheritance trees scale better

2021-02-05 Thread Robert Haas
On Thu, Feb 4, 2021 at 10:14 PM Amit Langote wrote: > I guess it would depend on how many of those hidden columns there need > to be (in addition to the existing "ctid" hidden column) and how many > levels of the plan tree they would need to climb through when bubbling > up. My memory is a bit fu

Re: [HACKERS] Custom compression methods

2021-02-05 Thread Robert Haas
On Fri, Feb 5, 2021 at 11:07 AM Robert Haas wrote: > Personally, my preference is to just update the test outputs. It's not > important whether many people look closely to verify the differences; > we just need to look them over on a one-time basis to see if they seem > OK. After that it's 0 effor

Re: [HACKERS] Custom compression methods

2021-02-05 Thread Robert Haas
On Fri, Feb 5, 2021 at 10:56 AM Justin Pryzby wrote: > Could you comment on the patch I sent on Jan 30 ? I think it would be > squished > into 0001. I don't see why we have to do that. Seems fine to have it as a separate patch. > Also, what about the idea to add HIDE_COMPRESSAM ? Right now, y

Re: [HACKERS] Custom compression methods

2021-02-05 Thread Justin Pryzby
+* If the called has passed an invalid slot then create a new slot. *caller +* Slot for storing the modified tuple, incase the targe

Re: Fuzz testing COPY FROM parsing

2021-02-05 Thread Stephen Frost
Greetings, * Heikki Linnakangas (hlinn...@iki.fi) wrote: > I've been mucking around with COPY FROM lately, and to test it, I wrote some > tools to generate input files and load them with COPY FROM: > > https://github.com/hlinnaka/pgcopyfuzz Neat! > I used a fuzz testing tool called honggfuzz [1

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2021-02-05 Thread Mark Rofail
Hello Stephen, > Thanks a lot for your persistence, by the way. > +100 > Hopefully we'll get this in during this cycle and perhaps then you'll work > on something else? :D Thank you for your kind words! Yes, hopefully, we'll get this in this time around. I would definitely love to work on somethi

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2021-02-05 Thread Mark Rofail
Hello Álvaro, Well, *I* think it makes sense to do it that way. I said so three years > ago :-) > https://postgr.es/m/20180410135917.odjh5coa4cjatz5v@alvherre.pgsql So this makes a lot of sense, let's do that. > I wonder if it can usefully get cross-type > operators, i.e., @>>(bigint[],smallin

Re: Single transaction in the tablesync worker?

2021-02-05 Thread Petr Jelinek
Hi, We had a bit high-level discussion about this patches with Amit off-list, so I decided to also take a look at the actual code. My main concern originally was the potential for left-over slots on publisher, but I think the state now is relatively okay, with couple of corner cases that are

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2021-02-05 Thread Stephen Frost
Greetings, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > On 2021-Feb-05, Mark Rofail wrote: > > I disagree -- I think we should get the second patch in, and consider it > > > a requisite for the other one. > > > > I just want to make sure I got your last message right. We should work on > >

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2021-02-05 Thread Alvaro Herrera
Hello Mark On 2021-Feb-05, Mark Rofail wrote: > I disagree -- I think we should get the second patch in, and consider it > > a requisite for the other one. > > I just want to make sure I got your last message right. We should work on > adding the <<@ and @>> operators and their GIN logic as a se

Re: [HACKERS] Custom compression methods

2021-02-05 Thread Dilip Kumar
On Tue, Feb 2, 2021 at 2:45 AM Robert Haas wrote: > > Some more review comments: > > 'git am' barfs on v0001 because it's got a whitespace error. Fixed > VARFLAGS_4B_C() doesn't seem to be used in any of the patches. I'm OK > with keeping it even if it's not used just because maybe someone will

Re: Dumping/restoring fails on inherited generated column

2021-02-05 Thread Peter Eisentraut
On 2021-02-04 01:17, Zhihong Yu wrote: Hi, +           if (attribute->attgenerated && !childatt->attgenerated) +               ereport(ERROR, ... +           if (attribute->attgenerated && childatt->attgenerated) +           { Looks like for the second if statement, checking attribute->attgener

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2021-02-05 Thread Mark Rofail
Hello Álvaro, I disagree -- I think we should get the second patch in, and consider it > a requisite for the other one. I just want to make sure I got your last message right. We should work on adding the <<@ and @>> operators and their GIN logic as a separate patch then submit the FKARRAY as a

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-05 Thread Amit Langote
On Fri, Feb 5, 2021 at 11:01 PM Greg Nancarrow wrote: > On Fri, Feb 5, 2021 at 11:12 PM Amit Langote wrote: > > That seems good enough as far as I am concerned. Although either an > > Assert as follows or a comment why the if (sub_action_ptr) is needed > > seems warranted. > > > > if (sub_actio

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-05 Thread Greg Nancarrow
On Fri, Feb 5, 2021 at 11:12 PM Amit Langote wrote: > > > That seems good enough as far as I am concerned. Although either an > Assert as follows or a comment why the if (sub_action_ptr) is needed > seems warranted. > > if (sub_action_ptr) > rule_action->hasModifyingCTE = true; > else >

Re: There doesn't seem to be any case where PQputCopyEnd() returns 0

2021-02-05 Thread Fujii Masao
On 2021/02/05 16:52, Kasahara Tatsuhito wrote: Hi, The following is written in the comments of PQputCopyEnd(). (snip) * Returns 1 if successful, 0 if data could not be sent (only possible * in nonblock mode), or -1 if an error occurs. (snip) The PQputCopyEnd() section of the manual

Re: Support ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION ... syntax

2021-02-05 Thread japin
On Fri, 05 Feb 2021 at 17:50, Bharath Rupireddy wrote: > On Wed, Feb 3, 2021 at 2:02 PM japin wrote: >> On Wed, 03 Feb 2021 at 13:15, Bharath Rupireddy >> wrote: >> > On Thu, Jan 28, 2021 at 10:07 AM japin wrote: >> >> Attaching v3 patches, please consider these for further review. >> > >> >

Re: pg_replication_origin_drop API potential race condition

2021-02-05 Thread Euler Taveira
On Fri, Feb 5, 2021, at 4:01 AM, Amit Kapila wrote: > I am not completely whether we should retire replorigin_drop or just > keep it for backward compatibility? What do you think? Anybody else > has any opinion? We could certainly keep some code for backward compatibility, however, we have to consi

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2021-02-05 Thread Alvaro Herrera
On 2021-Feb-05, Mark Rofail wrote: > I believe we should start merging these two patches as one, due to the Elem > addon's benefits. such as adding Composite Type support. I disagree -- I think we should get the second patch in, and consider it a requisite for the other one. Let's iron it out fu

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-05 Thread Amit Langote
On Fri, Feb 5, 2021 at 6:56 PM Greg Nancarrow wrote: > On Fri, Feb 5, 2021 at 8:07 PM Amit Langote wrote: > > > This is one reason for my original approach (though I admit, it was > > > not optimal) because at least it was reliable and detected the > > > modifyingCTE after all the rewriting and k

Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2021-02-05 Thread Bharath Rupireddy
Hi, pg_terminate_backend and pg_cancel_backend with postmaster PID produce "PID is not a PostgresSQL server process" warning [1], which basically implies that the postmaster is not a PostgreSQL process at all. This is a bit misleading because the postmaster is the parent of all PostgreSQL pro

Fuzz testing COPY FROM parsing

2021-02-05 Thread Heikki Linnakangas
Hi, I've been mucking around with COPY FROM lately, and to test it, I wrote some tools to generate input files and load them with COPY FROM: https://github.com/hlinnaka/pgcopyfuzz I used a fuzz testing tool called honggfuzz [1] to generate test inputs for COPY FROM. At first I tried to use a

Re: Single transaction in the tablesync worker?

2021-02-05 Thread Ajin Cherian
I did some basic cross-version testing, publisher on PG13 and subscriber on PG14 and publisher on PG14 and subscriber on PG13. Did some basic operations, CREATE, ALTER and STOP subscriptions and it seemed to work fine, no errors. regards, Ajin Cherian Fujitsu Australia.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-05 Thread Greg Nancarrow
On Fri, Feb 5, 2021 at 8:07 PM Amit Langote wrote: > > > > This is one reason for my original approach (though I admit, it was > > not optimal) because at least it was reliable and detected the > > modifyingCTE after all the rewriting and kludgy code had finished. > > Yeah it's hard to go through

Re: Support ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION ... syntax

2021-02-05 Thread Bharath Rupireddy
On Wed, Feb 3, 2021 at 2:02 PM japin wrote: > On Wed, 03 Feb 2021 at 13:15, Bharath Rupireddy > wrote: > > On Thu, Jan 28, 2021 at 10:07 AM japin wrote: > >> Attaching v3 patches, please consider these for further review. > > > > I think we can add a commitfest entry for this feature, so that t

Re: adding wait_start column to pg_locks

2021-02-05 Thread Fujii Masao
On 2021/02/05 0:03, torikoshia wrote: On 2021-02-03 11:23, Fujii Masao wrote: 64-bit fetches are not atomic on some platforms. So spinlock is necessary when updating "waitStart" without holding the partition lock? Also GetLockStatusData() needs spinlock when reading "waitStart"? Also it mi

Re: pg_replication_origin_drop API potential race condition

2021-02-05 Thread Amit Kapila
On Fri, Feb 5, 2021 at 1:50 PM Peter Smith wrote: > > On Fri, Feb 5, 2021 at 6:02 PM Amit Kapila wrote: > > > > On Fri, Feb 5, 2021 at 9:46 AM Peter Smith wrote: > > > > > > PSA patch updated per above suggestions. > > > > > > > Thanks, I have tested your patch and before the patch, I was gettin

Re: Correct comment in StartupXLOG().

2021-02-05 Thread Amul Sul
On Fri, Feb 5, 2021 at 11:53 AM Michael Paquier wrote: > > On Thu, Feb 04, 2021 at 12:58:29PM +0530, Dilip Kumar wrote: > > Looks good to me. > > Rather than using the term "recovery state", I would just use > SharedRecoveryState. This leads me to the attached. Alright, that too looks good. Tha

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-05 Thread Amit Langote
On Fri, Feb 5, 2021 at 4:53 PM Greg Nancarrow wrote: > On Fri, Feb 5, 2021 at 5:21 PM Amit Langote wrote: > > BTW, the original query's cteList is copied into sub_action query but > > not into rule_action for reasons I haven't looked very closely into, > > even though we'd like to ultimately set

Re: Single transaction in the tablesync worker?

2021-02-05 Thread Amit Kapila
On Fri, Feb 5, 2021 at 12:36 PM osumi.takami...@fujitsu.com wrote: > > We need to add some tests to prove the new checks of AlterSubscription() work. > I chose TAP tests as we need to set connect = true for the subscription. > When it can contribute to the development, please utilize this. > I use

Re: pg_replication_origin_drop API potential race condition

2021-02-05 Thread Peter Smith
On Fri, Feb 5, 2021 at 6:02 PM Amit Kapila wrote: > > On Fri, Feb 5, 2021 at 9:46 AM Peter Smith wrote: > > > > PSA patch updated per above suggestions. > > > > Thanks, I have tested your patch and before the patch, I was getting > errors like "tuple concurrently deleted" or "cache lookup failed