Re: perform_spin_delay() vs wait events

2022-11-21 Thread Andres Freund
Hi, On 2022-11-22 00:03:23 +0300, Alexander Korotkov wrote: > On Tue, Nov 22, 2022 at 12:01 AM Andres Freund wrote: > > On November 21, 2022 12:58:16 PM PST, Alexander Korotkov > > wrote: > > >On Mon, Nov 21, 2022 at 2:10 AM Andres Freund wrote: > > >+1 for making a group of individual names

RE: wake up logical workers after ALTER SUBSCRIPTION

2022-11-21 Thread houzj.f...@fujitsu.com
On Tuesday, November 22, 2022 2:49 PM Hayato Kuroda (Fujitsu) > > Dear Nathan, > > > I think you are correct. I did it this way in v2. I've also moved > > the bulk of the logic to logical/worker.c. > > Thanks for updating! It becomes better. Further comments: > > 01. AlterSubscription() >

Re: ps command does not show walsender's connected db

2022-11-21 Thread Michael Paquier
On Thu, Nov 17, 2022 at 01:32:11PM +0900, Ian Lawrence Barwick wrote: > Fujii-san is marked as committer on the commifest entry for this patch [1]; > are you able to go ahead and get it committed? That's the state of the patch since the 11th of October. Seeing the lack of activity, I propose to

RE: wake up logical workers after ALTER SUBSCRIPTION

2022-11-21 Thread Hayato Kuroda (Fujitsu)
Hi Nathan, I have done almost same thing locally for [1], but I thought your code seemed better. Just One comment: IIUC the statement "ALTER SUBSCRIPTION" can be executed inside the transaction. So if two subscriptions are altered in the same transaction, only one of them will awake. Is it

Re: perform_spin_delay() vs wait events

2022-11-21 Thread Michael Paquier
On Mon, Nov 21, 2022 at 07:01:18PM -0800, Andres Freund wrote: > I plan to push this soon unless somebody has further comments. > @@ -146,7 +146,8 @@ typedef enum > WAIT_EVENT_RECOVERY_RETRIEVE_RETRY_INTERVAL, > WAIT_EVENT_REGISTER_SYNC_REQUEST, > WAIT_EVENT_VACUUM_DELAY, > -

Re: proposal: possibility to read dumped table's name from file

2022-11-21 Thread Julien Rouhaud
Hi, On Sun, Nov 13, 2022 at 08:32:47PM +0100, Pavel Stehule wrote: > > updated patch attached Thanks! Some enhancement could probably be done by a native english speaker, but apart from that it looks good to me, so hearing no other complaints I'm marking the CF entry as Ready for Committer!

Re: Improve tab completion for ALTER FUNCTION/PROCEDURE/ROUTINE

2022-11-21 Thread vignesh C
On Tue, 22 Nov 2022 at 05:59, Michael Paquier wrote: > > On Fri, Oct 28, 2022 at 05:34:37PM +0530, vignesh C wrote: > > Those also should be handled, attached v2 version includes the changes > > for the same. > > The basic options supported by PROCEDURE are a subset of ROUTINE with a > difference

Re: Split index and table statistics into different types of stats

2022-11-21 Thread Bharath Rupireddy
On Mon, Nov 21, 2022 at 7:03 PM Drouvot, Bertrand wrote: > > On 11/21/22 12:19 AM, Andres Freund wrote: > > > > That's better, but still seems like quite a bit of repetition, given the > > number of accessors. I think I like my idea of a macro defining the whole > > function a bit better. > > > >

Re: Collation version tracking for macOS

2022-11-21 Thread Jeff Davis
On Sat, 2022-10-22 at 14:22 +1300, Thomas Munro wrote: > Problem 2: If ICU 67 ever decides to report a different version for > a > given collation (would it ever do that? I don't expect so, but ...), > we'd be unable to open the collation with the search-by-collversion > design, and

Re: wake up logical workers after ALTER SUBSCRIPTION

2022-11-21 Thread Thomas Munro
On Tue, Nov 22, 2022 at 1:41 PM Nathan Bossart wrote: > On my machine, the attached patch > improved 'check-world -j8' run time by ~12 seconds (from 3min 8sec to 2min > 56 sec) and src/test/subscription test time by ~17 seconds (from 139 > seconds to 122 seconds). Nice! Maybe a comment to

Re: Proposal: Allow user with pg_monitor role to call pg_stat_reset* functions

2022-11-21 Thread Julien Rouhaud
On Mon, Nov 21, 2022 at 03:47:38PM -0500, Robert Haas wrote: > On Mon, Nov 21, 2022 at 3:45 PM Andres Freund wrote: > > On 2022-11-21 00:16:20 -0800, sirisha chamarthi wrote: > > > At present, calling pg_stat_reset* functions requires super user access > > > unless explicitly grant execute

Re: proposal: possibility to read dumped table's name from file

2022-11-21 Thread Pavel Stehule
út 22. 11. 2022 v 6:26 odesílatel Julien Rouhaud napsal: > Hi, > > On Sun, Nov 13, 2022 at 08:32:47PM +0100, Pavel Stehule wrote: > > > > updated patch attached > > Thanks! > > Some enhancement could probably be done by a native english speaker, but > apart > from that it looks good to me, so

Re: ps command does not show walsender's connected db

2022-11-21 Thread Bharath Rupireddy
On Tue, Nov 22, 2022 at 6:14 AM Michael Paquier wrote: > > On Thu, Nov 17, 2022 at 01:32:11PM +0900, Ian Lawrence Barwick wrote: > > Fujii-san is marked as committer on the commifest entry for this patch [1]; > > are you able to go ahead and get it committed? > > That's the state of the patch

RE: wake up logical workers after ALTER SUBSCRIPTION

2022-11-21 Thread Hayato Kuroda (Fujitsu)
Dear Nathan, > I think you are correct. I did it this way in v2. I've also moved the > bulk of the logic to logical/worker.c. Thanks for updating! It becomes better. Further comments: 01. AlterSubscription() ``` + LogicalRepWorkersWakeupAtCommit(subid); + ``` Currently subids will be

Re: Partial aggregates pushdown

2022-11-21 Thread Ted Yu
On Mon, Nov 21, 2022 at 5:02 PM fujii.y...@df.mitsubishielectric.co.jp < fujii.y...@df.mitsubishielectric.co.jp> wrote: > Hi Mr.Vondra, Mr.Pyhalov, Everyone. > > I discussed with Mr.Pyhalov about the above draft by directly sending mail > to > him(outside of pgsql-hackers). Mr.Pyhalov allowed me

Re: Split index and table statistics into different types of stats

2022-11-21 Thread Drouvot, Bertrand
Hi, On 11/22/22 7:19 AM, Bharath Rupireddy wrote: On Mon, Nov 21, 2022 at 7:03 PM Drouvot, Bertrand wrote: On 11/21/22 12:19 AM, Andres Freund wrote: That's better, but still seems like quite a bit of repetition, given the number of accessors. I think I like my idea of a macro defining the

Re: proposal: possibility to read dumped table's name from file

2022-11-21 Thread Andres Freund
Hi, On 2022-11-13 20:32:47 +0100, Pavel Stehule wrote: > updated patch attached It fails with address sanitizer that's now part of CI: https://cirrus-ci.com/task/6031397744279552?logs=test_world#L2659 [06:33:11.271] # ==31965==ERROR: AddressSanitizer: heap-buffer-overflow on address

wake up logical workers after ALTER SUBSCRIPTION

2022-11-21 Thread Nathan Bossart
Hi hackers, While working on avoiding unnecessary wakeups in logical/worker.c (as was done for walreceiver.c in 05a7be9), I noticed that the tests began taking much longer. This seems to be caused by the reduced frequency of calls to maybe_reread_subscription() in LogicalRepApplyLoop().

Re: wake up logical workers after ALTER SUBSCRIPTION

2022-11-21 Thread Nathan Bossart
On Tue, Nov 22, 2022 at 03:16:05PM +1300, Thomas Munro wrote: > Maybe a comment to explain why a single variable is enough? This crossed my mind shortly after sending my previous message. Looking closer, I see that several types of ALTER SUBSCRIPTION do not call PreventInTransactionBlock(), so a

Re: Proposal: Allow user with pg_monitor role to call pg_stat_reset* functions

2022-11-21 Thread Bharath Rupireddy
On Tue, Nov 22, 2022 at 2:15 AM Andres Freund wrote: > > Hi, > > On 2022-11-21 00:16:20 -0800, sirisha chamarthi wrote: > > At present, calling pg_stat_reset* functions requires super user access > > unless explicitly grant execute permission on those. In this thread, I am > > proposing to grant

FOR UPDATE may leave spurious locks

2022-11-21 Thread Kyotaro Horiguchi
Hello. I noticed that $SUBJECT. "spurious" here means the locks on the rows that are not seemingly qualified by the query condition (that is, EPQ failure). It doesn't seem to be a bug to me (or it seems just inevitable.). But that doesn't seems to be described either in the doc. If I'm right

Re: [BUG] FailedAssertion in SnapBuildPurgeOlderTxn

2022-11-21 Thread Amit Kapila
On Tue, Nov 22, 2022 at 2:22 AM Andres Freund wrote: > > On 2022-11-21 15:47:12 +0300, Maxim Orlov wrote: > > After some investigation, I think, the problem is in the snapbuild.c > > (commit 272248a0c1b1, see [0]). We do allocate InitialRunningXacts > > array in the context of builder->context,

Generate pg_stat_get_* functions with Macros

2022-11-21 Thread Drouvot, Bertrand
Hi hackers, Please find attached a patch proposal to $SUBJECT. The idea has been proposed by Andres in [1] and can be seen as preparatory work for [1]. The patch introduces 2 new Macros, PGSTAT_DEFINE_REL_INT64_FIELD_ACCESSOR and PGSTAT_DEFINE_REL_TSTZ_FIELD_ACCESSOR. For some functions

Re: Bug in row_number() optimization

2022-11-21 Thread Richard Guo
On Wed, Nov 16, 2022 at 7:38 AM Sergey Shinderuk wrote: > The failing query is: > SELECT * FROM >(SELECT *, >count(salary) OVER (PARTITION BY depname || '') c1, -- w1 >row_number() OVER (PARTITION BY depname) rn, -- w2 >count(*) OVER (PARTITION BY depname)

Re: Error-safe user functions

2022-11-21 Thread Michael Paquier
On Mon, Nov 21, 2022 at 12:26:45AM -0500, Tom Lane wrote: > Corey Huinker writes: >> I'm making an attempt at this or something very similar to it. I don't yet >> have a patch ready. Nice to hear that. If a WIP or a proof of concept takes more than a few hours, how about beginning a new thread

Re: Add 64-bit XIDs into PostgreSQL 15

2022-11-21 Thread Chris Travers
On Mon, Nov 21, 2022 at 12:25 PM Aleksander Alekseev < aleksan...@timescale.com> wrote: > Hi hackers, > > > > I have a very serious concern about the current patch set. as someone > who has faced transaction id wraparound in the past. > > > > [...] > > > > I had a similar stance when I started

Re: Add 64-bit XIDs into PostgreSQL 15

2022-11-21 Thread Chris Travers
On Mon, Nov 21, 2022 at 10:40 AM Pavel Borisov wrote: > > I have a very serious concern about the current patch set. as someone > who has faced transaction id wraparound in the past. > > > > I can start by saying I think it would be helpful (if the other issues > are approached reasonably) to

Re: wake up logical workers after ALTER SUBSCRIPTION

2022-11-21 Thread Nathan Bossart
On Tue, Nov 22, 2022 at 03:03:52AM +, Hayato Kuroda (Fujitsu) wrote: > Just One comment: IIUC the statement "ALTER SUBSCRIPTION" can be executed > inside the transaction. So if two subscriptions are altered in the same > transaction, only one of them will awake. Is it expected behavior? > > I

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-21 Thread Peter Smith
Thanks for addressing my review comments on v47-0001. Here are my review comments for v49-0001. == src/backend/replication/logical/applyparallelworker.c 1. GENERAL - NULL checks There is inconsistent NULL checking in the patch. Sometimes it is like (!winfo) Sometimes explicit NULL checks

Re: Introduce a new view for checkpointer related stats

2022-11-21 Thread Drouvot, Bertrand
Hi, On 11/17/22 1:51 PM, Bharath Rupireddy wrote: Hi, pg_stat_bgwriter view currently reports checkpointer stats as well. It is that way because historically checkpointer was part of bgwriter until the commits 806a2ae and bf405ba, that went into PG 9.2, separated them out. I think it is time

Re: Improve tab completion for ALTER FUNCTION/PROCEDURE/ROUTINE

2022-11-21 Thread Michael Paquier
On Fri, Oct 28, 2022 at 05:34:37PM +0530, vignesh C wrote: > Those also should be handled, attached v2 version includes the changes > for the same. The basic options supported by PROCEDURE are a subset of ROUTINE with a difference of COST, IMMUTABLE, [NOT] LEAKPROOF, ROWS, STABLE and VOLATILE.

Re: missing indexes in indexlist with partitioned tables

2022-11-21 Thread Andres Freund
Hi, On 2022-11-02 01:50:38 +, Arne Roland wrote: > I mainly changed the comments, the Assert and some casing. The tests have been failing for a while https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/40/3452 https://api.cirrus-ci.com/v1/task/6190372803051520/logs/cores.log

Re: perform_spin_delay() vs wait events

2022-11-21 Thread Andres Freund
Hi, On 2022-11-22 12:51:25 +0900, Michael Paquier wrote: > On Mon, Nov 21, 2022 at 07:01:18PM -0800, Andres Freund wrote: > > I plan to push this soon unless somebody has further comments. > > > @@ -146,7 +146,8 @@ typedef enum > > WAIT_EVENT_RECOVERY_RETRIEVE_RETRY_INTERVAL, > >

RE: wake up logical workers after ALTER SUBSCRIPTION

2022-11-21 Thread Takamichi Osumi (Fujitsu)
On Tuesday, November 22, 2022 1:39 PM Nathan Bossart wrote: > On Tue, Nov 22, 2022 at 03:03:52AM +, Hayato Kuroda (Fujitsu) wrote: > > Just One comment: IIUC the statement "ALTER SUBSCRIPTION" can be > > executed inside the transaction. So if two subscriptions are altered > > in the same

Re: proposal: possibility to read dumped table's name from file

2022-11-21 Thread Pavel Stehule
út 22. 11. 2022 v 8:39 odesílatel Andres Freund napsal: > Hi, > > On 2022-11-13 20:32:47 +0100, Pavel Stehule wrote: > > updated patch attached > > It fails with address sanitizer that's now part of CI: > > https://cirrus-ci.com/task/6031397744279552?logs=test_world#L2659 > > [06:33:11.271] #

Re: TAP output format in pg_regress

2022-11-21 Thread Dagfinn Ilmari Mannsåker
Andres Freund writes: > But either way, it seems nicer to output the # inside a helper function? Note that the helper function should inject '# ' at the start of every line in the message, not just the first line. It might also be worth having two separate functions: one that prints to stdout,

Re: createuser doesn't tell default settings for some options

2022-11-21 Thread Daniel Gustafsson
> On 10 Aug 2022, at 10:28, Daniel Gustafsson wrote: > >> On 10 Aug 2022, at 08:12, Kyotaro Horiguchi wrote: >> >> (I suppose this is a pg15 issue) >> >> createuser --help shows the following help text. >> >>> --bypassrls role can bypass row-level security (RLS) policy >>>

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-11-21 Thread Maxim Orlov
> Pushed. > > -- > With Regards, > Amit Kapila. > > Hi! While working on 64–bit XID's patch set, I stumble into problems with contrib/test_decoding/catalog_change_snapshot test [0]. AFAICS, the problem is not related to the 64–bit XID's patch set and the problem is in InitialRunningXacts array,

Re: Operation log for major operations

2022-11-21 Thread Dmitry Koval
Thanks for references, Justin! Couple comments about these references. 1) "Make unlogged table resets detectable". https://www.postgresql.org/message-id/flat/62750df5b126e1d8ee039a79ef3cc64ac3d47cd5.camel%40j-davis.com This conversation is about specific problem (unlogged table repairing).

Re: Catalog_xmin is not advanced when a logical slot is lost

2022-11-21 Thread Alvaro Herrera
On 2022-Nov-21, Ashutosh Bapat wrote: > Maybe. In that case pg_get_replication_slots() should be changed. We > should use the same criteria to decide whether a slot is invalidated > or not at all the places. Right. > I am a fan of stricter, all-assumption-covering conditions. In case we > don't

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Robert Haas
On Mon, Nov 21, 2022 at 7:22 AM Alvaro Herrera wrote: > On 2022-Nov-21, Jakub Wartak wrote: > > b) I was wondering about creating a new wait class "Planner" with the > > event "ReadingMinMaxIndex" (or similar). The obvious drawback is the > > naming/categorization as wait events are ... well.. as

Re: Allow single table VACUUM in transaction block

2022-11-21 Thread Simon Riggs
On Fri, 18 Nov 2022 at 11:54, Simon Riggs wrote: > > On Thu, 17 Nov 2022 at 20:00, Justin Pryzby wrote: > > > > On Wed, Nov 16, 2022 at 05:14:07PM -0500, Greg Stark wrote: > > > I think this requesting autovacuum worker should be a distinct > > > command. Or at least an explicit option to

Re: Split index and table statistics into different types of stats

2022-11-21 Thread Drouvot, Bertrand
Hi, On 11/21/22 12:19 AM, Andres Freund wrote: Hi, On 2022-11-18 12:18:38 +0100, Drouvot, Bertrand wrote: On 11/16/22 9:12 PM, Andres Freund wrote: This still leaves a fair bit of boilerplate. ISTM that the function body really should just be a single line. Might even be worth defining the

Re: [PATCH] Allow specification of custom slot for custom nodes

2022-11-21 Thread Pavel Borisov
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested I've looked at this patch and don't see any problems with it. It is

Re: Allow single table VACUUM in transaction block

2022-11-21 Thread Simon Riggs
On Fri, 18 Nov 2022 at 18:26, Tom Lane wrote: > > Robert Haas writes: > > On Fri, Nov 18, 2022 at 7:04 AM Simon Riggs > > wrote: > >> So if consistency is also a strong requirement, then maybe we should > >> make that new command the default, i.e. make VACUUM always just a > >> request to

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Tom Lane
Robert Haas writes: > I don't think this is safe at all. Wait events can only bracket > individual operations, like the reads of the individual index blocks, > not report on which phase of a larger operation is in progress. If we > try to make them do the latter, we will have a hot mess on our

Proposal: Allow user with pg_monitor role to call pg_stat_reset* functions

2022-11-21 Thread sirisha chamarthi
Hi Hackers, At present, calling pg_stat_reset* functions requires super user access unless explicitly grant execute permission on those. In this thread, I am proposing to grant execute on them to users with pg_monitor role permissions. This comes handy to the monitoring users (part of pg_monitor

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-11-21 Thread Alexander Pyhalov
Justin Pryzby писал 2022-11-21 06:00: I finally found time to digest and integrate your changes into my local branch. This fixes the three issues you reported: FORCE_RELEASE, issue with INVALID partitions issue (for which I adapted your patch into an earlier patch in my series), and progress

Re: Reducing power consumption on idle servers

2022-11-21 Thread Simon Riggs
On Mon, 21 Nov 2022 at 08:40, Laurenz Albe wrote: > > On Mon, 2022-11-21 at 07:36 +, Simon Riggs wrote: > > On Mon, 21 Nov 2022 at 05:07, Laurenz Albe wrote: > > > > > > On Mon, 2022-11-21 at 10:13 +1300, Thomas Munro wrote: > > > > I'll wait 24 hours before committing, to > > > > provide a

Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Jakub Wartak
Hi hackers, the case of planner's src/backend/utils/adt/selfuncs.c:get_actual_variable_endpoint() spending literally seconds seems to be well known fact across hackers (in the extreme wild case it can be over 1+ hour on VLDBs). For those unfamiliar it is planner estimation that tries to read real

Re: Catalog_xmin is not advanced when a logical slot is lost

2022-11-21 Thread Alvaro Herrera
On 2022-Nov-21, Ashutosh Bapat wrote: > I think the condition should be > > if (!XLogRecPtrIsInvalid(invalidated_at_lsn)) LSN and XID are > different data types. Yeah, this bit is wrong. I agree with your suggestion to just keep a boolean flag, as we don't need more than that. > and to be

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Alvaro Herrera
On 2022-Nov-21, Jakub Wartak wrote: > b) I was wondering about creating a new wait class "Planner" with the > event "ReadingMinMaxIndex" (or similar). The obvious drawback is the > naming/categorization as wait events are ... well.. as the name "WAIT" > implies, while those btree lookups could

Re: Catalog_xmin is not advanced when a logical slot is lost

2022-11-21 Thread Ashutosh Bapat
On Mon, Nov 21, 2022 at 5:39 PM Alvaro Herrera wrote: > > On 2022-Nov-21, Ashutosh Bapat wrote: > > > I think the condition should be > > > > if (!XLogRecPtrIsInvalid(invalidated_at_lsn)) LSN and XID are > > different data types. > > Yeah, this bit is wrong. I agree with your suggestion to just

Re: Catalog_xmin is not advanced when a logical slot is lost

2022-11-21 Thread Alvaro Herrera
On 2022-Nov-20, sirisha chamarthi wrote: > Hi Hackers, > > forking this thread from the discussion [1] as suggested by Amit. > > Catalog_xmin is not advanced when a logical slot is invalidated (lost) > until the invalidated slot is dropped. This patch ignores invalidated slots > while computing

Re: New docs chapter on Transaction Management and related changes

2022-11-21 Thread Alvaro Herrera
Agreed on not using "unaborted", per previous discussion. On 2022-Nov-21, Laurenz Albe wrote: > Perhaps we should also avoid the term "transaction block". Even without > speaking > of a "block", way too many people confuse PL/pgSQL's BEGIN ... END blocks > with transactions. On the other

Re: Operation log for major operations

2022-11-21 Thread Justin Pryzby
See also prior discussions: https://www.postgresql.org/message-id/flat/62750df5b126e1d8ee039a79ef3cc64ac3d47cd5.camel%40j-davis.com https://www.postgresql.org/message-id/flat/20180228214311.jclah37cwh572t2c%40alap3.anarazel.de -- Justin

Re: Catalog_xmin is not advanced when a logical slot is lost

2022-11-21 Thread Ashutosh Bapat
Hi Sirisha, Thanks for identifying the bug and the solution. Some review comments inlined. On Mon, Nov 21, 2022 at 2:49 PM Alvaro Herrera wrote: > > On 2022-Nov-20, sirisha chamarthi wrote: > > > Hi Hackers, > > > > forking this thread from the discussion [1] as suggested by Amit. > > > >

RE: Perform streaming logical transactions by background workers and parallel apply

2022-11-21 Thread houzj.f...@fujitsu.com
On Monday, November 21, 2022 2:26 PM Peter Smith wrote: > On Fri, Nov 18, 2022 at 6:03 PM Peter Smith > wrote: > > > > Here are some review comments for v47-0001 > > > > (This review is a WIP - I will post more comments for this patch next > > week) > > > > Here are the rest of my comments for

Re: ExecRTCheckPerms() and many prunable partitions

2022-11-21 Thread Amit Langote
On Mon, Nov 21, 2022 at 9:03 PM Amit Langote wrote: > On Thu, Nov 10, 2022 at 8:58 PM Alvaro Herrera > wrote: > > Why do callers of add_rte_to_flat_rtable() have to modify the rte's > > perminfoindex themselves, instead of having the function do it for them? > > That looks strange. But also

Re: Getting rid of SQLValueFunction

2022-11-21 Thread Michael Paquier
On Sun, Nov 20, 2022 at 03:15:34PM -0800, Ted Yu wrote: > + * timestamp. These require a specific handling with their typmod is given > + * by the function caller through their SQL keyword. > > typo: typmod is given -> typmod given > > Other than the above, code looks good to me. Thanks for

Re: Logical Replication Custom Column Expression

2022-11-21 Thread Ashutosh Bapat
On Sat, Nov 19, 2022 at 6:47 PM Stavros Koureas wrote: > > Hi all, > > Working with PostgreSQL Logical Replication is just great! It helps a lot > doing real time replication for analytical purposes without using any other > 3d party service. Although all these years working as product

RE: Perform streaming logical transactions by background workers and parallel apply

2022-11-21 Thread houzj.f...@fujitsu.com
On Friday, November 18, 2022 8:36 AM Masahiko Sawada wrote: > > Here are review comments on v47-0001 and v47-0002 patches: Thanks for the comments! > When the parallel apply worker exited, I got the following server log. > I think this log is not appropriate since the worker was not

[BUG] FailedAssertion in SnapBuildPurgeOlderTxn

2022-11-21 Thread Maxim Orlov
Hi! PREAMBLE For a last couple of months, I stumbled into a problem while running tests on ARM (Debain, aarch64) and some more wired platforms for my 64–bit XIDs patch set. Test contrib/test_decoding (catalog_change_snapshot) rarely failed with the next message: TRAP:

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-21 Thread John Naylor
On Mon, Nov 21, 2022 at 3:43 PM Masahiko Sawada wrote: > > On Mon, Nov 21, 2022 at 4:20 PM John Naylor > wrote: > > Assuming the smallest node is fixed size (i.e. fanout/capacity member not part of the common set, so only part of variable-sized nodes), 3 has a nice property: no wasted padding

Re: Polyphase merge is obsolete

2022-11-21 Thread Peter Eisentraut
On 21.11.22 00:57, Heikki Linnakangas wrote: On 19/11/2022 13:00, Peter Eisentraut wrote: On 18.10.21 14:15, Heikki Linnakangas wrote: On 05/10/2021 20:24, John Naylor wrote: I've had a chance to review and test out the v5 patches. Thanks! I fixed the stray reference to PostgreSQL 14 that

Re: Add 64-bit XIDs into PostgreSQL 15

2022-11-21 Thread Pavel Borisov
> I have a very serious concern about the current patch set. as someone who has > faced transaction id wraparound in the past. > > I can start by saying I think it would be helpful (if the other issues are > approached reasonably) to have 64-bit xids, but there is an important piece > of

Re: New docs chapter on Transaction Management and related changes

2022-11-21 Thread Laurenz Albe
On Fri, 2022-11-18 at 14:28 -0500, Bruce Momjian wrote: > New patch attached. Thanks. > --- a/doc/src/sgml/ref/release_savepoint.sgml > +++ b/doc/src/sgml/ref/release_savepoint.sgml > + RELEASE SAVEPOINT releases the named savepoint and > + all active savepoints that were created after the

Cleanup: Duplicated, misplaced comment in HeapScanDescData

2022-11-21 Thread Matthias van de Meent
Hi, I noticed that the comment on/beneath rs_numblocks in HeapScanDescData is duplicated above rs_strategy. I don't know if there should have been a different comment above rs_strategy, but the current one is definitely out of place, so I propose to remove it as per attached. The comment was

Re: Add 64-bit XIDs into PostgreSQL 15

2022-11-21 Thread Aleksander Alekseev
Hi hackers, > > I have a very serious concern about the current patch set. as someone who > > has faced transaction id wraparound in the past. > > [...] > > I had a similar stance when I started working on this patch. Of > course, it seemed horrible just to postpone the consequences of >

Re: Cleanup: Duplicated, misplaced comment in HeapScanDescData

2022-11-21 Thread Matthias van de Meent
On Mon, 21 Nov 2022 at 12:12, Matthias van de Meent wrote: > > Hi, > > I noticed that the comment on/beneath rs_numblocks in HeapScanDescData > is duplicated above rs_strategy. I don't know if there should have > been a different comment above rs_strategy, but the current one is > definitely out

Re: Patch: Global Unique Index

2022-11-21 Thread Simon Riggs
On Thu, 17 Nov 2022 at 22:01, Cary Huang wrote: > > Patch: Global Unique Index Let me start by expressing severe doubt on the usefulness of such a feature, but also salute your efforts to contribute. > In other words, a global unique index and a regular partitioned index are > essentially the

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

2022-11-21 Thread Naeem Akhter
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested i've tested and verified the documentation.

Re: Reducing power consumption on idle servers

2022-11-21 Thread Laurenz Albe
On Mon, 2022-11-21 at 07:36 +, Simon Riggs wrote: > On Mon, 21 Nov 2022 at 05:07, Laurenz Albe wrote: > > > > On Mon, 2022-11-21 at 10:13 +1300, Thomas Munro wrote: > > > I'll wait 24 hours before committing, to > > > provide a last chance for anyone who wants to complain about dropping > >

Operation log for major operations

2022-11-21 Thread Dmitry Koval
Hi, hackers! It is important for customer support to know what system operations (pg_resetwal, pg_rewind, pg_upgrade, ...) have been executed on the database. A variant of implementation of the log for system operations (operation log) is attached to this email. Introduction. -

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-21 Thread Masahiko Sawada
On Mon, Nov 21, 2022 at 4:20 PM John Naylor wrote: > > > On Fri, Nov 18, 2022 at 2:48 PM I wrote: > > One issue with this patch: The "fanout" member is a uint8, so it can't hold > > 256 for the largest node kind. That's not an issue in practice, since we > > never need to grow it, and we only

Re: Reducing power consumption on idle servers

2022-11-21 Thread Laurenz Albe
On Mon, 2022-11-21 at 11:42 +0530, Bharath Rupireddy wrote: > On Mon, Nov 21, 2022 at 10:37 AM Laurenz Albe > wrote: > > > > On Mon, 2022-11-21 at 10:13 +1300, Thomas Munro wrote: > > > I'll wait 24 hours before committing, to > > > provide a last chance for anyone who wants to complain about

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Jakub Wartak
Hi, Draft version of the patch attached (it is based on Simon's) I would be happier if we could make that #define into GUC (just in case), although I do understand the effort to reduce the number of various knobs (as their high count causes their own complexity). -Jakub Wartak. On Mon, Nov 21,

Re: Reducing power consumption on idle servers

2022-11-21 Thread Robert Haas
On Mon, Nov 21, 2022 at 3:40 AM Laurenz Albe wrote: > We have had a deprecation period? I looked at the documentation, but found > no mention of a deprecation. How hard can it be to leave the GUC and only > poll for the existence of the file if it is set? > > I personally don't need the GUC,

Re: Sending SIGABRT to child processes (was Re: Strange failure on mamba)

2022-11-21 Thread Tom Lane
I wrote: > Andres Freund writes: >> I suspect that having a GUC would be a good idea. I needed something similar >> recently, debugging an occasional hang in the AIO patchset. I first tried >> something like your #define approach and it did cause a problematic flood of >> core files. > Yeah, the

postgres_fdw binary protocol support

2022-11-21 Thread Ilya Gladyshev
Hi everyone, I have made a patch that introduces support for libpq binary protocol in postgres_fdw. The idea is simple, when a user knows that the foreign server is binary compatible with the local and his workload could somehow benefit from using binary protocol, it can be switched on for a

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Tom Lane
Robert Haas writes: > Is there any reason to tie this into page costs? I'd be more inclined > to just make it a hard limit on the number of pages. I think that > would be more predictable and less prone to surprising (bad) behavior. Agreed, a simple limit of N pages fetched seems appropriate. >

Re: allowing for control over SET ROLE

2022-11-21 Thread Robert Haas
On Sat, Nov 19, 2022 at 1:00 AM Michael Paquier wrote: > On Fri, Nov 18, 2022 at 04:19:15PM -0500, Robert Haas wrote: > > Fixed that, and the other mistake Álvaro spotted, and also bumped > > catversion because I forgot that earlier. > > I was looking at this code yesterday, to see today that

Re: Catalog_xmin is not advanced when a logical slot is lost

2022-11-21 Thread Alvaro Herrera
On 2022-Nov-21, sirisha chamarthi wrote: > > > I am a fan of stricter, all-assumption-covering conditions. In case we > > > don't want to check restart_lsn, an Assert might be useful to validate > > > our assumption. > > > > Agreed. I'll throw in an assert. > > Changed this in the patch to

Re: Catalog_xmin is not advanced when a logical slot is lost

2022-11-21 Thread sirisha chamarthi
On Mon, Nov 21, 2022 at 8:05 AM Alvaro Herrera wrote: > On 2022-Nov-21, sirisha chamarthi wrote: > > > > > I am a fan of stricter, all-assumption-covering conditions. In case > we > > > > don't want to check restart_lsn, an Assert might be useful to > validate > > > > our assumption. > > > > > >

Re: Reducing power consumption on idle servers

2022-11-21 Thread Tom Lane
Robert Haas writes: > The reason that I pushed back -- not as successfully as I would have > liked -- on the changes to pg_stop_backup / pg_start_backup is that I > know there are people using the old method successfully, and it's not > just a 1:1 substitution. Here I don't, and it is. I'm

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Justin Pryzby
This patch version runs "continue" unconditionally (rather than conditionally, like the previous version). if (!index_fetch_heap(index_scan, tableslot)) continue; /* no visible tuple, try next index entry */

Re: Catalog_xmin is not advanced when a logical slot is lost

2022-11-21 Thread sirisha chamarthi
Thanks Alvaro, Ashutosh for your comments. On Mon, Nov 21, 2022 at 6:20 AM Alvaro Herrera wrote: > On 2022-Nov-21, Ashutosh Bapat wrote: > > > Maybe. In that case pg_get_replication_slots() should be changed. We > > should use the same criteria to decide whether a slot is invalidated > > or

Re: heavily contended lwlocks with long wait queues scale badly

2022-11-21 Thread Jonathan S. Katz
On 11/20/22 2:56 PM, Andres Freund wrote: Hi, On 2022-11-09 17:03:13 -0800, Andres Freund wrote: On 2022-11-09 09:38:08 -0800, Andres Freund wrote: I'm on a hike, without any connectivity, Thu afternoon - Sun. I think it's OK to push it to HEAD if I get it done in the next few hours. Bigger

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Simon Riggs
On Mon, 21 Nov 2022 at 15:23, Robert Haas wrote: > > On Mon, Nov 21, 2022 at 10:14 AM Simon Riggs > wrote: > > > > What we need is a solution that avoids reading an unbounded number of > > > > tuples under any circumstances. I previously suggested using > > > > SnapshotAny here, but Tom didn't

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Robert Haas
On Mon, Nov 21, 2022 at 10:32 AM Tom Lane wrote: > Robert Haas writes: > > Is there any reason to tie this into page costs? I'd be more inclined > > to just make it a hard limit on the number of pages. I think that > > would be more predictable and less prone to surprising (bad) behavior. > >

Re: perform_spin_delay() vs wait events

2022-11-21 Thread Robert Haas
On Sun, Nov 20, 2022 at 6:10 PM Andres Freund wrote: > I was wondering about that too - but decided against it because it would only > show a single wait event. And wouldn't really describe spinlocks as a whole, > just the "extreme" delays. If we wanted to report the spin waits more > granular,

Re: when the startup process doesn't (logging startup delays)

2022-11-21 Thread Robert Haas
On Sun, Nov 20, 2022 at 9:20 PM Kyotaro Horiguchi wrote: > I prefer Robert's approach as it is more robust for future changes and > simple. I prefer to avoid this kind of piggy-backing and it doesn't > seem to be needed in this case. XLogShutdownWalRcv() looks like a > similar case to me and

Re: Catalog_xmin is not advanced when a logical slot is lost

2022-11-21 Thread Alvaro Herrera
On 2022-Nov-21, sirisha chamarthi wrote: > On Mon, Nov 21, 2022 at 8:05 AM Alvaro Herrera > wrote: > > Thank you. I had pushed mine for CirrusCI to test, and it failed the > > assert I added in slot.c: > > https://cirrus-ci.com/build/4786354503548928 > > Not yet sure why, looking into it. > >

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Simon Riggs
On Mon, 21 Nov 2022 at 15:01, Tom Lane wrote: > > > What we need is a solution that avoids reading an unbounded number of > > tuples under any circumstances. I previously suggested using > > SnapshotAny here, but Tom didn't like that. I'm not sure if there are > > safety issues there or if Tom

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Robert Haas
On Mon, Nov 21, 2022 at 10:14 AM Simon Riggs wrote: > > > What we need is a solution that avoids reading an unbounded number of > > > tuples under any circumstances. I previously suggested using > > > SnapshotAny here, but Tom didn't like that. I'm not sure if there are > > > safety issues there

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Andres Freund
Hi, On 2022-11-21 17:06:16 +0100, Jakub Wartak wrote: > @@ -6213,14 +6216,26 @@ get_actual_variable_endpoint(Relation heapRel, > /* Fetch first/next tuple in specified direction */ > while ((tid = index_getnext_tid(index_scan, indexscandir)) != NULL) > { > +

Re: Catalog_xmin is not advanced when a logical slot is lost

2022-11-21 Thread sirisha chamarthi
On Mon, Nov 21, 2022 at 9:12 AM Alvaro Herrera wrote: > On 2022-Nov-21, sirisha chamarthi wrote: > > > On Mon, Nov 21, 2022 at 8:05 AM Alvaro Herrera > > wrote: > > > > Thank you. I had pushed mine for CirrusCI to test, and it failed the > > > assert I added in slot.c: > > >

Re: pgsql: Prevent instability in contrib/pageinspect's regression test.

2022-11-21 Thread Tom Lane
I wrote: > Andres Freund writes: >> Looks like a chunk of the buildfarm doesn't like this - presumably because >> they use force_parallel_mode = regress. Seems ok to just force that to off in >> this test? > Ugh ... didn't occur to me to try that. I'll take a look. Hmm, so the problem is:

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Robert Haas
On Mon, Nov 21, 2022 at 12:30 PM Andres Freund wrote: > This can't quite be right - isn't this only applying the limit if we found a > visible tuple? It doesn't look that way to me, but perhaps I'm just too dense to see the problem? -- Robert Haas EDB: http://www.enterprisedb.com

Re: Damage control for planner's get_actual_variable_endpoint() runaway

2022-11-21 Thread Tom Lane
Andres Freund writes: > This can't quite be right - isn't this only applying the limit if we found a > visible tuple? What it's restricting is the number of heap page fetches, which might be good enough. We don't have a lot of visibility here into how many index pages were scanned before

  1   2   >