Re: make MaxBackends available in _PG_init

2022-01-25 Thread Bossart, Nathan
On 1/25/22, 12:01 AM, "Michael Paquier" wrote: > So, where are we on this patch? It looks like there is an agreement > that MaxBackends is used widely enough that it justifies the use of a > separate function to set and get a better value computed. There may > be other parameters that could use

Re: Correct error message for end-of-recovery record TLI

2022-01-25 Thread Bossart, Nathan
On 1/24/22, 8:42 PM, "Amul Sul" wrote: > On Tue, Jan 25, 2022 at 10:08 AM Michael Paquier wrote: >> >> On Wed, Dec 01, 2021 at 07:09:34PM +, Bossart, Nathan wrote: >> > The patch no longer applied, so I went ahead and rebased it. >> >> This was

Re: Bug in ProcArrayApplyRecoveryInfo for snapshots crossing 4B, breaking replicas

2022-01-24 Thread Bossart, Nathan
On 1/22/22, 4:43 PM, "Tomas Vondra" wrote: > There's a bug in ProcArrayApplyRecoveryInfo, introduced by 8431e296ea, > which may cause failures when starting a replica, making it unusable. > The commit message for 8431e296ea is not very clear about what exactly > is being done and why, but the

Re: Catalog version access

2022-01-24 Thread Bossart, Nathan
Thanks for taking a look! On 1/23/22, 7:31 PM, "Michael Paquier" wrote: > On Mon, Aug 16, 2021 at 06:12:54PM +0000, Bossart, Nathan wrote: >> I was looking at the --check switch for the postgres binary recently >> [0], and this sounds like something

Re: do only critical work during single-user vacuum?

2022-01-21 Thread Bossart, Nathan
On 1/21/22, 2:43 PM, "John Naylor" wrote: > - to have a simple, easy to type, command AFAICT the disagreement is really just about the grammar. Sawada-san's idea would look something like VACUUM (FREEZE, INDEX_CLEANUP OFF, MIN_XID_AGE 16, MIN_MXID_AGE 16); while your

Re: Avoid erroring out when unable to remove or parse logical rewrite files to save checkpoint work

2022-01-20 Thread Bossart, Nathan
Thanks for your feedback. On 1/20/22, 11:47 AM, "Andres Freund" wrote: > It seems odd to change a bunch of things to not be errors anymore, but then > add new sources of errors. If we try to deal with concurrent deletions or > permission issues - otherwise what's the point of making unlink() not

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-20 Thread Bossart, Nathan
On 1/19/22, 5:15 PM, "James Coleman" wrote: > I'm open to the idea of wordsmithing here, of course, but I strongly > disagree that this is irrelevant data. There are plenty of > optimizations Postgres could theoretically implement but doesn't, so > measuring what should happen by what you think

Re: Add checkpoint and redo LSN to LogCheckpointEnd log message

2022-01-19 Thread Bossart, Nathan
On 1/3/22, 5:52 PM, "Kyotaro Horiguchi" wrote: > It seems to me "LSN" or just "location" is more confusing or > mysterious than "REDO LSN" for the average user. If we want to avoid > being technically too detailed, we would use just "start LSN=%X/%X, > end LSN=%X/%X". And it is equivalent to

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-19 Thread Bossart, Nathan
On 9/24/21, 7:30 AM, "James Coleman" wrote: > When PG11 added the ability for ALTER TABLE ADD COLUMN to set a constant > default value without rewriting the table the doc changes did not note > how the new feature interplayed with ADD COLUMN DEFAULT NOT NULL. > Previously such a new column

Re: Avoid erroring out when unable to remove or parse logical rewrite files to save checkpoint work

2022-01-19 Thread Bossart, Nathan
On 1/19/22, 11:08 AM, "Andres Freund" wrote: > On 2022-01-19 13:34:21 -0500, Tom Lane wrote: >> As far as the patch itself goes, I agree that failure to unlink >> is noncritical, because such a file would have no further effect >> and we can just ignore it. > > I don't agree. We iterate through

Re: do only critical work during single-user vacuum?

2022-01-19 Thread Bossart, Nathan
On 1/19/22, 11:15 AM, "John Naylor" wrote: > This seems to be the motivating reason for wanting new configurability > on the server side. In any case, new knobs are out of scope for this > thread. If the use case is compelling enough, may I suggest starting a > new thread? Sure. Perhaps the new

Re: do only critical work during single-user vacuum?

2022-01-19 Thread Bossart, Nathan
On 1/18/22, 9:47 PM, "Masahiko Sawada" wrote: > IIUC what we want to do here are two things: (1) select only old > tables and (2) set INDEX_CLEANUP = off, TRUNCATE = off, and FREEZE = > on. VACUUM LIMIT statement does both things at the same time. Although > I’m concerned a bit about its

Re: O(n) tasks cause lengthy startups and checkpoints

2022-01-18 Thread Bossart, Nathan
On 1/14/22, 11:26 PM, "Bharath Rupireddy" wrote: > On Sat, Jan 15, 2022 at 12:46 AM Bossart, Nathan wrote: >> I'd personally like to avoid creating two code paths for the same >> thing. Are there really cases when this one extra auxiliary process >> would be

Re: docs: pg_replication_origin_oid() description does not match behaviour

2022-01-18 Thread Bossart, Nathan
On 1/17/22, 5:24 PM, "Michael Paquier" wrote: > On Tue, Jan 18, 2022 at 10:19:41AM +0900, Ian Lawrence Barwick wrote: >> Given that the code has remained unchanged since the function was >> introduced in 9.5, it seems reasonable to change the documentation >> to match the function behaviour

Re: Add sub-transaction overflow status in pg_stat_activity

2022-01-14 Thread Bossart, Nathan
On 1/14/22, 8:26 AM, "Tom Lane" wrote: > Julien Rouhaud writes: >> Like many I previously had to investigate a slowdown due to sub-transaction >> overflow, and even with the information available in a monitoring view (I had >> to rely on a quick hackish extension as I couldn't patch postgres)

Re: O(n) tasks cause lengthy startups and checkpoints

2022-01-14 Thread Bossart, Nathan
On 1/14/22, 3:43 AM, "Maxim Orlov" wrote: > The code seems to be in good condition. All the tests are running ok with no > errors. Thanks for your review. > I like the whole idea of shifting additional checkpointer jobs as much as > possible to another worker. In my view, it is more

Re: Add sub-transaction overflow status in pg_stat_activity

2022-01-13 Thread Bossart, Nathan
Thanks for the new patch! + +Returns a record of information about the backend's subtransactions. +The fields returned are subxact_count identifies +number of active subtransaction and subxact_overflow + shows whether the backend's subtransaction cache is +

Re: do only critical work during single-user vacuum?

2022-01-13 Thread Bossart, Nathan
On 1/13/22, 4:58 AM, "John Naylor" wrote: > On Wed, Jan 12, 2022 at 12:26 PM Bossart, Nathan wrote: >> As I've stated upthread, Sawada-san's suggested approach was my >> initial reaction to this thread. I'm not wedded to the idea of adding >> new options,

Re: Avoid erroring out when unable to remove or parse logical rewrite files to save checkpoint work

2022-01-13 Thread Bossart, Nathan
On 1/12/22, 10:03 PM, "Bharath Rupireddy" wrote: > On Thu, Jan 13, 2022 at 3:47 AM Bossart, Nathan wrote: >> The only feedback I have for the patch is that I don't think the new >> comments are necessary. > > I borrowed the comments as-is from the CheckPointSnapB

Re: parse/analyze API refactoring

2022-01-12 Thread Bossart, Nathan
On 12/28/21, 8:25 AM, "Peter Eisentraut" wrote: > (The "withcb" naming maybe isn't great; better ideas welcome.) FWIW I immediately understood that this meant "with callback," so it might be okay. > Not included in this patch set, but food for further thought: The > pg_analyze_and_rewrite_*()

Re: Avoid erroring out when unable to remove or parse logical rewrite files to save checkpoint work

2022-01-12 Thread Bossart, Nathan
On 12/31/21, 4:44 AM, "Bharath Rupireddy" wrote: > Currently the server is erroring out when unable to remove/parse a > logical rewrite file in CheckPointLogicalRewriteHeap wasting the > amount of work the checkpoint has done and preventing the checkpoint > from finishing. This is unlike

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-12 Thread Bossart, Nathan
On 1/11/22, 11:46 PM, "Masahiko Sawada" wrote: > Regarding the new pg_stat_progress_vacuum_index view, why do we need > to have a separate view? Users will have to check two views. If this > view is expected to be used together with and joined to > pg_stat_progress_vacuum, why don't we provide

Re: do only critical work during single-user vacuum?

2022-01-12 Thread Bossart, Nathan
On 1/12/22, 7:43 AM, "John Naylor" wrote: > On Wed, Jan 12, 2022 at 1:49 AM Masahiko Sawada wrote: >> As another idea, we might be able to add a new option that takes an >> optional integer value, like VACUUM (MIN_XID), VACUUM (MIN_MXID), and >> VACUUM (MIN_XID 50). We vacuum only tables

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-11 Thread Bossart, Nathan
On 1/11/22, 12:33 PM, "Imseih (AWS), Sami" wrote: > What about something like "The number of indexes that are eligible for > vacuuming". > This covers the cases where either an individual index is skipped or the > entire "index vacuuming" phase is skipped. Hm. I don't know if "eligible" is

Re: Add jsonlog log_destination for JSON server logs

2022-01-11 Thread Bossart, Nathan
On 1/10/22, 4:51 AM, "Michael Paquier" wrote: > The issue comes from an incorrect change in syslogger_parseArgs() > where I missed that the incrementation of argv by 3 has no need to be > changed. A build with -DEXEC_BACKEND is enough to show the failure, > which caused a crash when starting up

Re: improve CREATE EXTENSION error message

2022-01-11 Thread Bossart, Nathan
On 1/11/22, 11:23 AM, "Tom Lane" wrote: > "Bossart, Nathan" writes: >> Okay, the message looks like this in v5: > >> postgres=# CREATE EXTENSION does_not_exist; >> ERROR: extension "does_not_exist" is not available >

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-11 Thread Bossart, Nathan
On 1/10/22, 5:01 PM, "Imseih (AWS), Sami" wrote: > I have attached the 3rd revision of the patch which also includes the > documentation changes. Also attached is a rendered html of the docs for > review. > > "max_index_vacuum_cycle_time" has been removed. > "index_rows_vacuumed" renamed to

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2022-01-11 Thread Bossart, Nathan
On 1/11/22, 10:06 AM, "John Naylor" wrote: > I pushed this with one small change -- I felt the comment didn't need > to explain the warning message, since it now simply matches the coding > more exactly. Also, v5 was a big enough change from v4 that I put > Nathan as the first author. Thanks!

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2022-01-10 Thread Bossart, Nathan
I noticed this thread and thought I'd share my experiences building something similar for Multi-AZ DB clusters [0]. It's not a strict RPO mechanism, but it does throttle backends in an effort to keep the replay lag below a configured maximum. I can share the code if there is interest. I wrote

Re: make MaxBackends available in _PG_init

2022-01-10 Thread Bossart, Nathan
On 1/7/22, 12:27 PM, "Robert Haas" wrote: > On Fri, Jan 7, 2022 at 1:09 PM Bossart, Nathan wrote: >> While that approach would provide a way to safely retrieve the value, >> I think it would do little to prevent the issue in practice. If the >> size of the pa

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-10 Thread Bossart, Nathan
On 1/6/22, 6:14 PM, "Imseih (AWS), Sami" wrote: > I am hesitant to make column name changes for obvious reasons, as it breaks > existing tooling. However, I think there is a really good case to change > "index_vacuum_count" as the name is confusing. > "index_vacuum_cycles_completed" is the

Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

2022-01-07 Thread Bossart, Nathan
On 11/23/21, 11:29 AM, "Thomas Munro" wrote: > Here's a patch for Linux and also FreeBSD. The latter OS decided to > turn on ASLR by default recently, causing my workstation to fail like > this quite reliably, which reminded me to follow up with this. It > disables ASLR in pg_ctl and

Re: Disallow quorum uncommitted (with synchronous standbys) txns in logical replication subscribers

2022-01-07 Thread Bossart, Nathan
On 1/6/22, 11:25 PM, "Jeff Davis" wrote: > On Wed, 2022-01-05 at 23:59 -0800, SATYANARAYANA NARLAPURAM wrote: >> I would like to propose a GUC send_Wal_after_quorum_committed which >> when set to ON, walsenders corresponds to async standbys and logical >> replication workers wait until the LSN is

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-01-07 Thread Bossart, Nathan
On 1/7/22, 5:52 AM, "David Steele" wrote: > On 1/6/22 20:20, Euler Taveira wrote: >> On Thu, Jan 6, 2022, at 9:48 PM, Bossart, Nathan wrote: >>> After a quick glance, I didn't see an easy way to hold a session open >>> while the test does other t

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-01-06 Thread Bossart, Nathan
On 12/2/21, 1:34 PM, "Bossart, Nathan" wrote: > On 12/2/21, 9:50 AM, "David Steele" wrote: >> On 12/2/21 12:38, David Steele wrote: >>> On 12/2/21 11:00, Andrew Dunstan wrote: >>>> >>>> Should we really be getting rid of >>>

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2022-01-06 Thread Bossart, Nathan
On 12/21/21, 11:42 AM, "Mark Dilger" wrote: > + /* pre-v9.3 lock-only bit pattern */ > + ereport(ERROR, > + (errcode(ERRCODE_DATA_CORRUPTED), > +errmsg_internal("found tuple with HEAP_XMAX_COMMITTED > and" > +

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-06 Thread Bossart, Nathan
On 12/29/21, 8:44 AM, "Imseih (AWS), Sami" wrote: > In "pg_stat_progress_vacuum", introduce 2 columns: > > * total_index_vacuum : This is the # of indexes that will be vacuumed. Keep > in mind that if failsafe mode kicks in mid-flight to the vacuum, Postgres may > choose to forgo index scans.

Re: skip replication slot snapshot/map file removal during end-of-recovery checkpoint

2022-01-05 Thread Bossart, Nathan
On 12/23/21, 3:17 AM, "Bharath Rupireddy" wrote: > Currently the end-of-recovery checkpoint can be much slower, impacting > the server availability, if there are many replication slot files > .snap or map- to be enumerated and deleted. How about skipping > the .snap and map- file

Re: Pre-allocating WAL files

2022-01-05 Thread Bossart, Nathan
On 12/30/21, 3:52 AM, "Maxim Orlov" wrote: > I did check the patch too and found it to be ok. Check and check-world are > passed. > Overall idea seems to be good in my opinion, but I'm not sure where is the > optimal place to put the pre-allocation. > > On Thu, Dec 30, 2021 at 2:46 PM Pavel

Re: Strange path from pgarch_readyXlog()

2021-12-30 Thread Bossart, Nathan
On 12/29/21, 3:11 PM, "Tom Lane" wrote: > "Bossart, Nathan" writes: >> This crossed my mind, too. I also think one of the arrays can be >> eliminated in favor of just using the heap (after rebuilding with a >> reversed comparator). Here is a minimally-t

Re: Strange path from pgarch_readyXlog()

2021-12-29 Thread Bossart, Nathan
On 12/29/21, 12:22 PM, "Thomas Munro" wrote: > Isn't this a corrupted pathname? > > 2021-12-29 03:39:55.708 CST [79851:1] WARNING: removal of orphan > archive status file > "pg_wal/archive_status/00010003.0028.backup00010004.ready" > failed too many times,

Re: Add index scan progress to pg_stat_progress_vacuum

2021-12-15 Thread Bossart, Nathan
On 12/1/21, 3:02 PM, "Imseih (AWS), Sami" wrote: > The current implementation of pg_stat_progress_vacuum does not > provide progress on which index is being vacuumed making it > difficult for a user to determine if the "vacuuming indexes" phase > is making progress. By exposing which index is

Re: archive modules

2021-12-15 Thread Bossart, Nathan
On 11/2/21, 8:07 AM, "Bossart, Nathan" wrote: > The main motivation is provide a way to archive without shelling out. > This reduces the amount of overhead, which can improve archival rate > significantly. It should also make it easier to archive more safely. > For examp

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-14 Thread Bossart, Nathan
On 12/14/21, 12:09 PM, "Bossart, Nathan" wrote: > On 12/14/21, 9:00 AM, "Bruce Momjian" wrote: >> Have we changed temporary file handling in any recent major releases, >> meaning is this a current problem or one already improved in PG 14. > > I have

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-14 Thread Bossart, Nathan
On 12/14/21, 9:00 AM, "Bruce Momjian" wrote: > Have we changed temporary file handling in any recent major releases, > meaning is this a current problem or one already improved in PG 14. I haven't noticed any recent improvements while working in this area. Nathan

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-13 Thread Bossart, Nathan
On 12/13/21, 12:37 PM, "Robert Haas" wrote: > On Mon, Dec 13, 2021 at 1:21 PM Bossart, Nathan wrote: >> > But against all that, if these tasks are slowing down checkpoints and >> > that's avoidable, that seems pretty important too. Interestingly, I >>

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-13 Thread Bossart, Nathan
On 12/13/21, 6:30 AM, "Dilip Kumar" wrote: > On Tue, Dec 7, 2021 at 11:11 AM Justin Pryzby wrote: >> Since I think this field is usually not interesting to most users of >> pg_stat_activity, maybe this should instead be implemented as a function like >> pg_backend_get_subxact_status(pid). >> >>

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-13 Thread Bossart, Nathan
On 12/13/21, 9:20 AM, "Justin Pryzby" wrote: > On Mon, Dec 13, 2021 at 08:53:37AM -0500, Robert Haas wrote: >> Another issue is that we don't want to increase the number of >> processes without bound. Processes use memory and CPU resources and if >> we run too many of them it becomes a burden on

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-13 Thread Bossart, Nathan
On 12/13/21, 5:54 AM, "Robert Haas" wrote: > I don't know whether this kind of idea is good or not. Thanks for chiming in. I have an almost-complete patch set that I'm hoping to post to the lists in the next couple of days. > One thing we've seen a number of times now is that entrusting the

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Bossart, Nathan
On 12/9/21, 5:27 PM, "Peter Geoghegan" wrote: > I imagine that this new function (to handle maintenance tasks in the > event of a wraparound emergency) would output information about its > progress. For example, it would make an up-front decision about which > tables needed to be vacuumed in

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Bossart, Nathan
On 12/9/21, 5:06 PM, "Bossart, Nathan" wrote: > On 12/9/21, 4:36 PM, "Peter Geoghegan" wrote: >> We could then apply this criteria in new code that implements this >> "big red button" (maybe this is a new option for the postgres >> exe

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Bossart, Nathan
On 12/9/21, 4:36 PM, "Peter Geoghegan" wrote: > We could then apply this criteria in new code that implements this > "big red button" (maybe this is a new option for the postgres > executable, a little like --single?). Something that's reasonably > targeted, and dead simple to use. +1 Nathan

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Bossart, Nathan
On 12/9/21, 12:33 PM, "Bossart, Nathan" wrote: > On 12/9/21, 11:34 AM, "John Naylor" wrote: >> Now that we have a concept of a fail-safe vacuum, maybe it would be >> beneficial to skip a vacuum in single-user mode if the fail-safe >> criteria were not met

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Bossart, Nathan
On 12/9/21, 11:34 AM, "John Naylor" wrote: > When a user must shut down and restart in single-user mode to run > vacuum on an entire database, that does a lot of work that's > unnecessary for getting the system online again, even without > index_cleanup. We had a recent case where a single-user

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-08 Thread Bossart, Nathan
On 12/8/21, 3:29 AM, "Bharath Rupireddy" wrote: > Thanks for your thoughts. I'm fine either way, hence attaching two > patches here with and I will leave it for the committer 's choice. > 1) v1-0001-Add-DB_IN_END_OF_RECOVERY_CHECKPOINT-state-for-co.patch -- > adds new db state

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 8:42 PM, "Bharath Rupireddy" wrote: > On Wed, Dec 8, 2021 at 9:49 AM Bossart, Nathan wrote: >> I think that's alright. The only other small suggestion I have would >> be to say "during end-of-recovery checkpoint" instead of "while in &g

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 5:21 PM, "Bharath Rupireddy" wrote: > On Wed, Dec 8, 2021 at 2:50 AM Bossart, Nathan wrote: >> I noticed that some (but not all) of the surrounding messages say >> "last known up at" the control file time. I'm curious why you chose >&

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 5:21 PM, "Bharath Rupireddy" wrote: > On Wed, Dec 8, 2021 at 4:17 AM Bossart, Nathan wrote: >> I agree with Tom. I would just s/server/backend/ (as per the >> attached) and call it a day. > > Thanks. v5 patch looks good to me. I've marke

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 12:10 AM, "Bharath Rupireddy" wrote: > Here's a patch that I've come up with. Please see if this looks okay > and let me know if we want to take it forward so that I can add a CF > entry. Overall, the patch seems reasonable to me. + case

Re: Pre-allocating WAL files

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 9:35 AM, "Bossart, Nathan" wrote: > On 12/7/21, 12:29 AM, "Bharath Rupireddy" > wrote: >> Why can't the walwriter pre-allocate some of the WAL segments instead >> of a new background process? Of course, it might delay the main >> functi

Re: Alter all tables in schema owner fix

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 2:47 AM, "Greg Nancarrow" wrote: > On Tue, Dec 7, 2021 at 9:01 PM Amit Kapila wrote: >> >> Thanks, the patch looks mostly good to me. I have slightly modified it >> to incorporate one of Michael's suggestions, ran pgindent, and >> modified the commit message. >> > > LGTM, except in

Re: Pre-allocating WAL files

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 12:29 AM, "Bharath Rupireddy" wrote: > Why can't the walwriter pre-allocate some of the WAL segments instead > of a new background process? Of course, it might delay the main > functionality of the walwriter i.e. flush and sync the WAL files, but > having checkpointer do the

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-07 Thread Bossart, Nathan
On 12/6/21, 8:19 PM, "Dilip Kumar" wrote: > If the subtransaction cache is overflowed in some of the transactions > then it will affect all the concurrent queries as they need to access > the SLRU for checking the visibility of each tuple. But currently > there is no way to identify whether in

Re: Do we need pre-allocate WAL files during end-of-recovery checkpoint?

2021-12-06 Thread Bossart, Nathan
On 12/6/21, 4:54 AM, "Bharath Rupireddy" wrote: > The function PreallocXlogFiles doesn't get called during > end-of-recovery checkpoint in CreateCheckPoint, see [1]. The server > becomes operational after the end-of-recovery checkpoint and may need > WAL files. However, I'm not sure how

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-06 Thread Bossart, Nathan
On 12/6/21, 4:34 AM, "Bharath Rupireddy" wrote: > While the database is performing end-of-recovery checkpoint, the > control file gets updated with db state as "shutting down" in > CreateCheckPoint (see the code snippet at [1]) and at the end it sets > it back to "shut down" for a brief moment

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-06 Thread Bossart, Nathan
On 12/6/21, 3:44 AM, "Bharath Rupireddy" wrote: > On Fri, Dec 3, 2021 at 11:50 PM Bossart, Nathan wrote: >> I might hack something together for the separate worker approach, if >> for no other reason than to make sure I really understand how these >> functio

Re: pg_replslotdata - a tool for displaying replication slot information

2021-12-06 Thread Bossart, Nathan
On 12/5/21, 11:10 PM, "Michael Paquier" wrote: > On Thu, Dec 02, 2021 at 08:32:08AM +0530, Bharath Rupireddy wrote: >> On Thu, Dec 2, 2021 at 4:22 AM Andres Freund wrote: I don't have any other compelling use- cases at the moment, but I will say that it is typically nice from an

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-03 Thread Bossart, Nathan
On 12/3/21, 5:57 AM, "Bharath Rupireddy" wrote: > On Fri, Dec 3, 2021 at 3:01 AM Bossart, Nathan wrote: >> >> On 12/1/21, 6:48 PM, "Bharath Rupireddy" >> wrote: >> > +1 for the overall idea of making the checkpoint faster. In fact,

Re: should we document an example to set multiple libraries in shared_preload_libraries?

2021-12-03 Thread Bossart, Nathan
On 12/3/21, 6:21 AM, "Bharath Rupireddy" wrote: > +1 to add here in the "Parameter Names and Values section", but do we > want to backlink every string parameter to this section? I think it > needs more effort. IMO, we can just backlink for > shared_preload_libraries alone. Thoughts? IMO this

Re: Skip vacuum log report code in lazy_scan_heap() if possible

2021-12-03 Thread Bossart, Nathan
On 12/3/21, 7:40 AM, "Peter Geoghegan" wrote: > If my patch to unite vacuum verbose and the autovacuum logging gets > in, then this issue also goes away. Perhaps this patch should be marked Rejected in favor of that one, then. Nathan

Re: Alter all tables in schema owner fix

2021-12-03 Thread Bossart, Nathan
On 12/2/21, 11:57 PM, "tanghy.f...@fujitsu.com" wrote: > Thanks for your patch. > I tested it and it fixed this problem as expected. It also passed "make > check-world". +1, the patch looks good to me, too. My only other suggestion would be to move IsSchemaPublication() to pg_publication.c

Re: Alter all tables in schema owner fix

2021-12-02 Thread Bossart, Nathan
On 12/2/21, 7:07 PM, "vignesh C" wrote: > Currently while changing the owner of ALL TABLES IN SCHEMA > publication, it is not checked if the new owner has superuser > permission or not. Added a check to throw an error if the new owner > does not have superuser permission. > Attached patch has the

Re: Temporary tables versus wraparound... again

2021-12-02 Thread Bossart, Nathan
On 10/12/21, 3:07 PM, "Greg Stark" wrote: > Here's an updated patch. I added some warning messages to autovacuum. I think this is useful optimization, and I intend to review the patch more closely soon. It looks reasonable to me after a quick glance. > One thing I learned trying to debug this

Re: should we document an example to set multiple libraries in shared_preload_libraries?

2021-12-02 Thread Bossart, Nathan
On 12/1/21, 5:59 AM, "Bharath Rupireddy" wrote: > On Wed, Dec 1, 2021 at 6:45 PM Tom Lane wrote: >> Considering the vanishingly small number of actual complaints we've >> seen about this, that sounds ridiculously over-engineered. >> A documentation example should be sufficient. > > Thanks.

Re: Skip vacuum log report code in lazy_scan_heap() if possible

2021-12-02 Thread Bossart, Nathan
On 10/29/21, 10:49 AM, "Bossart, Nathan" wrote: > On 10/29/21, 3:54 AM, "Greg Nancarrow" wrote: >> When recently debugging a vacuum-related issue, I noticed that there >> is some log-report processing/formatting code at the end of >> lazy_scan_heap

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-12-02 Thread Bossart, Nathan
On 12/2/21, 9:50 AM, "David Steele" wrote: > On 12/2/21 12:38, David Steele wrote: >> On 12/2/21 11:00, Andrew Dunstan wrote: >>> >>> Should we really be getting rid of >>> PostgreSQL::Test::Cluster::backup_fs_hot() ? >> >> Agreed, it would be better to update backup_fs_hot() to use exclusive >>

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-02 Thread Bossart, Nathan
On 12/1/21, 6:48 PM, "Bharath Rupireddy" wrote: > +1 for the overall idea of making the checkpoint faster. In fact, we > here at our team have been thinking about this problem for a while. If > there are a lot of files that checkpoint has to loop over and remove, > IMO, that task can be

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-02 Thread Bossart, Nathan
On 12/1/21, 6:06 PM, "Euler Taveira" wrote: > Saying that a certain task is O(n) doesn't mean it needs a separate process to > handle it. Did you have a use case or even better numbers (% of checkpoint / > startup time) that makes your proposal worthwhile? I don't have specific numbers on hand,

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-01 Thread Bossart, Nathan
On 12/1/21, 2:56 PM, "Andres Freund" wrote: > On 2021-12-01 20:24:25 +0000, Bossart, Nathan wrote: >> I realize adding a new maintenance worker might be a bit heavy-handed, >> but I think it would be nice to have somewhere to offload tasks that >> really shouldn't i

Re: Fix inappropriate uses of PG_GETARG_UINT32()

2021-12-01 Thread Bossart, Nathan
On 12/1/21, 10:29 AM, "Peter Eisentraut" wrote: > The attached patch fixes this by accepting the argument using > PG_GETARG_INT32(), doing some checks, and then casting it to unsigned > for the rest of the code. > > The patch also fixes another inappropriate use in an example in the >

O(n) tasks cause lengthy startups and checkpoints

2021-12-01 Thread Bossart, Nathan
Hi hackers, Thanks to 61752af, SyncDataDirectory() can make use of syncfs() to avoid individually syncing all database files after a crash. However, as noted earlier this year [0], there are still a number of O(n) tasks that affect startup and checkpointing that I'd like to improve. Below, I've

Re: SKIP LOCKED assert triggered

2021-12-01 Thread Bossart, Nathan
On 11/12/21, 8:56 AM, "Simon Riggs" wrote: > The combination of these two statements in a transaction hits an > Assert in heapam.c at line 4770 on REL_14_STABLE I've been unable to reproduce this. Do you have any tips for how to do so? Does there need to be some sort of concurrent workload?

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-12-01 Thread Bossart, Nathan
On 12/1/21, 8:27 AM, "David Steele" wrote: > On 11/30/21 18:31, Bossart, Nathan wrote: >> Do you think it's still worth trying to make it safe, or do you think >> we should just remove exclusive mode completely? > > My preference would be to remove it completel

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread Bossart, Nathan
On 11/30/21, 2:58 PM, "David Steele" wrote: > I did figure out how to keep the safe part of exclusive backup (not > having to maintain a connection) while removing the dangerous part > (writing backup_label into PGDATA), but it was a substantial amount of > work and I felt that it had little

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread Bossart, Nathan
On 11/30/21, 2:27 PM, "Tom Lane" wrote: > If we're willing to outright remove it, I don't have any great objection. > My original two cents was that we shouldn't put effort into improving it; > but removing it isn't that. I might try to put a patch together for the January commitfest, given

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread Bossart, Nathan
On 11/30/21, 9:51 AM, "Stephen Frost" wrote: > I disagree that that’s a satisfactory approach. It certainly wasn’t > intended or documented as part of the original feature and therefore > to call it satisfactory strikes me quite strongly as revisionist > history. It looks like the exclusive way

Re: pg_replslotdata - a tool for displaying replication slot information

2021-11-30 Thread Bossart, Nathan
On 11/30/21, 6:14 AM, "Peter Eisentraut" wrote: > On 23.11.21 06:09, Bharath Rupireddy wrote: >> The replication slots data is stored in binary format on the disk under >> the pg_replslot/<> directory which isn't human readable. If >> the server is crashed/down (for whatever reasons) and unable

Re: improve CREATE EXTENSION error message

2021-11-29 Thread Bossart, Nathan
On 11/29/21, 2:32 PM, "Chapman Flack" wrote: > If it were me, I would combine that DETAIL and HINT as one larger HINT, > and use DETAIL for specific details about what actually happened (such > as the exact filename sought and the %m). > > The need for those details doesn't go away; they're still

Re: improve CREATE EXTENSION error message

2021-11-29 Thread Bossart, Nathan
On 11/29/21, 2:13 PM, "Bossart, Nathan" wrote: > Alright, here's v3. In this version, I actually removed the message > about the control file entirely, so now the error message looks like > this: > > postgres=# CREATE EXTENSION does_not_exist; > ERROR

Re: improve CREATE EXTENSION error message

2021-11-29 Thread Bossart, Nathan
On 11/29/21, 1:38 PM, "Chapman Flack" wrote: > On 11/29/21 16:31, Daniel Gustafsson wrote: >> That's a good point, the hint is targeting users who might not even know that >> an extension needs to be physically and separately installed on the machine >> before it can be installed in their

Re: improve CREATE EXTENSION error message

2021-11-29 Thread Bossart, Nathan
On 11/29/21, 1:03 PM, "Tom Lane" wrote: > If we issue the hint only for errno == ENOENT, I think we could be > less wishy-washy (and if it's not ENOENT, the hint is likely > inappropriate anyway). I'm thinking something more like > > HINT: This means the extension is not installed on the

Re: improve CREATE EXTENSION error message

2021-11-29 Thread Bossart, Nathan
On 11/29/21, 12:33 PM, "Daniel Gustafsson" wrote: > I haven't given the suggested wording too much thought, but in general that > sounds like a good idea. Thanks. I'm flexible with the wording. Nathan

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-29 Thread Bossart, Nathan
On 11/26/21, 7:33 AM, "Tom Lane" wrote: > Michael Paquier writes: >> On Thu, Nov 25, 2021 at 06:19:03PM -0800, SATYANARAYANA NARLAPURAM wrote: >>> If we are keeping it then why not make it better? > >> Well, non-exclusive backups are better by design in many aspects, so I >> don't quite see the

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2021-11-29 Thread Bossart, Nathan
On 11/25/21, 9:16 AM, "Mark Dilger" wrote: >> On Nov 24, 2021, at 12:53 PM, Bossart, Nathan wrote: >> >> Another option we might consider is only checking for the >> HEAP_XMAX_LOCK_ONLY bit instead of everything in >> HEAP_XMAX_IS_LOCKED_ONLY. IIUC every

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2021-11-24 Thread Bossart, Nathan
On 11/23/21, 4:59 PM, "Mark Dilger" wrote: >> On Nov 23, 2021, at 4:51 PM, Bossart, Nathan wrote: >> >> This is a good point. Right now, you'd have to manually inspect the >> infomask field to determine the exact form of the invalid combination. >> M

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2021-11-23 Thread Bossart, Nathan
On 11/23/21, 4:36 PM, "Mark Dilger" wrote: > I have to wonder if, when corruption is reported for conditions like this: > > + if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) && > + HEAP_XMAX_IS_LOCKED_ONLY(ctx->tuphdr->t_infomask)) > > if the first thing we're going to

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2021-11-23 Thread Bossart, Nathan
The archives seem unhappy with my attempt to revive this old thread, so here is a link to it in case anyone is looking for more context: https://www.postgresql.org/message-id/flat/3476708e-7919-20cb-ca45-6603470565f7%40amazon.com Nathan

Re: Sequence's value can be rollback after a crashed recovery.

2021-11-23 Thread Bossart, Nathan
On 11/23/21, 1:41 PM, "Tom Lane" wrote: > I wrote: >> I wonder though if we shouldn't try to improve the existing text. >> The phrasing "never rolled back" seems like it's too easily >> misinterpreted. Maybe rewrite the block like >> ... > > A bit of polishing later, maybe like the attached.

Re: Sequence's value can be rollback after a crashed recovery.

2021-11-22 Thread Bossart, Nathan
On 11/22/21, 5:10 AM, "Laurenz Albe" wrote: > On Mon, 2021-11-22 at 15:43 +0800, Andy Fan wrote: >> The performance argument was expected before this writing. If we look at the >> nextval_interval more carefully, we can find it would not flush the xlog >> every >> time even the sequence's

Re: Improving psql's \password command

2021-11-22 Thread Bossart, Nathan
On 11/21/21, 11:15 AM, "Tom Lane" wrote: > "Bossart, Nathan" writes: >> Yeah, I was looking for a way to simplify this, too. Your approach >> seems reasonable enough to me. > > Hearing no contrary opinions, pushed that way. Thanks! Nathan

  1   2   3   4   5   >