Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-06-16 Thread Dilip Kumar
()log_newpage()On Thu, Jun 17, 2021 at 3:28 AM Andres Freund wrote: > > Hi, > > On 2021-06-15 16:50:24 +0530, Dilip Kumar wrote: > > The patch modifies both CREATE DATABASE and ALTER DATABASE..SET > > TABLESPACE to be fully WAL-logged. > > Generally quite a bit in favor of this - the current

Re: Fix for segfault in logical replication on master

2021-06-16 Thread Mark Dilger
> On Jun 16, 2021, at 10:19 PM, osumi.takami...@fujitsu.com wrote: > > I started to analyze your report and > will reply after my idea to your modification is settled. Thank you. — Mark Dilger EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Skip partition tuple routing with constant partition key

2021-06-16 Thread Amit Langote
Hi, Thanks for reading the patch. On Thu, Jun 17, 2021 at 1:46 PM Zhihong Yu wrote: > On Wed, Jun 16, 2021 at 9:29 PM Amit Langote wrote: >> Attached a slightly revised version of that patch, with a commit >> message this time. > > + int n_tups_inserted; > + int

RE: Fix for segfault in logical replication on master

2021-06-16 Thread osumi.takami...@fujitsu.com
On Thursday, June 17, 2021 1:31 PM Mark Dilger wrote: > In commit e7eea52b2d, you introduced a new function, > RelationGetIdentityKeyBitmap(), which uses some odd logic for determining > if a relation has a replica identity index. That code segfaults under certain > conditions. A test case to

Re: pgbench logging broken by time logic changes

2021-06-16 Thread Kyotaro Horiguchi
At Thu, 17 Jun 2021 12:23:42 +0900, Yugo NAGATA wrote in > On Wed, 16 Jun 2021 21:11:41 +0200 (CEST) > Fabien COELHO wrote: > > I cannot say that I'm thrilled by having multiple tv stuff back in several > > place. I can be okay with one, though. What about the attached? Does it > > make

Re: Decoding speculative insert with toast leaks memory

2021-06-16 Thread Amit Langote
On Thu, Jun 17, 2021 at 12:56 PM Amit Kapila wrote: > On Wed, Jun 16, 2021 at 8:18 PM Tom Lane wrote: > > > > Amit Kapila writes: > > > Pushed! > > > > skink reports that this has valgrind issues: > > > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink=2021-06-15%2020%3A49%3A26 >

Re: Skip partition tuple routing with constant partition key

2021-06-16 Thread Zhihong Yu
On Wed, Jun 16, 2021 at 9:29 PM Amit Langote wrote: > On Wed, Jun 16, 2021 at 4:27 PM Amit Langote > wrote: > > On Mon, Jun 7, 2021 at 8:38 PM houzj.f...@fujitsu.com > > wrote: > > > The rough idea is to check the average batch number every 1000 rows. > > > If the average batch num is greater

Fix for segfault in logical replication on master

2021-06-16 Thread Mark Dilger
Hi Amit, In commit e7eea52b2d, you introduced a new function, RelationGetIdentityKeyBitmap(), which uses some odd logic for determining if a relation has a replica identity index. That code segfaults under certain conditions. A test case to demonstrate that is attached. Prior to patching

Re: Skip partition tuple routing with constant partition key

2021-06-16 Thread Amit Langote
On Wed, Jun 16, 2021 at 4:27 PM Amit Langote wrote: > On Mon, Jun 7, 2021 at 8:38 PM houzj.f...@fujitsu.com > wrote: > > The rough idea is to check the average batch number every 1000 rows. > > If the average batch num is greater than 1, then we enable the cache check, > > if not, disable cache

Re: Added schema level support for publication.

2021-06-16 Thread vignesh C
On Wed, Jan 20, 2021 at 6:27 PM Rahila Syed wrote: > > Hi Vignesh, > >> >> I have handled the above scenario(drop schema should automatically >> remove the schema entry from publication schema relation) & addition >> of tests in the new v2 patch attached. >> Thoughts? > > > Please see some

Re: pgbench bug candidate: negative "initial connection time"

2021-06-16 Thread Yugo NAGATA
On Wed, 16 Jun 2021 20:25:31 +0200 (CEST) Fabien COELHO wrote: > > Hello Yugo-san, > > >> When connection break while the bench has already started, > >> maybe it makes more sense to proceed, > > > > The result would be incomplete also in this case. However, the reason why > > it is worth to

Re: Decoding speculative insert with toast leaks memory

2021-06-16 Thread Amit Kapila
On Wed, Jun 16, 2021 at 8:18 PM Tom Lane wrote: > > Amit Kapila writes: > > Pushed! > > skink reports that this has valgrind issues: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink=2021-06-15%2020%3A49%3A26 > The problem happens at line: rel_sync_cache_relation_cb() { .. if

Re: Improving isolationtester's data output

2021-06-16 Thread Tom Lane
Michael Paquier writes: > Nobody has complained about the removal of --dry-run with 13~. The > second one would cause tests to fail after a minor upgrade for > extensions using isolationtester, but it seems like a good thing to > inform people about anyway. So, okay, both parts are done.

Re: pgbench logging broken by time logic changes

2021-06-16 Thread Yugo NAGATA
On Wed, 16 Jun 2021 21:11:41 +0200 (CEST) Fabien COELHO wrote: > > > pg_time_now(). This uses INSTR_TIME_SET_CURRENT in it, but this macro > > can call clock_gettime(CLOCK_MONOTONIC[_RAW], ) instead of gettimeofday > > or clock_gettime(CLOCK_REALTIME, ). When CLOCK_MONOTONIC[_RAW] is used, > >

RE: locking [user] catalog tables vs 2pc vs logical rep

2021-06-16 Thread osumi.takami...@fujitsu.com
On Wednesday, June 16, 2021 7:21 PM Amit Kapila wrote: > On Mon, Jun 14, 2021 at 5:33 PM osumi.takami...@fujitsu.com > wrote: > > > > On Friday, June 11, 2021 2:13 PM vignesh C > wrote: > > > > Attached the patch-set that addressed those two comments. > > > > Few minor comments: > 1. > +

Re: Improving isolationtester's data output

2021-06-16 Thread Michael Paquier
On Wed, Jun 16, 2021 at 09:10:25PM -0400, Tom Lane wrote: > Yeah, it only results in a message in the output file anyway. That itself would blow up the buildfarm, as 06fdc4e has proved. > Yes please. Nobody has complained about the removal of --dry-run with 13~. The second one would cause

Re: Centralizing protective copying of utility statements

2021-06-16 Thread Zhihong Yu
On Wed, Jun 16, 2021 at 6:40 PM Tom Lane wrote: > I haven't pushed my quick-hack fix for bug #17053 ([1]) because > I wasn't really satisfied with band-aiding that problem in one > more place. I took a look around to see if we could protect > against the whole class of

Re: Unresolved repliaction hang and stop problem.

2021-06-16 Thread Kyotaro Horiguchi
At Wed, 16 Jun 2021 18:28:28 -0400, Alvaro Herrera wrote in > On 2021-Jun-16, Ha Ka wrote: > # Children Self Command Shared Object Symbol > > # . > .. > # >100.00% 0.00%

Re: Teaching users how they can get the most out of HOT in Postgres 14

2021-06-16 Thread Peter Geoghegan
On Sun, May 30, 2021 at 6:30 PM Masahiko Sawada wrote: > We need to accept "yes" and "no" too? Currently, the parsing of a > boolean type reloption accepts those words. Added those in the attached revision, version 2. This is much closer to being commitable than v1 was. I plan on committing this

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-06-16 Thread vignesh C
On Wed, Jun 16, 2021 at 3:51 PM Amit Kapila wrote: > > On Mon, Jun 14, 2021 at 5:33 PM osumi.takami...@fujitsu.com > wrote: > > > > On Friday, June 11, 2021 2:13 PM vignesh C wrote: > > > > Attached the patch-set that addressed those two comments. > > > > Few minor comments: > 1. > + >

Centralizing protective copying of utility statements

2021-06-16 Thread Tom Lane
I haven't pushed my quick-hack fix for bug #17053 ([1]) because I wasn't really satisfied with band-aiding that problem in one more place. I took a look around to see if we could protect against the whole class of scribble-on-a-utility-statement issues in a more centralized way. What I found is

Re: A qsort template

2021-06-16 Thread Thomas Munro
On Thu, Jun 17, 2021 at 1:14 PM Tom Lane wrote: > The big problem in my mind, which would not be alleviated in the > slightest by having a separate file, is that it'd be easy to miss > removing entries if they ever become obsolete. I suppose you could invent some kind of declaration syntax in a

Re: Different compression methods for FPI

2021-06-16 Thread Michael Paquier
On Wed, Jun 16, 2021 at 01:17:26PM +0500, Andrey Borodin wrote: > I agree that allowing just lz4 - is already a huge step ahead. Yeah, I am tempted to just add LZ4 as a first step as the patch footprint would be minimal, and we could come back to zstd once we have more feedback from the field, if

Re: A qsort template

2021-06-16 Thread Tom Lane
Thomas Munro writes: > Perhaps one day we could add a > secondary file, not updated by that mechanism, that holds a manually > maintained list for cases like this. Yeah, the comments in pgindent already speculate about that. For now, those include and exclude lists are short enough that keeping

Re: Different compression methods for FPI

2021-06-16 Thread Michael Paquier
On Wed, Jun 16, 2021 at 11:49:51AM +0300, Heikki Linnakangas wrote: > Hmm, do we currently compress each block in a WAL record separately, for > records that contain multiple full-page images? That could make a big > difference e.g. for GiST index build that WAL-logs 32 pages in each record. > If

Re: Improving isolationtester's data output

2021-06-16 Thread Tom Lane
Michael Paquier writes: > On Wed, Jun 16, 2021 at 03:33:29PM -0400, Tom Lane wrote: >> After checking cross-version diffs to see how painful that is likely >> to be, I'm inclined to also back-patch Michael's v13 commits >> 989d23b04beac0c26f44c379b04ac781eaa4265e >> Detect unused steps in

Re: A qsort template

2021-06-16 Thread Thomas Munro
On Thu, Jun 17, 2021 at 11:40 AM Tom Lane wrote: > Thomas Munro writes: > > Hmm, well it was only recently damaged by commit def5b065, and that's > > because I'd forgotten to put ST_ELEMENT_TYPE into typedefs.list, and I > > was correcting that in this patch. > > If ST_ELEMENT_TYPE isn't

Re: Added schema level support for publication.

2021-06-16 Thread vignesh C
On Wed, Jun 16, 2021 at 4:59 PM Amit Kapila wrote: > > On Sat, Jan 9, 2021 at 8:08 PM Bharath Rupireddy > wrote: > > > > And, with this feature, since there can be many huge tables inside a > > schema, the initial table sync phase of the replication can take a > > while. > > > > Say a user has

Re: pgbench logging broken by time logic changes

2021-06-16 Thread Michael Paquier
On Thu, Jun 17, 2021 at 12:36:10PM +1200, Thomas Munro wrote: > For that reason, I'm not super attached to that new pg_time_usec_t > stuff at all, and wouldn't be sad if we reverted that piece. I am > moderately attached to the sync changes, though. pgbench 13 is > objectively producing

Re: pgbench logging broken by time logic changes

2021-06-16 Thread Thomas Munro
Hi Greg, On Thu, Jun 17, 2021 at 2:49 AM Gregory Smith wrote: > Back on March 10 Thomas Munro committed and wrestled multiple reworks of the > pgbench code from Fabien and the crew. The feature to synchronize startup > I'm looking forward to testing now that I have a packaged beta.

Re: PoC: Using Count-Min Sketch for join cardinality estimation

2021-06-16 Thread Tomas Vondra
On 6/17/21 1:31 AM, John Naylor wrote: > On Wed, Jun 16, 2021 at 12:23 PM Tomas Vondra > mailto:tomas.von...@enterprisedb.com>> > wrote: > > > The attached patch is a very simple (and perhaps naive) implementation > > adding count-min sketch to pg_statistic for all attributes with a hash > >

Re: Improving isolationtester's data output

2021-06-16 Thread Michael Paquier
On Wed, Jun 16, 2021 at 03:33:29PM -0400, Tom Lane wrote: > After checking cross-version diffs to see how painful that is likely > to be, I'm inclined to also back-patch Michael's v13 commits > > 989d23b04beac0c26f44c379b04ac781eaa4265e > Detect unused steps in isolation specs and do some

Re: A qsort template

2021-06-16 Thread Tom Lane
Thomas Munro writes: > Hmm, well it was only recently damaged by commit def5b065, and that's > because I'd forgotten to put ST_ELEMENT_TYPE into typedefs.list, and I > was correcting that in this patch. If ST_ELEMENT_TYPE isn't recognized as a typedef by the buildfarm's typedef collectors, this

Re: PoC: Using Count-Min Sketch for join cardinality estimation

2021-06-16 Thread John Naylor
On Wed, Jun 16, 2021 at 12:23 PM Tomas Vondra wrote: > The attached patch is a very simple (and perhaps naive) implementation > adding count-min sketch to pg_statistic for all attributes with a hash > function (as a new statistics slot kind), and considering it in > equijoinsel_inner. There's a

Re: Outdated replication protocol error?

2021-06-16 Thread Andres Freund
Hi, On 2021-06-16 15:59:11 -0700, Jeff Davis wrote: > [ digging up old thread ] > > It seems everyone agrees that the current behavior is strange. Yea. I don't remember the details, but I've also hit this problem since in some odd circumstance while reviewing the "logical decoding on standbys"

Re: Split xlog.c

2021-06-16 Thread Andres Freund
Hi, On 2021-06-16 16:30:45 +0300, Heikki Linnakangas wrote: > xlog.c is very large. We've split off some functions from it over the years, > but it's still large and it keeps growing. > > Attached is a proposal to split functions related to WAL replay, standby > mode, fetching files from archive,

Re: Outdated replication protocol error?

2021-06-16 Thread Jeff Davis
On Fri, 2021-01-15 at 13:55 -0800, Andres Freund wrote: > > > We should either: > > > > > > 1. Document that IDENTIFY_SYSTEM must always be run before > > > START_REPLICATION, and always issue a WARNING if that's not done > > > (an > > > ERROR might break existing applications); or > > > > > >

Re: Question about StartLogicalReplication() error path

2021-06-16 Thread Jeff Davis
On Tue, 2021-06-15 at 15:19 +0900, Kyotaro Horiguchi wrote: > I don't think the message is neded, but I don't oppose it as far as > the level is LOG and the messages were changed as something like > this: > > > - elog(DEBUG1, "cannot stream from %X/%X, minimum is > %X/%X,

Re: Unresolved repliaction hang and stop problem.

2021-06-16 Thread Alvaro Herrera
On 2021-Jun-16, Ha Ka wrote: > Here is the upload with generated reports: https://easyupload.io/p38izx > passwd: johS5jeewo OK, so I downloaded that and this is the interesting entry in the profile for the broken case: # Samples: 5K of event 'cpu-clock' # Event count (approx.): 59989898390 # #

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-06-16 Thread Tomas Vondra
On 6/15/21 3:31 PM, Andrew Dunstan wrote: On 6/15/21 8:04 AM, Heikki Linnakangas wrote: Yeah, WAL-logging the contents of the source database would certainly be less weird than the current system. As Julien also pointed out, the question is, are there people using on "CREATE DATABASE foo

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-06-16 Thread Andres Freund
Hi, On 2021-06-15 18:11:23 +0530, Dilip Kumar wrote: > On Tue, Jun 15, 2021 at 5:34 PM Heikki Linnakangas wrote: > > > > On 15/06/2021 14:20, Dilip Kumar wrote: > > > Design Idea: > . Then > > > we can get the relfilenode of every file we need to copy, and prepare > > > a list of all such

Re: a path towards replacing GEQO with something better

2021-06-16 Thread John Naylor
On Wed, Jun 16, 2021 at 12:01 PM Robert Haas wrote: > > I feel like these are completely equivalent. Either way, the planner > is going to deduce that all the ".col" columns are equal to each other > via the equivalence class machinery, and then the subsequent planning > will be absolutely

Re: A qsort template

2021-06-16 Thread Zhihong Yu
On Wed, Jun 16, 2021 at 2:54 PM Thomas Munro wrote: > Hi Zhihong, > > On Thu, Jun 17, 2021 at 8:13 AM Zhihong Yu wrote: > > In 0001-Add-bsearch-and-unique-templates-to-sort_template.h.patch : > > > > - const ST_ELEMENT_TYPE * > ST_SORT_PROTO_ARG); > > +

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-06-16 Thread Andres Freund
Hi, On 2021-06-15 16:50:24 +0530, Dilip Kumar wrote: > The patch modifies both CREATE DATABASE and ALTER DATABASE..SET > TABLESPACE to be fully WAL-logged. Generally quite a bit in favor of this - the current approach is very heavyweight, slow and I think we have a few open corner bugs related

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-16 Thread Tomas Vondra
On 6/16/21 4:23 PM, Tomas Vondra wrote: On 6/16/21 2:36 PM, Alexander Pyhalov wrote: Hi. It seems this commit commit b676ac443b6a83558d4701b2dd9491c0b37e17c4 Author: Tomas Vondra Date:   Fri Jun 11 20:19:48 2021 +0200 Optimize creation of slots for FDW bulk inserts has broken batch

Re: A qsort template

2021-06-16 Thread Thomas Munro
Hi Zhihong, On Thu, Jun 17, 2021 at 8:13 AM Zhihong Yu wrote: > In 0001-Add-bsearch-and-unique-templates-to-sort_template.h.patch : > > - const ST_ELEMENT_TYPE * > ST_SORT_PROTO_ARG); > + const ST_ELEMENT_TYPE >

Re: Replication protocol doc fix

2021-06-16 Thread Tom Lane
Jeff Davis writes: > One thing I don't fully understand is what would happen if the client > issued the Sync as the *first* message in an extended-protocol series. That'd cause the backend to send ReadyForQuery, which'd likely confuse the client. > But I think you're correct in saying that the

Re: Replication protocol doc fix

2021-06-16 Thread Jeff Davis
On Mon, 2021-06-14 at 10:51 -0400, Robert Haas wrote: > but if > > > you use the extended query protocol, then the result is a > > > hopeless > > > mess, because the protocol is badly designed: > > > After looking in more detail, I think I understand a bit better. Clients don't differentiate

Re: Python 3.10 breaks regression tests with traceback changes

2021-06-16 Thread Tom Lane
Peter Eisentraut writes: > Patch applied to master. I suppose we should backpatch this. Python 3.10 has evidently now percolated into Fedora Rawhide, thus caiman is failing in the back branches. Time for that back-patch. regards, tom lane

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 12:22 PM Andres Freund wrote: > I think it's more complicated than that - "before" isn't a guarantee when the > horizon can go backwards. Consider the case where a hot_standby_feedback=on > replica without a slot connects - that can result in the xid horizon going >

Re: A qsort template

2021-06-16 Thread Zhihong Yu
On Tue, Jun 15, 2021 at 10:55 PM Thomas Munro wrote: > On Mon, Mar 15, 2021 at 1:09 PM Thomas Munro > wrote: > > On Sun, Mar 14, 2021 at 5:03 PM Zhihong Yu wrote: > > > + * Remove duplicates from an array. Return the new size. > > > + */ > > > +ST_SCOPE size_t > > > +ST_UNIQUE(ST_ELEMENT_TYPE

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Matthias van de Meent
On Wed, 16 Jun 2021 at 21:22, Andres Freund wrote: > > Hi, > > On 2021-06-16 09:46:07 -0700, Peter Geoghegan wrote: > > On Wed, Jun 16, 2021 at 9:03 AM Peter Geoghegan wrote: > > > On Wed, Jun 16, 2021 at 3:59 AM Matthias van de Meent > > > > So the implicit assumption in heap_page_prune that >

Re: Support for NSS as a libpq TLS backend

2021-06-16 Thread Daniel Gustafsson
> On 16 Jun 2021, at 18:15, Jacob Champion wrote: > > On Wed, 2021-06-16 at 15:31 +0200, Daniel Gustafsson wrote: >>> On 16 Jun 2021, at 01:50, Jacob Champion wrote: >>> I've been tracking down reference leaks in the client. These open >>> references prevent NSS from shutting down cleanly,

Re: Improving isolationtester's data output

2021-06-16 Thread Andres Freund
Hi, On Wed, Jun 16, 2021, at 12:37, Tom Lane wrote: > Andres Freund writes: > > On 2021-06-15 22:44:29 -0400, Tom Lane wrote: > >> + memset(, 0, sizeof(popt)); > >> + popt.header = true; > >> + popt.align = true; > >> + popt.fieldSep = "|"; > >> + PQprint(stdout, res, ); > > > Is there an

Re: Improving isolationtester's data output

2021-06-16 Thread Tom Lane
Andres Freund writes: > On 2021-06-15 22:44:29 -0400, Tom Lane wrote: >> +memset(, 0, sizeof(popt)); >> +popt.header = true; >> +popt.align = true; >> +popt.fieldSep = "|"; >> +PQprint(stdout, res, ); > Is there an argument for not aligning because that can make diffs larger

Re: Improving isolationtester's data output

2021-06-16 Thread Tom Lane
Alvaro Herrera writes: > On 2021-Jun-16, Tom Lane wrote: >> Hearing few objections, I'll plan on back-patching. I'm thinking that the >> best thing to do is apply these changes after beta2 wraps, but before we >> branch v14. > Great. After checking cross-version diffs to see how painful that

Re: Improving isolationtester's data output

2021-06-16 Thread Andres Freund
Hi, On 2021-06-15 22:44:29 -0400, Tom Lane wrote: > Here's a really quick-and-dirty patch to see what that would look > like. I haven't bothered here to update the expected-files outside > the main src/test/isolation directory, nor to fix the variant files. Neat. > + memset(, 0,

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Matthias van de Meent
On Wed, 16 Jun 2021 at 21:12, Andres Freund wrote: > > Hi, > > On 2021-06-16 12:59:33 +0200, Matthias van de Meent wrote: > > PFA my adapted patch that fixes this new-ish issue, and does not > > include the (incorrect) assertions in GlobalVisUpdateApply. I've > > tested this against the

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Andres Freund
Hi, On 2021-06-16 09:46:07 -0700, Peter Geoghegan wrote: > On Wed, Jun 16, 2021 at 9:03 AM Peter Geoghegan wrote: > > On Wed, Jun 16, 2021 at 3:59 AM Matthias van de Meent > > > So the implicit assumption in heap_page_prune that > > > HeapTupleSatisfiesVacuum(OldestXmin) is always consistent

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 12:06 PM Andres Freund wrote: > > I would think that it wouldn't really matter inside VACUUM -- it would > > only really need to be either an opportunistic pruning or an > > opportunistic index deletion thing -- probably both. Most of the time > > VACUUM doesn't seem to

Re: pgbench logging broken by time logic changes

2021-06-16 Thread Andrew Dunstan
On 6/16/21 2:59 PM, Fabien COELHO wrote: > > Hello Greg, > >> I have a lot of community oriented work backed up behind this right >> now, so >> I'm gonna be really honest.  This time rework commit in its current form >> makes me uncomfortable at this point in the release schedule.  The >> commit

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Andres Freund
Hi, On 2021-06-16 12:59:33 +0200, Matthias van de Meent wrote: > PFA my adapted patch that fixes this new-ish issue, and does not > include the (incorrect) assertions in GlobalVisUpdateApply. I've > tested this against the reproducing case, both with and without the > fix in

Re: pgbench logging broken by time logic changes

2021-06-16 Thread Fabien COELHO
pg_time_now(). This uses INSTR_TIME_SET_CURRENT in it, but this macro can call clock_gettime(CLOCK_MONOTONIC[_RAW], ) instead of gettimeofday or clock_gettime(CLOCK_REALTIME, ). When CLOCK_MONOTONIC[_RAW] is used, clock_gettime doesn't return epoch time. Therefore, we can use

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 11:27 AM Andres Freund wrote: > 2) Modeling when it is safe to remove row versions. It is easy to remove >a tuple that was inserted and deleted within one "not needed" xid >range, but it's far less obvious when it is safe to remove row >versions where

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Andres Freund
Hi, On 2021-06-16 10:44:49 -0700, Peter Geoghegan wrote: > On Wed, Jun 16, 2021 at 10:04 AM Tom Lane wrote: > > Of course, there's still the question of how VACUUM could cheaply > > apply such info to decide what could be purged. > I would think that it wouldn't really matter inside VACUUM --

Re: pgbench logging broken by time logic changes

2021-06-16 Thread Fabien COELHO
Hello Greg, I have a lot of community oriented work backed up behind this right now, so I'm gonna be really honest. This time rework commit in its current form makes me uncomfortable at this point in the release schedule. The commit has already fought through two rounds of platform specific

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 11:06 AM Andres Freund wrote: > > 2) (a) Some hackers want the feature gone so they can implement changes > >without making those changes cooperate with this feature. (b) Bugs in > > this > >feature make such cooperation materially harder. > > I think the a) part

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Andres Freund
Hi, On 2021-06-16 13:04:07 -0400, Tom Lane wrote: > Yeah, I think this scenario of a few transactions with old snapshots > and the rest with very new ones could be improved greatly if we exposed > more info about backends' snapshot state than just "oldest xmin". But > that might be expensive to

Re: pgbench bug candidate: negative "initial connection time"

2021-06-16 Thread Fabien COELHO
Hello Yugo-san, When connection break while the bench has already started, maybe it makes more sense to proceed, The result would be incomplete also in this case. However, the reason why it is worth to proceed is that such information is still useful for users, or we don't want to waste

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Andres Freund
Hi, On 2021-06-15 21:59:45 -0700, Noah Misch wrote: > Hackers are rather wise, but the variety of PostgreSQL use is enormous. We > see that, among other ways, when regression fixes spike in each vN.1. The > $SUBJECT feature was born in response to a user experience; a lack of hacker > interest

Re: pgbench logging broken by time logic changes

2021-06-16 Thread Yugo NAGATA
On Wed, 16 Jun 2021 10:49:36 -0400 Gregory Smith wrote: > A lot of things are timed in pgbench now so I appreciate the idea. Y'all > started that whole big thread about sync on my birthday though and I didn't > follow the details of what that was reviewed against. For the logging use > case I

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 10:04 AM Tom Lane wrote: > I remember that Heikki was fooling with a patch that reduced snapshots > to LSNs. If we got that done, it'd be practical to expose complete > info about backends' snapshot state in a lot of cases (i.e., anytime > you had less than N live

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Tom Lane
Stephen Frost writes: > I've long felt that the appropriate approach to addressing that is to > improve on VACUUM and find a way to do better than just having the > conditional of 'xmax < global min' drive if we can mark a given tuple as > no longer visible to anyone. Yeah, I think this scenario

Re: a path towards replacing GEQO with something better

2021-06-16 Thread Tom Lane
John Naylor writes: > On Wed, Jun 16, 2021 at 12:01 PM Robert Haas wrote: >> I feel like these are completely equivalent. Either way, the planner >> is going to deduce that all the ".col" columns are equal to each other >> via the equivalence class machinery, and then the subsequent planning >>

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 9:03 AM Peter Geoghegan wrote: > On Wed, Jun 16, 2021 at 3:59 AM Matthias van de Meent > > So the implicit assumption in heap_page_prune that > > HeapTupleSatisfiesVacuum(OldestXmin) is always consistent with > > heap_prune_satisfies_vacuum(vacrel) has never been true. In

Allowing regular identifiers in isolationtester scripts

2021-06-16 Thread Tom Lane
[ new subject, new thread, new patch ] Alvaro Herrera writes: > On 2021-Jun-16, Tom Lane wrote: >> BTW, as long as we're thinking of back-patching nontrivial specfile >> changes, I have another modest proposal. What do people think of >> removing the requirement for step/session names to be

Re: Avoid stuck of pbgench due to skipped transactions

2021-06-16 Thread Yugo NAGATA
On Mon, 14 Jun 2021 16:06:10 +0900 Yugo NAGATA wrote: > On Mon, 14 Jun 2021 08:47:40 +0200 (CEST) > Fabien COELHO wrote: > > > I attached the fixed patch that uses return instead of break, and I > > > confirmed > > > that this made the progress reporting work property. > > > > I'm hesitating

Re: a path towards replacing GEQO with something better

2021-06-16 Thread John Naylor
On Wed, Jun 16, 2021 at 12:01 PM Robert Haas wrote: > > On Tue, Jun 15, 2021 at 2:16 PM John Naylor > wrote: > > > I don't quite understand the difference between the "chain" case and > > > the "star" case. Can you show sample queries for each one? e.g. SELECT > > > ... FROM a_1, a_2, ..., a_n

PoC: Using Count-Min Sketch for join cardinality estimation

2021-06-16 Thread Tomas Vondra
Hi, During the recent "CMU vaccination" talk given by Robert [1], a couple of the attendees (some of which were engineers working on various other database systems) asked whether PostgreSQL optimizer uses sketches. Which it does not, as far as I'm aware. Perhaps some of our statistics could

Re: Support for NSS as a libpq TLS backend

2021-06-16 Thread Jacob Champion
On Wed, 2021-06-16 at 15:31 +0200, Daniel Gustafsson wrote: > > On 16 Jun 2021, at 01:50, Jacob Champion wrote: > > I've been tracking down reference leaks in the client. These open > > references prevent NSS from shutting down cleanly, which then makes it > > impossible to open a new context in

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Stephen Frost
Greetings, * Greg Stark (st...@mit.edu) wrote: > I think Andres's point earlier is the one that stands out the most for me: > > > I still think that's the most reasonable course. I actually like the > > feature, but I don't think a better implementation of it would share > > much if any of the

Re: Transactions involving multiple postgres foreign servers, take 2

2021-06-16 Thread Robert Haas
On Tue, Jun 15, 2021 at 5:51 AM tsunakawa.ta...@fujitsu.com wrote: > Postgres can do that, but other implementations can not necessaily do it, I'm > afraid. But before that, the FDW interface documentation doesn't describe > anything about how to handle interrupts. Actually, odbc_fdw and

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 3:59 AM Matthias van de Meent wrote: > On Tue, 15 Jun 2021 at 03:22, Andres Freund wrote: > > > @@ -4032,6 +4039,24 @@ GlobalVisTestShouldUpdate(GlobalVisState *state) > > > static void > > > GlobalVisUpdateApply(ComputeXidHorizonsResult *horizons) > > > { > > > +

Re: a path towards replacing GEQO with something better

2021-06-16 Thread Robert Haas
On Tue, Jun 15, 2021 at 2:16 PM John Naylor wrote: > > I don't quite understand the difference between the "chain" case and > > the "star" case. Can you show sample queries for each one? e.g. SELECT > > ... FROM a_1, a_2, ..., a_n WHERE ? > > SELECT * > FROMtab1, tab2, tab3, tab4 > WHERE

Re: pgbench bug candidate: negative "initial connection time"

2021-06-16 Thread Yugo NAGATA
Hello Fabien, On Mon, 14 Jun 2021 11:30:14 +0200 (CEST) Fabien COELHO wrote: > >>> Hmmm. Possibly. Another option could be not to report anything after some > >>> errors. I'm not sure, because it would depend on the use case. I guess the > >>> command returned an error status as well. > >> > >>

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Tom Lane
Greg Stark writes: > Fwiw I too think the basic idea of the feature is actually awesome. > There are tons of use cases where you might have one long-lived > transaction working on a dedicated table (or even a schema) that will > never look at the rapidly mutating tables in another schema and

Re: SQLSTATE for replication connection failures

2021-06-16 Thread Tom Lane
Masahiko Sawada writes: > On Mon, Jun 14, 2021 at 6:18 PM Amit Kapila wrote: >>> Shall we just use ERRCODE_CONNECTION_FAILURE for these failures, or >>> would it be better to invent another SQLSTATE code? Arguably, >>> ERRCODE_CONNECTION_FAILURE is meant for failures of client connections; >>>

Re: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-06-16 Thread Robert Haas
On Tue, Jun 15, 2021 at 10:41 AM Amit Kapila wrote: > > I don't think that finding the relation involved and registering an > > invalidation for the same will work properly. Suppose there is a > > concurrently-running transaction which has created a new table and > > attached a trigger function

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Greg Stark
I think Andres's point earlier is the one that stands out the most for me: > I still think that's the most reasonable course. I actually like the > feature, but I don't think a better implementation of it would share > much if any of the current infrastructure. That makes me wonder whether

Re: Unresolved repliaction hang and stop problem.

2021-06-16 Thread Ha Ka
> Hello, thanks, I downloaded the files but since you sent the perf.data > files there's not much I can do to usefully interpret them. Can you > please do "perf report -g > perf_report.txt" on each subdir with a > perf.data file and upload those text files? (You don't need to rerun > the test

Re: Improving isolationtester's data output

2021-06-16 Thread Alvaro Herrera
On 2021-Jun-16, Tom Lane wrote: > Noah Misch writes: > > I'm +1 for back-patching this class of change. I've wasted time adapting a > > back-patch's test case to account for non-back-patched test infrastructure > > changes. Every back-patch of test infrastructure has been a strict win from > >

pgbench logging broken by time logic changes

2021-06-16 Thread Gregory Smith
Back on March 10 Thomas Munro committed and wrestled multiple reworks of the pgbench code from Fabien and the crew. The feature to synchronize startup I'm looking forward to testing now that I have a packaged beta. Variations on that problem have bit me so many times I added code last year to my

Re: Decoding speculative insert with toast leaks memory

2021-06-16 Thread Tom Lane
Amit Kapila writes: > Pushed! skink reports that this has valgrind issues: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink=2021-06-15%2020%3A49%3A26 2021-06-16 01:20:13.344 UTC [2198271][4/0:0] LOG: received replication command: IDENTIFY_SYSTEM 2021-06-16 01:20:13.384 UTC

Re: Duplicate history file?

2021-06-16 Thread Stephen Frost
Greetings, * Julien Rouhaud (rjuju...@gmail.com) wrote: > On Wed, Jun 16, 2021 at 9:19 PM Stephen Frost wrote: > > This is exactly it. I don't agree that we can, or should, treat every > > sensible thing that we realize about what the archive command or the > > backup tool should be doing as

Re: Should wal receiver reply to wal sender more aggressively?

2021-06-16 Thread Paul Guo
[ Resending the mail since I found my previous email has a very bad format that is hard to read]. While working on some related issues I found that the wal receiver tries to call walrcv_receive() loop before replying the write/flush/apply LSN to wal senders in XLogWalRcvSendReply(). It is

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-16 Thread Tomas Vondra
On 6/16/21 2:36 PM, Alexander Pyhalov wrote: Hi. It seems this commit commit b676ac443b6a83558d4701b2dd9491c0b37e17c4 Author: Tomas Vondra Date:   Fri Jun 11 20:19:48 2021 +0200     Optimize creation of slots for FDW bulk inserts has broken batch insert for partitions with unique indexes.

Re: Improving isolationtester's data output

2021-06-16 Thread Tom Lane
Noah Misch writes: > I'm +1 for back-patching this class of change. I've wasted time adapting a > back-patch's test case to account for non-back-patched test infrastructure > changes. Every back-patch of test infrastructure has been a strict win from > my perspective. Hearing few objections,

Re: pg_stat_statements and "IN" conditions

2021-06-16 Thread Dmitry Dolgov
> On Tue, Jun 15, 2021 at 05:18:50PM +0200, Dmitry Dolgov wrote: > > On Thu, Mar 18, 2021 at 04:50:02PM +0100, Dmitry Dolgov wrote: > > > On Thu, Mar 18, 2021 at 09:38:09AM -0400, David Steele wrote: > > > On 1/5/21 10:51 AM, Zhihong Yu wrote: > > > > > > > > +   int         lastExprLenght = 0; >

Re: Duplicate history file?

2021-06-16 Thread Julien Rouhaud
On Wed, Jun 16, 2021 at 9:19 PM Stephen Frost wrote: > > This is exactly it. I don't agree that we can, or should, treat every > sensible thing that we realize about what the archive command or the > backup tool should be doing as some bug in our documentation that has to > be backpatched. > If

Re: [Proposal] Add accumulated statistics for wait event

2021-06-16 Thread Jehan-Guillaume de Rorthais
Hi Andres, On Mon, 14 Jun 2021 15:01:14 -0700 Andres Freund wrote: > On 2021-06-14 23:20:47 +0200, Jehan-Guillaume de Rorthais wrote: > > > On 2021-06-14 16:10:32 +0200, Jehan-Guillaume de Rorthais wrote: > > > > In the patch in attachment, I tried to fix this by using kind of an > > > >

  1   2   >