Re: fixing old_snapshot_threshold's time->xid mapping

2020-04-21 Thread Dilip Kumar
On Tue, Apr 21, 2020 at 4:52 PM Dilip Kumar wrote: > > On Tue, Apr 21, 2020 at 3:44 PM Thomas Munro wrote: > > > > On Tue, Apr 21, 2020 at 2:05 PM Thomas Munro wrote: > > > As before, these two apply on top of Robert's patches (or at least his > > > 0001 and 0002). > > > > While trying to

Re: HEAPDEBUGALL is broken

2020-04-21 Thread Alexander Lakhin
21.04.2020 21:01, Peter Eisentraut wrote: > On 2020-04-19 22:00, Alexander Lakhin wrote: >> To the point, I've tried to use HAVE_ALLOCINFO on master today and it >> failed too: > > Do you have a proposed patch? > As this is broken at least since the invention of the generational allocator

Re: WAL usage calculation patch

2020-04-21 Thread Justin Pryzby
On Wed, Apr 22, 2020 at 09:15:08AM +0530, Amit Kapila wrote: > > > > And add the acronym to the docs: > > > > > > > > $ git grep 'full page' '*/explain.sgml' > > > > doc/src/sgml/ref/explain.sgml: number of records, number of full > > > > page writes and amount of WAL bytes > > > > > > > >

Re: WAL usage calculation patch

2020-04-21 Thread Amit Kapila
On Mon, Apr 20, 2020 at 1:17 PM Kyotaro Horiguchi wrote: > > At Sun, 19 Apr 2020 16:22:26 +0200, Julien Rouhaud wrote > in > > Hi Justin, > > > > Thanks for the review! > > > > On Sat, Apr 18, 2020 at 10:41 PM Justin Pryzby wrote: > > > > > > Should capitalize at least the non-text one ? And

Re: WIP: Aggregation push-down

2020-04-21 Thread Andy Fan
> > > 1) v14-0001-Introduce-RelInfoList-structure.patch > > - > > > > - I'm not entirely sure why we need this change. We had the list+hash > > before, so I assume we do this because we need the output functions? > > I believe that this is what Tom

Re: Parallel Append can break run-time partition pruning

2020-04-21 Thread David Rowley
On Tue, 21 Apr 2020 at 15:03, David Rowley wrote: > I wonder if the fix should be more something along the lines of trying > to merge things do we only generate a single partial path. That way > we wouldn't be at the mercy of the logic in add_partial_path() to > accept or reject the path based

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Kyotaro Horiguchi
At Wed, 22 Apr 2020 11:51:42 +0900, Fujii Masao wrote in > > I meant that we always have EOR at the end of recovery. So in the > > missing latest checkpoint (and crash recovery) case, we insert EOR > > after the immediate checkpoint. That also means we no longer set > >

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Fujii Masao
On 2020/04/22 9:13, Kyotaro Horiguchi wrote: At Tue, 21 Apr 2020 22:08:56 +0900, Fujii Masao wrote in On 2020/04/21 17:15, Kyotaro Horiguchi wrote: At Mon, 20 Apr 2020 15:26:16 +0900, Fujii Masao wrote in Patch attached. I will add this into the first CF for v14. -

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Fujii Masao
On 2020/04/22 10:53, Kyotaro Horiguchi wrote: At Wed, 22 Apr 2020 10:28:07 +0900, Ian Barwick wrote in On 2020/04/22 6:53, Alvaro Herrera wrote: On 2020-Apr-21, Jehan-Guillaume de Rorthais wrote: On Tue, 21 Apr 2020 15:36:22 +0900 Michael Paquier wrote: Also in that case, non-fast

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Fujii Masao
On 2020/04/22 6:57, Alvaro Herrera wrote: On 2020-Apr-20, Fujii Masao wrote: + /* +* In 9.1 and 9.2 the postmaster unlinked the promote file inside the +* signal handler. It now leaves the file in place and lets the +* Startup process do the unlink. +*/

Re: Add "-Wimplicit-fallthrough" to default flags (was Re: pgsql: Support FETCH FIRST WITH TIES)

2020-04-21 Thread Tom Lane
Alvaro Herrera writes: > Do we intend to see this done in the current cycle? I don't have an objection to doing it now. It's just a new compiler warning option, it shouldn't be able to break any code. (Plus there's plenty of time to revert, if somehow it causes a problem.)

Re: [PATCH] Implement INSERT SET syntax

2020-04-21 Thread movead li
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed It builds failed by applying to the latest code version, and

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Kyotaro Horiguchi
At Wed, 22 Apr 2020 10:28:07 +0900, Ian Barwick wrote in > On 2020/04/22 6:53, Alvaro Herrera wrote: > > On 2020-Apr-21, Jehan-Guillaume de Rorthais wrote: > > > >> On Tue, 21 Apr 2020 15:36:22 +0900 > >> Michael Paquier wrote: > > > Also in that case, non-fast promotion is triggered.

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Ian Barwick
On 2020/04/22 6:53, Alvaro Herrera wrote: On 2020-Apr-21, Jehan-Guillaume de Rorthais wrote: On Tue, 21 Apr 2020 15:36:22 +0900 Michael Paquier wrote: Also in that case, non-fast promotion is triggered. Since my patch tries to remove non-fast promotion, it's intentional to prevent them

Re: MYSQL_FDW trigger BEFORE UPDATE changes to NEW on a col not in the update statement don't go through

2020-04-21 Thread Etsuro Fujita
Hi Francois, On Wed, Apr 22, 2020 at 8:09 AM Francois Payette wrote: > I create a trigger on an imported foreign table. In the procedure, I change > the value of a column that is not in the triggering update statement. This > change does not make it to the mysql side. I'm not an expert on

Re: Remove page-read callback from XLogReaderState.

2020-04-21 Thread Kyotaro Horiguchi
At Tue, 21 Apr 2020 17:04:27 +0900 (JST), Kyotaro Horiguchi wrote in > Mmm. The message body seems disappearing for uncertain reason. cd12323440 conflicts with this. Rebased. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From b3b780c2143ae70b82fb1e8256e771f11276af31 Mon

Re: DETACH PARTITION and FOR EACH ROW triggers on partitioned tables

2020-04-21 Thread Justin Pryzby
On Tue, Apr 21, 2020 at 07:03:30PM -0400, Alvaro Herrera wrote: > On 2020-Apr-20, Justin Pryzby wrote: > > > On Mon, Apr 20, 2020 at 06:35:44PM +0900, Amit Langote wrote: > > > > Also, how about, for consistency, making the parent table labeling of > > > the trigger look similar to that for the

Re: forgotten initalization of a variable

2020-04-21 Thread Kyotaro Horiguchi
At Wed, 22 Apr 2020 08:13:02 +0900, Michael Paquier wrote in > On Tue, Apr 21, 2020 at 06:09:30PM +0900, Kyotaro Horiguchi wrote: > > At Tue, 21 Apr 2020 17:34:26 +0900, Michael Paquier > > wrote in > >> a checkpoint record now, but this routine could be called elsewhere in > >> the future.

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Kyotaro Horiguchi
At Tue, 21 Apr 2020 22:08:56 +0900, Fujii Masao wrote in > > > On 2020/04/21 17:15, Kyotaro Horiguchi wrote: > > At Mon, 20 Apr 2020 15:26:16 +0900, Fujii Masao > > wrote in > >> Patch attached. I will add this into the first CF for v14. > > - if (!fast_promoted) > > +

Re: ALTER TABLE ... SET STORAGE does not propagate to indexes

2020-04-21 Thread Alvaro Herrera
I'm surprised that this hasn't applied yet, because: On 2020-Apr-09, Peter Eisentraut wrote: > One thing to remember is that the current situation is broken. While you > can set index columns to have different storage than the corresponding table > columns, pg_dump does not preserve that,

Re: 2pc leaks fds

2020-04-21 Thread Alvaro Herrera
On 2020-Apr-08, Antonin Houska wrote: > Specifically for 2PC, isn't it better to close some OS-level FD of an > unrelated table scan and then succeed than to ERROR immediately? Anyway, > 0dc8ead46 hasn't changed this. I think for full generality of the interface, we pass a "close" callback in

Re: Add "-Wimplicit-fallthrough" to default flags (was Re: pgsql: Support FETCH FIRST WITH TIES)

2020-04-21 Thread Alvaro Herrera
Do we intend to see this done in the current cycle? -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: forgotten initalization of a variable

2020-04-21 Thread Michael Paquier
On Tue, Apr 21, 2020 at 06:09:30PM +0900, Kyotaro Horiguchi wrote: > At Tue, 21 Apr 2020 17:34:26 +0900, Michael Paquier > wrote in >> a checkpoint record now, but this routine could be called elsewhere in >> the future. Please see the attached. > > It looks fine to me. Fixed this way, then.

MYSQL_FDW trigger BEFORE UPDATE changes to NEW on a col not in the update statement don't go through

2020-04-21 Thread Francois Payette
Hi All, I was pleasantly surprised to see that triggers can be created on FDW tables. I'm running into a problem. I create a trigger on an imported foreign table. In the procedure, I change the value of a column that is not in the triggering update statement. This change does not make it to

Re: DETACH PARTITION and FOR EACH ROW triggers on partitioned tables

2020-04-21 Thread Alvaro Herrera
On 2020-Apr-20, Justin Pryzby wrote: > On Mon, Apr 20, 2020 at 06:35:44PM +0900, Amit Langote wrote: > > Also, how about, for consistency, making the parent table labeling of > > the trigger look similar to that for the foreign constraint, so > > Triggers: > > TABLE "f1" TRIGGER "trig"

Re: design for parallel backup

2020-04-21 Thread Andres Freund
Hi, On 2020-04-21 17:09:50 -0400, Robert Haas wrote: > On Tue, Apr 21, 2020 at 4:14 PM Andres Freund wrote: > > It was local TCP. The speeds I can reach are faster than the 10GiB/s > > (unidirectional) I can do between the laptop & workstation, so testing > > it over "actual" network isn't

Re: Do we need to handle orphaned prepared transactions in the server?

2020-04-21 Thread Bruce Momjian
On Tue, Apr 21, 2020 at 04:03:53PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Tue, Apr 21, 2020 at 01:52:46PM +0900, Michael Paquier wrote: > >> On Mon, Apr 20, 2020 at 10:35:15PM -0400, Bruce Momjian wrote: > >>> On Thu, Apr 16, 2020 at 03:11:51PM -0400, Tom Lane wrote: > If we

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Alvaro Herrera
On 2020-Apr-20, Fujii Masao wrote: > + /* > + * In 9.1 and 9.2 the postmaster unlinked the promote file inside the > + * signal handler. It now leaves the file in place and lets the > + * Startup process do the unlink. > + */ > + if (IsPromoteSignaled() &&

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Alvaro Herrera
On 2020-Apr-21, Jehan-Guillaume de Rorthais wrote: > On Tue, 21 Apr 2020 15:36:22 +0900 > Michael Paquier wrote: > > > Also in that case, non-fast promotion is triggered. Since my patch > > > tries to remove non-fast promotion, it's intentional to prevent them > > > from doing that. But you

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Jehan-Guillaume de Rorthais
Hello, On Tue, 21 Apr 2020 15:36:22 +0900 Michael Paquier wrote: > On Tue, Apr 21, 2020 at 03:29:54PM +0900, Fujii Masao wrote: > > Yeah, but that's not documented. So I don't think that we need to keep > > the backward-compatibility for that. > > > > Also in that case, non-fast promotion is

Re: [IBM z Systems] Getting server crash when jit_above_cost =0

2020-04-21 Thread Thomas Munro
On Wed, Apr 22, 2020 at 2:34 AM tushar wrote: > (gdb) bt > #0 0x03ffa9841220 in raise () from /lib64/libc.so.6 > #1 0x03ffa9842aa8 in abort () from /lib64/libc.so.6 > #2 0x03ff9f7881b4 in __gnu_cxx::__verbose_terminate_handler() () from > /lib64/libstdc++.so.6 > #3

Re: design for parallel backup

2020-04-21 Thread Robert Haas
On Tue, Apr 21, 2020 at 4:14 PM Andres Freund wrote: > It was local TCP. The speeds I can reach are faster than the 10GiB/s > (unidirectional) I can do between the laptop & workstation, so testing > it over "actual" network isn't informative - I basically can reach line > speed between them with

Re: Do we need to handle orphaned prepared transactions in the server?

2020-04-21 Thread Tom Lane
Bruce Momjian writes: > On Tue, Apr 21, 2020 at 01:52:46PM +0900, Michael Paquier wrote: >> On Mon, Apr 20, 2020 at 10:35:15PM -0400, Bruce Momjian wrote: >>> On Thu, Apr 16, 2020 at 03:11:51PM -0400, Tom Lane wrote: If we were going to go down the path of periodically logging warnings

Re: More efficient RI checks - take 2

2020-04-21 Thread Tom Lane
Alvaro Herrera writes: > I do wonder if the RI stuff would actually end up being faster without > SPI. If not, we'd only end up writing more code to do the same thing. > Now that tables can be partitioned, it is much more of a pain than when > only regular tables could be supported. Obviously

Re: design for parallel backup

2020-04-21 Thread Andres Freund
Hi, On 2020-04-21 14:01:28 -0400, Robert Haas wrote: > On Tue, Apr 21, 2020 at 11:36 AM Andres Freund wrote: > > It's all CRC overhead. I don't see a difference with > > --manifest-checksums=none anymore. We really should look for a better > > "fast" checksum. > > Hmm, OK. I'm wondering exactly

Re: Do we need to handle orphaned prepared transactions in the server?

2020-04-21 Thread Bruce Momjian
On Tue, Apr 21, 2020 at 01:52:46PM +0900, Michael Paquier wrote: > On Mon, Apr 20, 2020 at 10:35:15PM -0400, Bruce Momjian wrote: > > On Thu, Apr 16, 2020 at 03:11:51PM -0400, Tom Lane wrote: > >> If we were going to go down the path of periodically logging warnings > >> about old prepared

Re: HEAPDEBUGALL is broken

2020-04-21 Thread Tom Lane
Peter Eisentraut writes: > On 2020-04-19 15:37, Tom Lane wrote: >> +1 for removing both. There are a lot of such debug "features" >> in the code, and few of them are worth anything IME. > removed I don't see a commit? regards, tom lane

Re: HEAPDEBUGALL is broken

2020-04-21 Thread Peter Eisentraut
On 2020-04-19 15:37, Tom Lane wrote: Peter Eisentraut writes: The HEAPDEBUGALL define has been broken since PG12 due to tableam changes. Should we just remove this? It doesn't look very useful. It's been around since Postgres95. If we opt for removing: PG12 added an analogous

Re: design for parallel backup

2020-04-21 Thread Robert Haas
On Tue, Apr 21, 2020 at 11:36 AM Andres Freund wrote: > It's all CRC overhead. I don't see a difference with > --manifest-checksums=none anymore. We really should look for a better > "fast" checksum. Hmm, OK. I'm wondering exactly what you tested here. Was this over your 20GiB/s connection

Re: HEAPDEBUGALL is broken

2020-04-21 Thread Peter Eisentraut
On 2020-04-19 22:00, Alexander Lakhin wrote: To the point, I've tried to use HAVE_ALLOCINFO on master today and it failed too: $ CPPFLAGS="-DHAVE_ALLOCINFO" ./configure --enable-tap-tests --enable-debug --enable-cassert  >/dev/null && make -j16 >/dev/null generation.c: In function

Re: DETACH PARTITION and FOR EACH ROW triggers on partitioned tables

2020-04-21 Thread Justin Pryzby
On Tue, Apr 21, 2020 at 12:20:38PM -0400, Alvaro Herrera wrote: > diff --git a/doc/src/sgml/ref/alter_table.sgml > b/doc/src/sgml/ref/alter_table.sgml > index 7595e609b5..233905552c 100644 > --- a/doc/src/sgml/ref/alter_table.sgml > +++ b/doc/src/sgml/ref/alter_table.sgml > @@ -941,13 +943,14 @@

Re: Autovacuum on partitioned table (autoanalyze)

2020-04-21 Thread yuzuko
Hello, On Sat, Apr 18, 2020 at 2:08 PM Justin Pryzby wrote: > > On Fri, Apr 17, 2020 at 10:09:07PM +0900, Amit Langote wrote: > > On Thu, Apr 16, 2020 at 11:19 PM Justin Pryzby wrote: > > > On Thu, Apr 16, 2020 at 06:16:45PM +0900, yuzuko wrote: > > > I don't think that adequately allows what's

Re: DETACH PARTITION and FOR EACH ROW triggers on partitioned tables

2020-04-21 Thread Alvaro Herrera
I think I also owe the attached doc updates. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml index 7595e609b5..233905552c 100644

Re: DETACH PARTITION and FOR EACH ROW triggers on partitioned tables

2020-04-21 Thread Alvaro Herrera
On 2020-Apr-20, Alvaro Herrera wrote: > + while (HeapTupleIsValid(trigtup = systable_getnext(scan))) > + { > + Form_pg_trigger pg_trigger = (Form_pg_trigger) > GETSTRUCT(trigtup); > + ObjectAddress trig; > + > + /* Ignore triggers that weren't cloned

Re: design for parallel backup

2020-04-21 Thread Andres Freund
Hi, On 2020-04-21 07:18:20 -0400, Robert Haas wrote: > On Tue, Apr 21, 2020 at 2:44 AM Andres Freund wrote: > > FWIW, I just tested pg_basebackup locally. > > > > Without compression and a stock postgres I get: > > unixtcp tcp+ssl: > > 1.74GiB/s

Re: More efficient RI checks - take 2

2020-04-21 Thread Alvaro Herrera
On 2020-Apr-20, Corey Huinker wrote: > > I can imagine removal of the SPI from the current implementation (and > > constructing the plans "manually"), but note that the queries I use in my > > patch are no longer that trivial. So the SPI makes sense to me because it > > ensures regular query

Re: [PATCH] distinct aggregates within a window function WIP

2020-04-21 Thread Andreas Karlsson
On 4/21/20 5:06 PM, Eugen Konkov wrote: Hi. I read the thread. Probably this fiddle will be helpful for testing: https://dbfiddle.uk/?rdbms=postgres_12=abe845142a5099d921d3729043fb8491 I recently encountered a problem: Why Window-specific functions do not allow DISTINCT to be used within the

[PATCH] distinct aggregates within a window function WIP

2020-04-21 Thread Eugen Konkov
Hi. I read the thread. Probably this fiddle will be helpful for testing: https://dbfiddle.uk/?rdbms=postgres_12=abe845142a5099d921d3729043fb8491 I recently encountered a problem: Why Window-specific functions do not allow DISTINCT to be used within the function argument list? sum( DISTINCT

[IBM z Systems] Getting server crash when jit_above_cost =0

2020-04-21 Thread tushar
Hi, We are  getting a server crash on zlinux machine  if we set jit_above_cost=0 in postgresql.conf file after configuring  PG v12 server  with --with-llvm ( llvm-ttoolset-6.0) We configured  PG v12 sources with switch --with-llvm  ( after setting these variables on command prompt )  

Re: WIP/PoC for parallel backup

2020-04-21 Thread Amit Kapila
On Tue, Apr 21, 2020 at 5:26 PM Ahsan Hadi wrote: > > On Tue, Apr 21, 2020 at 4:50 PM Amit Kapila wrote: >> >> On Tue, Apr 21, 2020 at 5:18 PM Amit Kapila wrote: >> > >> > On Tue, Apr 21, 2020 at 1:00 PM Asif Rehman wrote: >> > > >> > > I did some tests a while back, and here are the results.

Re: Fix for pg_statio_all_tables

2020-04-21 Thread Tom Lane
Alexander Korotkov writes: > On Tue, Apr 21, 2020 at 7:58 AM Tom Lane wrote: >> Yeah, but that was for a security hole. I am doubtful that the >> severity of this problem is bad enough to justify jumping through >> similar hoops. Even if we fixed it and documented it, how many >> users would

[SSPI] Windows group support

2020-04-21 Thread The Dude
Hi, I have some code that I've been using in production that supports adding and authenticating Windows groups via the pg_ident file. It has a new indicator (+), that signifies the identifier is a Windows group, as in the following example: # MAPNAME SYSTEM-USERNAME PG-USERNAME "Users"

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Fujii Masao
On 2020/04/21 17:15, Kyotaro Horiguchi wrote: At Mon, 20 Apr 2020 15:26:16 +0900, Fujii Masao wrote in Patch attached. I will add this into the first CF for v14. - if (!fast_promoted) + if (!promoted)

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-04-21 Thread Juan José Santamaría Flecha
On Tue, Apr 21, 2020 at 2:22 PM Ranier Vilela wrote: > More few comments. > > 1. Comments about order: > /* > * Callback function for EnumSystemLocalesEx. > * Stop enumerating if a match is found for a locale with the format > * _. > * The order for search locale is essential: > * Find

Re: Concurrency bug in amcheck

2020-04-21 Thread Alexander Korotkov
On Tue, Apr 21, 2020 at 12:54 PM Alexander Korotkov wrote: > I found concurrency bug in amcheck running on replica. When > btree_xlog_unlink_page() replays changes to replica, deleted page is > left with no items. But if amcheck steps on such deleted page > palloc_btree_page() expects it would

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-04-21 Thread Ranier Vilela
Em ter., 21 de abr. de 2020 às 09:02, Juan José Santamaría Flecha < juanjo.santama...@gmail.com> escreveu: > > On Tue, Apr 21, 2020 at 12:41 PM Amit Kapila > wrote: > >> On Tue, Apr 21, 2020 at 12:51 PM Amit Kapila >> wrote: >> > >> > I have tried a simple test with the latest patch and it

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-04-21 Thread Juan José Santamaría Flecha
On Tue, Apr 21, 2020 at 12:41 PM Amit Kapila wrote: > On Tue, Apr 21, 2020 at 12:51 PM Amit Kapila > wrote: > > > > I have tried a simple test with the latest patch and it failed for me. > > > > Set LC_MESSAGES="English_United Kingdom"; > > -- returns en-GB, then code changes it to en_NZ when

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-04-21 Thread Dilip Kumar
On Sat, Apr 18, 2020 at 6:12 PM Erik Rijkers wrote: > > On 2020-04-18 11:10, Erik Rijkers wrote: > > On 2020-04-18 11:07, Erik Rijkers wrote: > Hi Erik, > > While setting up the cascading replication I have hit one issue on > base code[1]. After fixing that I have got one

Re: WIP/PoC for parallel backup

2020-04-21 Thread Ahsan Hadi
On Tue, Apr 21, 2020 at 4:50 PM Amit Kapila wrote: > On Tue, Apr 21, 2020 at 5:18 PM Amit Kapila > wrote: > > > > On Tue, Apr 21, 2020 at 1:00 PM Asif Rehman > wrote: > > > > > > I did some tests a while back, and here are the results. The tests > were done to simulate > > > a live database

Re: WIP/PoC for parallel backup

2020-04-21 Thread Amit Kapila
On Tue, Apr 21, 2020 at 5:18 PM Amit Kapila wrote: > > On Tue, Apr 21, 2020 at 1:00 PM Asif Rehman wrote: > > > > I did some tests a while back, and here are the results. The tests were > > done to simulate > > a live database environment using pgbench. > > > > machine configuration used for

Re: WIP/PoC for parallel backup

2020-04-21 Thread Amit Kapila
On Tue, Apr 21, 2020 at 1:00 PM Asif Rehman wrote: > > I did some tests a while back, and here are the results. The tests were done > to simulate > a live database environment using pgbench. > > machine configuration used for this test: > Instance Type:t2.xlarge > Volume Type :io1 >

Re: fixing old_snapshot_threshold's time->xid mapping

2020-04-21 Thread Dilip Kumar
On Tue, Apr 21, 2020 at 3:44 PM Thomas Munro wrote: > > On Tue, Apr 21, 2020 at 2:05 PM Thomas Munro wrote: > > As before, these two apply on top of Robert's patches (or at least his > > 0001 and 0002). > > While trying to figure out if Robert's 0003 patch was correct, I added > yet another

Re: design for parallel backup

2020-04-21 Thread Robert Haas
On Tue, Apr 21, 2020 at 2:44 AM Andres Freund wrote: > FWIW, I just tested pg_basebackup locally. > > Without compression and a stock postgres I get: > unixtcp tcp+ssl: > 1.74GiB/s 1.02GiB/s699MiB/s > > That turns out to be bottlenecked by

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-04-21 Thread Amit Kapila
On Tue, Apr 21, 2020 at 12:51 PM Amit Kapila wrote: > > On Mon, Apr 20, 2020 at 6:32 PM Amit Kapila wrote: > > > > On Sun, Apr 19, 2020 at 3:46 PM Juan José Santamaría Flecha > > wrote: > > > > > > I cannot find a single place where all supported locales are listed, but > > > I have created a

RE: extension patch of CREATE OR REPLACE TRIGGER

2020-04-21 Thread osumi.takami...@fujitsu.com
Dear Tom Lane Thanks for your so many fruitful comments ! I have fixed my patch again. On the other hand, there're some questions left that I'd like to discuss. > * You missed updating equalfuncs.c/copyfuncs.c. Pretty much any change in a > Node struct will require touching backend/nodes/

Re: Fix for pg_statio_all_tables

2020-04-21 Thread Alexander Korotkov
On Tue, Apr 21, 2020 at 7:58 AM Tom Lane wrote: > Michael Paquier writes: > > On Tue, Apr 21, 2020 at 02:44:45AM +0300, Alexander Korotkov wrote: > >> As a bugfix, I think this should be backpatched. But this patch > >> requires catalog change. Were similar cases there before? If so, > >>

Re: Fix for pg_statio_all_tables

2020-04-21 Thread Alexander Korotkov
On Tue, Apr 21, 2020 at 4:38 AM Michael Paquier wrote: > On Tue, Apr 21, 2020 at 02:44:45AM +0300, Alexander Korotkov wrote: > > Among all the joined tables, only "pg_index I" is expected to have > > multiple rows associated with single relation. But we do sum() for > > toast index "pg_index X"

Re: WIP/PoC for parallel backup

2020-04-21 Thread Asif Rehman
On Tue, 21 Apr 2020 at 2:36 PM, Jeevan Ladhe wrote: > Hi Asif, > > On Tue, Apr 21, 2020 at 1:00 PM Asif Rehman > wrote: > >> Hi, >> >> I did some tests a while back, and here are the results. The tests were >> done to simulate >> a live database environment using pgbench. >> >> machine

Re: fixing old_snapshot_threshold's time->xid mapping

2020-04-21 Thread Thomas Munro
On Tue, Apr 21, 2020 at 2:05 PM Thomas Munro wrote: > As before, these two apply on top of Robert's patches (or at least his > 0001 and 0002). While trying to figure out if Robert's 0003 patch was correct, I added yet another patch to this stack to test it. 0006 does basic xid map maintenance

Concurrency bug in amcheck

2020-04-21 Thread Alexander Korotkov
Hi! I found concurrency bug in amcheck running on replica. When btree_xlog_unlink_page() replays changes to replica, deleted page is left with no items. But if amcheck steps on such deleted page palloc_btree_page() expects it would have items. (lldb_on_primary) b btbulkdelete primary=# drop

Re: WIP/PoC for parallel backup

2020-04-21 Thread Jeevan Ladhe
Hi Asif, On Tue, Apr 21, 2020 at 1:00 PM Asif Rehman wrote: > Hi, > > I did some tests a while back, and here are the results. The tests were > done to simulate > a live database environment using pgbench. > > machine configuration used for this test: > Instance Type:t2.xlarge > Volume Type

Re: forgotten initalization of a variable

2020-04-21 Thread Kyotaro Horiguchi
At Tue, 21 Apr 2020 17:34:26 +0900, Michael Paquier wrote in > On Tue, Apr 21, 2020 at 03:08:30PM +0900, Kyotaro Horiguchi wrote: > > The commit a7e8ece41c adds a new member restoreCommand to > > XLogPageReadPrivate. readOneRecord doesn't make use of it but forgets > > to set NULL. That can

Re: WIP: Aggregation push-down

2020-04-21 Thread Andy Fan
On Thu, Feb 27, 2020 at 4:50 PM Antonin Houska wrote: > legrand legrand wrote: > > > Antonin Houska-2 wrote > > > > Right now I recall two problems: 1) is the way I currently store > > > RelOptInfo for the grouped relations correct?, 2) how should we handle > > > types for which logical

Re: forgotten initalization of a variable

2020-04-21 Thread Michael Paquier
On Tue, Apr 21, 2020 at 03:08:30PM +0900, Kyotaro Horiguchi wrote: > The commit a7e8ece41c adds a new member restoreCommand to > XLogPageReadPrivate. readOneRecord doesn't make use of it but forgets > to set NULL. That can lead to illegal pointer access. That's an oversight of the original

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Kyotaro Horiguchi
At Mon, 20 Apr 2020 15:26:16 +0900, Fujii Masao wrote in > Patch attached. I will add this into the first CF for v14. - if (!fast_promoted) + if (!promoted) RequestCheckpoint(CHECKPOINT_END_OF_RECOVERY |

Re: WAL page magic errors (and plenty others) got hard to debug.

2020-04-21 Thread Andres Freund
Hi, On 2020-04-05 15:49:16 -0700, Andres Freund wrote: > When starting with on a data directory with an older WAL page magic we > currently make that hard to debug. E.g.: > > 2020-04-05 15:31:04.314 PDT [1896669][:0] LOG: database system was shut down > at 2020-04-05 15:24:56 PDT > 2020-04-05

Re: Remove page-read callback from XLogReaderState.

2020-04-21 Thread Kyotaro Horiguchi
I found this conficts with a7e8ece41cf7a96d8a9c4c037cdfef304d950831. Rebased on it. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From e67149578c750977a2a2872d3f4dbb3a86c82a6d Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Thu, 5 Sep 2019 20:21:55 +0900 Subject: [PATCH

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Kyotaro Horiguchi
At Tue, 21 Apr 2020 15:48:02 +0900, Fujii Masao wrote in > > > On 2020/04/21 15:36, Michael Paquier wrote: > > On Tue, Apr 21, 2020 at 03:29:54PM +0900, Fujii Masao wrote: > >> Yeah, but that's not documented. So I don't think that we need to keep > >> the backward-compatibility for that. >

Re: WIP/PoC for parallel backup

2020-04-21 Thread Asif Rehman
Hi, I did some tests a while back, and here are the results. The tests were done to simulate a live database environment using pgbench. machine configuration used for this test: Instance Type:t2.xlarge Volume Type :io1 Memory (MiB) :16384 vCPU # :4 Architecture:

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-04-21 Thread Amit Kapila
On Mon, Apr 20, 2020 at 6:32 PM Amit Kapila wrote: > > On Sun, Apr 19, 2020 at 3:46 PM Juan José Santamaría Flecha > wrote: > > > > I cannot find a single place where all supported locales are listed, but I > > have created a small test program (WindowsNLSLocales.c) based on: > > [_] format

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Fujii Masao
On 2020/04/21 15:36, Michael Paquier wrote: On Tue, Apr 21, 2020 at 03:29:54PM +0900, Fujii Masao wrote: Yeah, but that's not documented. So I don't think that we need to keep the backward-compatibility for that. Also in that case, non-fast promotion is triggered. Since my patch tries to

Re: design for parallel backup

2020-04-21 Thread Andres Freund
Hi, On 2020-04-20 22:31:49 -0700, Andres Freund wrote: > On 2020-04-21 10:20:01 +0530, Amit Kapila wrote: > > It is quite likely that compression can benefit more from parallelism > > as compared to the network I/O as that is mostly a CPU intensive > > operation but I am not sure if we can just

Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2

2020-04-21 Thread Fujii Masao
On 2020/04/21 14:54, Michael Paquier wrote: On Tue, Apr 21, 2020 at 02:27:20PM +0900, Fujii Masao wrote: On 2020/04/21 10:59, Michael Paquier wrote: With your patch, this code now means that in order to finish recovery you need to send SIGUSR2 to the startup process *and* to create the

forgotten initalization of a variable

2020-04-21 Thread Kyotaro Horiguchi
Hello. The commit a7e8ece41c adds a new member restoreCommand to XLogPageReadPrivate. readOneRecord doesn't make use of it but forgets to set NULL. That can lead to illegal pointer access. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From