Re: refactoring - share str2*int64 functions

2019-08-27 Thread Fabien COELHO
Bonjour Michaël, So I'll set out to write and use "pg_strtou?int64", i.e. 2 functions, and then possibly generalize to lower sizes, 32, 16, depending on what is actually needed. I am interested in this patch, and the next commit fest is close by. Are you working on an updated version? If not

Improve error detections in TAP tests by spreading safe_psql

2019-08-27 Thread Michael Paquier
Hi all, This is a follow-up of the discussion which happened here after Tom has committed fb57f40: https://www.postgresql.org/message-id/20190828012439.ga1...@paquier.xyz I have reviewed the TAP tests, and we have much more spots where it is better to use PostgresNode::safe_psql instead PostgresN

Re: Re: Email to hackers for test coverage

2019-08-27 Thread movead...@highgo.ca
On Wed, 28 Aug 2019 11:30:23 +0800 mich...@paquier.xyz wrote >- numeric is not a test suite designed for sorting, and hijacking it >to do so it not a good approach. Absolutely agreement. We can wait for repling from Robert and Peter G. >- it would be good to get coverage for the two extra code pa

Performance improvement of WAL writing?

2019-08-27 Thread Moon, Insung
Dear Hackers. Currently, the XLogWrite function is written in 8k(or 16kb) units regardless of the size of the new record. For example, even if a new record is only 300 bytes, pg_pwrite is called to write data in 8k units (if it cannot be writing on one page is 16kb written). Let's look at the wors

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-27 Thread Kyotaro Horiguchi
Hello, Noah. At Tue, 27 Aug 2019 15:49:32 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190827.154932.250364935.horikyota@gmail.com> > I'm not sure whether the knob shows apparent performance gain and > whether we can offer the criteria to identify the proper > value. But I'll

Re: pgbench - implement strict TPC-B benchmark

2019-08-27 Thread Fabien COELHO
Hello Dmitry, Well, it could be added? While doing benchmarking using different tools, including pgbench, I found it useful as a temporary hack to add copy freeze and maintenance_work_mem options (the last one not as an env variable, just as a set before, although not sure if it's a best idea

Re: REINDEX filtering in the backend

2019-08-27 Thread Michael Paquier
On Wed, Aug 28, 2019 at 02:02:08PM +0900, Michael Paquier wrote: > + index = index_open(indexOid, AccessShareLock); > + numAtts = index->rd_index->indnatts; > + index_close(index, AccessShareLock); > Wouldn't it be better to close that after doing the scan? > > Nit: I am pretty s

Re: REINDEX filtering in the backend

2019-08-27 Thread Michael Paquier
On Thu, Jul 11, 2019 at 11:14:20PM +0200, Julien Rouhaud wrote: > I didn't want to spend too much time enjoying bison and adding new > unreserved keywords, so for now I just implemented this syntax to > start a discussion for this feature in the next commitfest: > > REINDEX ( FILTER = COLLATION )

Re: Re: Email to hackers for test coverage

2019-08-27 Thread Michael Paquier
On Tue, Aug 27, 2019 at 03:57:20PM +0800, movead...@highgo.ca wrote: > I think your way is much better, so I change the patch and it is > 'regression_20190827.patch' now. Thanks for the new patch, I have committed the part for float4. > There are code lines related to NULL values in > ApplySort

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-08-27 Thread Peter Geoghegan
On Fri, Aug 16, 2019 at 8:56 AM Anastasia Lubennikova wrote: > Now the algorithm is the following: > - In case page split is needed, pass both tuples to _bt_split(). > _bt_findsplitloc() is now aware of upcoming replacement of origtup with > neworigtup, so it uses correct item size where needed.

Re: Cleanup isolation specs from unused steps

2019-08-27 Thread Michael Paquier
On Tue, Aug 27, 2019 at 07:05:50PM +0530, Asim R P wrote: > Thank you for the feedback. I've changed patch 0002 accordingly, please > take another look: > https://www.postgresql.org/message-id/CANXE4TdvSi7Yia_5sV82%2BMHf0WcUSN9u6_X8VEUBv-YStphd%3DQ%40mail.gmail.com Thanks! Let's move the discuss

Re: Statement timeout in pg_rewind

2019-08-27 Thread Michael Paquier
On Tue, Aug 27, 2019 at 10:45:27AM +0200, Alexander Kukushkin wrote: > Done, please see the next version attached. I have made the new error message consistent with run_simple_query to avoid more work to translators and because it is possible to know immediately the code path involved thanks to th

Re: pg11.5: ExecHashJoinNewBatch: glibc detected...double free or corruption (!prev)

2019-08-27 Thread Merlin Moncure
On Mon, Aug 26, 2019 at 12:01 PM Tom Lane wrote: > > Justin Pryzby writes: > > On Mon, Aug 26, 2019 at 12:45:24PM -0400, Tom Lane wrote: > >> However ... there is some pretty interesting info at > >> https://bugzilla.redhat.com/show_bug.cgi?id=1338673 > >> suggesting that compiling with a late-mo

Re: [PATCH] Make configuration file "include" directive handling more robust

2019-08-27 Thread Ian Barwick
On 8/25/19 4:39 AM, Tom Lane wrote: Ian Barwick writes: On 7/17/19 5:34 PM, Kyotaro Horiguchi wrote:> Hello. I don't think this is new to 12. No, though I'm not sure how much this would be seen as a bugfix and how far back it would be sensible to patch. I think this is worth considering a

Re: doc: clarify "pg_signal_backend" default role

2019-08-27 Thread Ian Barwick
On 8/28/19 7:04 AM, Tom Lane wrote: Ian Barwick writes: Currently the documentation for the default role "pg_signal_backend" states, somewhat ambiguously, "Send signals to other backends (eg: cancel query, terminate)", giving the impression other signals (e.g. SIGHUP) can be sent too, which is

Re: PostgreSQL and Real Application Testing (RAT)

2019-08-27 Thread Jaime Casanova
On Tue, 27 Aug 2019 at 19:33, Nikolay Samokhvalov wrote: > On Tue, Aug 27, 2019 at 3:47 AM ROS Didier wrote: > >> Hi >> >> >> >> In my business, one of the things blocking the migration from Oracle to >> PostgreSQL is not having the equivalent of Oracle Real Application Testing . >> >> This prod

Re: PostgreSQL and Real Application Testing (RAT)

2019-08-27 Thread Nikolay Samokhvalov
On Tue, Aug 27, 2019 at 3:47 AM ROS Didier wrote: > Hi > > > > In my business, one of the things blocking the migration from Oracle to > PostgreSQL is not having the equivalent of Oracle Real Application Testing . > > This product captures a charge in production and replay it in a test > environm

Re: Zedstore - compressed in-core columnar storage

2019-08-27 Thread Alexandra Wang
On Tue, Aug 27, 2019 at 12:03 AM Ashutosh Sharma wrote: > My point is, once we find the leaf page containing the given tid, we go > through each item in the page until we find the data corresponding to the > given tid which means we kind of perform a sequential scan at the page > level. I'm refer

Re: pg11.5: ExecHashJoinNewBatch: glibc detected...double free or corruption (!prev)

2019-08-27 Thread Merlin Moncure
On Sun, Aug 25, 2019 at 9:35 PM Thomas Munro wrote: > > On Mon, Aug 26, 2019 at 1:44 PM Justin Pryzby wrote: > > On Mon, Aug 26, 2019 at 01:09:19PM +1200, Thomas Munro wrote: > > > On Sun, Aug 25, 2019 at 3:15 PM Peter Geoghegan wrote: > > > > I was reminded of this issue from last year, which a

Re: doc: clarify "pg_signal_backend" default role

2019-08-27 Thread Tom Lane
Ian Barwick writes: > Currently the documentation for the default role "pg_signal_backend" states, > somewhat ambiguously, "Send signals to other backends (eg: cancel query, > terminate)", > giving the impression other signals (e.g. SIGHUP) can be sent too, which is > currently not the case. > At

Re: row filtering for logical replication

2019-08-27 Thread a . kondratov
Hi Euler, On 2019-02-03 13:14, Andres Freund wrote: On 2018-11-23 13:15:08 -0300, Euler Taveira wrote: Besides the problem presented by Hironobu-san, I'm doing some cleanup and improving docs. I also forget to declare pg_publication_rel TOAST table. Thanks for your review. As far as I can t

Re: Missing newline in pg_upgrade usage()

2019-08-27 Thread Tom Lane
Daniel Gustafsson writes: > In 959f6d6a1821 it seems that I fat-fingered and missed a newline in the usage > output, which makes --check become appended to -B rather than on its own line. > The attached diff fixes that. Roger, pushed. regards, tom lane

Missing newline in pg_upgrade usage()

2019-08-27 Thread Daniel Gustafsson
In 959f6d6a1821 it seems that I fat-fingered and missed a newline in the usage output, which makes --check become appended to -B rather than on its own line. The attached diff fixes that. cheers ./daniel pg_upgrade_help_check.diff Description: Binary data

Re: PostgreSQL and Real Application Testing (RAT)

2019-08-27 Thread Jaime Casanova
On Tue, 27 Aug 2019 at 05:47, ROS Didier wrote: > Hi > > > > In my business, one of the things blocking the migration from Oracle to > PostgreSQL is not having the equivalent of Oracle Real Application Testing . > > This product captures a charge in production and replay it in a test > environmen

Re: block-level incremental backup

2019-08-27 Thread Robert Haas
On Fri, Aug 16, 2019 at 6:23 AM Jeevan Chalke wrote: > [ patches ] Reviewing 0002 and 0003: - Commit message for 0003 claims magic number and checksum are 0, but that (fortunately) doesn't seem to be the case. - looks_like_rel_name actually checks whether it looks like a *non-temporary* relatio

basebackup.c's sendFile() ignores read errors

2019-08-27 Thread Robert Haas
While reviewing a proposed patch to basebackup.c this morning, I found myself a bit underwhelmed by the quality of the code and comments in basebackup.c's sendFile(). I believe it's already been pointed out that the the retry logic here is not particularly correct, and the comments demonstrate a pr

Re: pgbench - implement strict TPC-B benchmark

2019-08-27 Thread Dmitry Dolgov
> On Mon, Aug 5, 2019 at 10:46 PM Fabien COELHO wrote: > > > The index builds are done serially. The vacuum could be replaced by COPY > > FREEZE. > > Well, it could be added? While doing benchmarking using different tools, including pgbench, I found it useful as a temporary hack to add copy freez

Re: range bug in resolve_generic_type?

2019-08-27 Thread Paul A Jungwirth
On Tue, Aug 27, 2019 at 8:23 AM Tom Lane wrote: > > resolve_generic_type(ANYARRAYOID, x, ANYRANGEOID) - this will return > > an array of the *range type*, but that contracts the normal > > relationship between anyelement and anyrange. It should return an > > array of the range's element type. > >

Re: range bug in resolve_generic_type?

2019-08-27 Thread Tom Lane
Paul A Jungwirth writes: > I was looking at resolve_generic_type to add anymultirange support, > and the range logic doesn't seem correct to me. Hmmm... > resolve_generic_type(ANYARRAYOID, x, ANYRANGEOID) - this will return > an array of the *range type*, but that contracts the normal > relation

Re: old_snapshot_threshold vs indexes

2019-08-27 Thread Tom Lane
Thomas Munro writes: > On Tue, Aug 27, 2019 at 1:54 PM Tom Lane wrote: >> +1. That fix is also back-patchable, which adding fields to relcache >> entries would not be. > There is a fly in the ointment: REL9_6_STABLE's copy of > RelationHasUnloggedIndex() is hardcoded to return true for hash > i

range bug in resolve_generic_type?

2019-08-27 Thread Paul A Jungwirth
Hello, I was looking at resolve_generic_type to add anymultirange support, and the range logic doesn't seem correct to me. This function takes 3 type Oids: - declared_type is the declared type of a function parameter whose actual type it would like to deduce. - context_{declared,actual}_type are

Re: "ago" times on buildfarm status page

2019-08-27 Thread Andrew Dunstan
On 8/27/19 10:15 AM, Tom Lane wrote: > "Tom Turelinckx" writes: >> On Mon, Aug 26, 2019, at 9:08 PM, Andrew Dunstan wrote: >>> I think this is the problem: >>> 'scmrepo' => '/home/pgbf/pgmirror.git', >>> Probably this isn't updated often enough. It probably has little to do with >>> the clock s

Re: "ago" times on buildfarm status page

2019-08-27 Thread Tom Lane
"Tom Turelinckx" writes: > On Mon, Aug 26, 2019, at 9:08 PM, Andrew Dunstan wrote: >> I think this is the problem: >> 'scmrepo' => '/home/pgbf/pgmirror.git', >> Probably this isn't updated often enough. It probably has little to do with >> the clock settings. > The configuration is intentionally

Re: "ago" times on buildfarm status page

2019-08-27 Thread Andrew Dunstan
On 8/27/19 8:45 AM, Andrew Dunstan wrote: > On 8/27/19 4:33 AM, Tom Turelinckx wrote: >> On Mon, Aug 26, 2019, at 9:08 PM, Andrew Dunstan wrote: >>> I think this is the problem: >>> >>> 'scmrepo' => '/home/pgbf/pgmirror.git', >>> >>> Probably this isn't updated often enough. It probably has litt

Re: PostgreSQL and Real Application Testing (RAT)

2019-08-27 Thread Thomas Kellerer
ROS Didier schrieb am 27.08.2019 um 12:47: > In my business, one of the things blocking the migration from Oracle > to PostgreSQL is not having the equivalent of Oracle Real Application > Testing . > > This product captures a charge in production and replay it in a test > environment. > > this allo

Re: Why overhead of SPI is so large?

2019-08-27 Thread Robert Haas
On Sat, Aug 24, 2019 at 12:01 PM David Fetter wrote: > No, it's lying to the RDBMS, so it's pilot error. The problem of > determining from the function itself whether it is in fact immutable > is, in general, equivalent to the Halting Problem, so no, we can't > figure it out. We do need to trust o

Re: Cleanup isolation specs from unused steps

2019-08-27 Thread Asim R P
On Fri, Aug 23, 2019 at 9:08 PM Alvaro Herrera wrote: > > On 2019-Aug-23, Asim R P wrote: > > > As part of the fault injector patch set [1], I added a new "blocking" > > keyword to isolation grammar so that a step can be declared as blocking. > > See patch 0002-Add-syntax-to-declare-a-step-that-is

Re: Fault injection framework

2019-08-27 Thread Asim R P
On Tue, Aug 27, 2019 at 12:35 PM Michael Paquier wrote: > > You may want to double-check whitespaces in your patch set, and 0002 > does not apply because of conflicts in isolationtester.h (my fault!). > I've rebased the patch set against the latest master and tried to resolve whitespace issues.

Re: "ago" times on buildfarm status page

2019-08-27 Thread Andrew Dunstan
On 8/27/19 4:33 AM, Tom Turelinckx wrote: > On Mon, Aug 26, 2019, at 9:08 PM, Andrew Dunstan wrote: >> I think this is the problem: >> >> 'scmrepo' => '/home/pgbf/pgmirror.git', >> >> Probably this isn't updated often enough. It probably has little to do with >> the clock settings. >> >> This i

Re: block-level incremental backup

2019-08-27 Thread vignesh C
On Fri, Aug 16, 2019 at 8:07 PM Ibrar Ahmed wrote: > > What do you mean by bigger file, a file greater than 1GB? In which case you > get file > 1GB? > > > Few comments: Comment: + buf = (char *) malloc(statbuf->st_size); + if (buf == NULL) + ereport(ERROR, + (errcode(ERRCODE_OUT_OF_MEMORY), + err

PostgreSQL and Real Application Testing (RAT)

2019-08-27 Thread ROS Didier
Hi In my business, one of the things blocking the migration from Oracle to PostgreSQL is not having the equivalent of Oracle Real Application Testing . This product captures a charge in production and replay it in a test environment. this allows to know the impacts of a migration to a newer vers

Re: Statement timeout in pg_rewind

2019-08-27 Thread Alexander Kukushkin
On Tue, 27 Aug 2019 at 08:36, Michael Paquier wrote: > I'd rather be on the safe side and as we are looking at this at this > area.. Who knows if this logic is going to change in the future and > how it will change. Agree. > Oops, I misread this part. What about a simple wrapper > run_simple_

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-27 Thread Kyotaro Horiguchi
Hello. At Sun, 25 Aug 2019 22:08:43 -0700, Noah Misch wrote in <20190826050843.gb3153...@rfd.leadboat.com> noah> On Thu, Aug 22, 2019 at 09:06:06PM +0900, Kyotaro Horiguchi wrote: noah> > At Mon, 19 Aug 2019 23:03:14 -0700, Noah Misch wrote in <20190820060314.ga3086...@rfd.leadboat.com> > > >

Re: Creating partitions automatically at least on HASH?

2019-08-27 Thread Rafia Sabih
On Mon, 26 Aug 2019 at 19:46, Fabien COELHO wrote: > > Hello Rafia, > > >>CREATE TABLE Stuff (...) > >> PARTITION BY [HASH | RANGE | LIST] (…) > >>DO NONE -- this is the default > >>DO [IMMEDIATE|DEFERRED] USING (…) > >> > >> Where the USING part would be generic keword v

Re: "ago" times on buildfarm status page

2019-08-27 Thread Tom Turelinckx
On Mon, Aug 26, 2019, at 9:08 PM, Andrew Dunstan wrote: > I think this is the problem: > > 'scmrepo' => '/home/pgbf/pgmirror.git', > > Probably this isn't updated often enough. It probably has little to do with > the clock settings. > > This is the kind of old-fashioned way of doing things. Th

Re: pgbench - allow to create partitioned tables

2019-08-27 Thread Asif Rehman
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested Thanks. All looks good, making it ready for committer. Regards, Asif

Re: A problem about partitionwise join

2019-08-27 Thread Richard Guo
On Tue, Aug 27, 2019 at 8:51 AM Amit Langote wrote: > Hi Richard, > > On Mon, Aug 26, 2019 at 6:33 PM Richard Guo wrote: > > > > Hi All, > > > > To generate partitionwise join, we need to make sure there exists an > > equi-join condition for each pair of partition keys, which is performed > > by

Re: Re: Email to hackers for test coverage

2019-08-27 Thread movead...@highgo.ca
On Tue, 27 Aug 2019 14:07:48 +0800 mich...@paquier.xyz wrote: > There is a section in float4.sql which deals with overflow and > underflow, so wouldn't it be better to move the tests there? You > could just trigger the failures with that: > =# insert into float4_tbl values ('-10e-70'::float8);

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-27 Thread Kyotaro Horiguchi
At Tue, 27 Aug 2019 15:49:32 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190827.154932.250364935.horikyota@gmail.com> > 128GB shared buffers contain 16M buffers. On my > perhaps-Windows-Vista-era box, such loop takes 15ms. (Since it > has only 6GB, the test is ignoring the eff

Re: Optimize single tuple fetch from nbtree index

2019-08-27 Thread Floris Van Nee
>> It seems that it contradicts the very idea of your patch, so probably we >> should look for other ways to optimize this use-case. >> Maybe this restriction can be relaxed for write only tables, that never >> have to reread the page because of visibility, or something like that. >> Also we proba

Re: Fault injection framework

2019-08-27 Thread Michael Paquier
On Thu, Aug 22, 2019 at 07:45:09PM +0530, Asim R P wrote: > Fault injection was discussed a few months ago at PGCon in Ottawa. At > least a few folks showed interest and so I would like to present what we > have been using in Greenplum. > > The attached patch set contains the fault injector frame

Re: Zedstore - compressed in-core columnar storage

2019-08-27 Thread Ashutosh Sharma
On Tue, Aug 27, 2019 at 6:03 AM Ashwin Agrawal wrote: > Hope that helps to clarify the confusion. > Thanks for the explanation. Yes, it does clarify my doubt to some extent. My point is, once we find the leaf page containing the given tid, we go through each item in the page until we find the da