Re: Joins on TID

2018-12-21 Thread Tom Lane
BTW, if we're to start taking joins on TID seriously, we should also add the missing hash opclass for TID, so that you can do hash joins when dealing with a lot of rows. (In principle this also enables things like hash aggregation, though I'm not very clear on a use-case for grouping by TID.)

Re: pgsql: Check for conflicting queries during replay of gistvacuumpage()

2018-12-21 Thread Alexander Korotkov
On Fri, Dec 21, 2018 at 7:09 PM Tom Lane wrote: > Alvaro Herrera writes: > >> Hmmm, I'm fairly sure you should have bumped XLOG_PAGE_MAGIC for this > >> change. Otherwise, what is going to happen to an unpatched standby (of > >> released versions) that receives the new WAL record from a patched

Re: Change pgarch_readyXlog() to return .history files first

2018-12-21 Thread Michael Paquier
On Fri, Dec 21, 2018 at 08:17:12AM +0200, David Steele wrote: > I thought about doing that, but wanted to focus on the task at hand. It > does save a strcpy and a bit of stack space, so seems like a win. > > Overall, the patch looks good to me. I think breaking up the if does make > the code mor

Re: [PATCH] Improve tab completion for CREATE TABLE

2018-12-21 Thread Michael Paquier
On Fri, Dec 21, 2018 at 03:14:36PM +, Dagfinn Ilmari Mannsåker wrote: > Here's a patch that does this (and in passing alphabetises the list of > options). Cool, thanks. The position of the option list is fine. However list_TABLEOPTIONS is not a name consistent with the surroundings. So we c

Re: Clean up some elog messages and comments for do_pg_stop_backup and do_pg_start_backup

2018-12-21 Thread Michael Paquier
On Fri, Dec 21, 2018 at 10:43:57AM -0300, Alvaro Herrera wrote: > errhint("Check that your archive_command is executing properly. " > + "Backup can be canceled safely, " > "but the database backup will not be usable without all the WAL > segments."))) > > I think repeat

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-21 Thread John Naylor
On 12/20/18, Tom Lane wrote: > I'd be inclined to put the script in src/tools, I think. IMO src/common > is for code that actually gets built into our executables. Done. >> which takes >> pl_unreserved_kwlist.h as input and outputs >> pl_unreserved_kwlist_offset.h and pl_unreserved_kwlist_strin

Re: could recovery_target_timeline=latest be the default in standby mode?

2018-12-21 Thread Michael Paquier
On Fri, Dec 21, 2018 at 01:54:20PM +0300, Sergei Kornilov wrote: > I am +1 for recovery_target_timeline=latest by default. This is > common case in my opinion. I agree also that switching to the latest timeline should be the default. People get confused because of the current default. -- Michael

Re: [PATCH] Improve tab completion for CREATE TABLE

2018-12-21 Thread Michael Paquier
On Fri, Dec 21, 2018 at 01:57:40PM +, Dagfinn Ilmari Mannsåker wrote: > Yeah, because of that we can't do the obvious HeadMatches("CREATE", > "TABLE") && (TailMatches(...) || TailMatches(...) || ...). I believe > this would require extending the match syntax with regex-like grouping, > alterna

Joins on TID

2018-12-21 Thread Tom Lane
I decided to spend an afternoon seeing exactly how much work would be needed to support parameterized TID scans, ie nestloop-with-inner-TID- scan joins, as has been speculated about before, most recently here: https://www.postgresql.org/message-id/flat/CAMqTPq%3DhNg0GYFU0X%2BxmuKy8R2ARk1%2BA_uQpS%

Re: Offline enabling/disabling of data checksums

2018-12-21 Thread Michael Paquier
On Fri, Dec 21, 2018 at 09:16:16PM +0100, Michael Banck wrote: > It adds an (now mandatory) --action parameter that takes either verify, > enable or disable as argument. There are two discussion points which deserve attention here: 1) Do we want to rename pg_verify_checksums to something else, lik

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-12-21 Thread David Rowley
On Fri, 21 Dec 2018 at 10:05, Robert Haas wrote: > On Thu, Dec 20, 2018 at 3:58 PM Alvaro Herrera > wrote: > > Namely: how does this handle the case of partition pruning structure > > being passed from planner to executor, if an attach happens in the > > middle of it and puts a partition in betw

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-12-21 Thread David Rowley
On Fri, 21 Dec 2018 at 09:43, Robert Haas wrote: > - I refactored expand_inherited_rtentry() to drive partition expansion > entirely off of PartitionDescs. The reason why this is necessary is > that it clearly will not work to have find_all_inheritors() use a > current snapshot to decide what chil

Re: Compiling on Termux

2018-12-21 Thread Tom Lane
Thomas Munro writes: > On Sat, Dec 22, 2018 at 9:19 AM David Fetter wrote: >> I don't know precisely how it's going to play with Postgres, but >> Termux does supply a Postgres in its native packages. That package >> appears to work, at least in the single-connection case, so they're >> doing som

Re: Compiling on Termux

2018-12-21 Thread Thomas Munro
On Sat, Dec 22, 2018 at 9:19 AM David Fetter wrote: > On Fri, Dec 21, 2018 at 04:24:20PM -0500, Andrew Dunstan wrote: > > On 12/21/18 4:04 PM, Thomas Munro wrote: > > > On Sat, Dec 22, 2018 at 7:32 AM David Fetter wrote: > > >> On Sat, Dec 22, 2018 at 07:03:32AM +1100, Thomas Munro wrote: > > >>>

Re: Compiling on Termux

2018-12-21 Thread David Fetter
On Fri, Dec 21, 2018 at 04:24:20PM -0500, Andrew Dunstan wrote: > On 12/21/18 4:04 PM, Thomas Munro wrote: > > On Sat, Dec 22, 2018 at 7:32 AM David Fetter wrote: > >> On Sat, Dec 22, 2018 at 07:03:32AM +1100, Thomas Munro wrote: > >>> That talks about using -D__ANDROID_API__=23 (or presumably hig

Re: Use an enum for RELKIND_*?

2018-12-21 Thread Greg Stark
Out of curiosity I built with -Wswitch-enum to see if it would be possible to just enable it. It looks like the main culprits are the node types and if those were switched to #defines it might be feasible to do so though it would still be a lot of hassle to add case labels all over the place. But

Re: [HACKERS] ArrayLists instead of List (for some things)

2018-12-21 Thread David Rowley
On Fri, 3 Nov 2017 at 03:17, Tom Lane wrote: > > David Rowley writes: > > Comments on the design are welcome, but I was too late to the > > commitfest, so there are other priorities. However, if you have a > > strong opinion, feel free to voice it. > > I do not like replacing Lists piecemeal; tha

Re: Performance issue in foreign-key-aware join estimation

2018-12-21 Thread David Rowley
On Sat, 22 Dec 2018 at 09:28, David Rowley wrote: > Going by my profiler this drops match_eclasses_to_foreign_key_col() > down to just 10% of total planner time for this query. The new > bms_is_member() call is pretty hot inside that function though. I should have said 28% instead of 10%. 10% is

Re: Compiling on Termux

2018-12-21 Thread Andrew Dunstan
On 12/21/18 4:04 PM, Thomas Munro wrote: > On Sat, Dec 22, 2018 at 7:32 AM David Fetter wrote: >> On Sat, Dec 22, 2018 at 07:03:32AM +1100, Thomas Munro wrote: >>> That talks about using -D__ANDROID_API__=23 (or presumably higher) to >>> make sure that sigtimedwait is exposed by signal.h. Somet

Re: Compiling on Termux

2018-12-21 Thread Thomas Munro
On Sat, Dec 22, 2018 at 7:32 AM David Fetter wrote: > On Sat, Dec 22, 2018 at 07:03:32AM +1100, Thomas Munro wrote: > > That talks about using -D__ANDROID_API__=23 (or presumably higher) to > > make sure that sigtimedwait is exposed by signal.h. Something similar > > may be afoot here. > > That w

Re: Compiling on Termux

2018-12-21 Thread David Fetter
On Sat, Dec 22, 2018 at 07:03:32AM +1100, Thomas Munro wrote: > On Sat, Dec 22, 2018 at 5:56 AM David Fetter wrote: > > > > Folks, > > > > I'm trying to compile master (c952eae52a33069e2e92d34f217b43d0eca3d7de) > > on Termux, using the supplied settings, as follows. > > > > pg_config --configure |

Re: Performance issue in foreign-key-aware join estimation

2018-12-21 Thread David Rowley
On Fri, 21 Dec 2018 at 06:44, Tom Lane wrote: > I was distressed to discover via perf that 69% of the runtime of this > test now goes into match_eclasses_to_foreign_key_col(). That seems > clearly unacceptable. Agreed. That's pretty terrible. I looked at this a bit and see that match_eclasses_t

Offline enabling/disabling of data checksums

2018-12-21 Thread Michael Banck
Hi, the attached patch adds offline enabling/disabling of checksums to pg_verify_checksums. It is based on independent work both Michael (Paquier) and me did earlier this year and takes changes from both, see https://github.com/credativ/pg_checksums and https://github.com/michaelpq/pg_plugins/tree

Re: Compiling on Termux

2018-12-21 Thread Thomas Munro
On Sat, Dec 22, 2018 at 5:56 AM David Fetter wrote: > > Folks, > > I'm trying to compile master (c952eae52a33069e2e92d34f217b43d0eca3d7de) > on Termux, using the supplied settings, as follows. > > pg_config --configure | xargs ./configure > configure.out 2>configure.err > make -j 4 > make.out 2> m

Re: slow queries over information schema.tables

2018-12-21 Thread Greg Stark
Just brainstorming here. Another crazy idea would be to get rid of "name" data type, at least from the user-visible planner point of view. It would probably have to be stored as a fixed length data type like today but with a one-byte length header. That would make it possible for the planner to use

Re: Tid scan improvements

2018-12-21 Thread Tom Lane
BTW, with respect to this bit in 0001: @@ -1795,6 +1847,15 @@ nulltestsel(PlannerInfo *root, NullTestType nulltesttype, Node *arg, return (Selectivity) 0; /* keep compiler quiet */ } } +else if (vardata.var && IsA(vardata.var, Var) && + ((Var *) vard

Re: A few new options for vacuumdb

2018-12-21 Thread Bossart, Nathan
On 12/21/18, 10:51 AM, "Robert Haas" wrote: > On Thu, Dec 20, 2018 at 11:48 AM Bossart, Nathan wrote: >> Either way, we'll still have to decide whether to fail or to silently >> skip the option if you do something like specify --min-mxid-age for a >> 9.4 server. > > +1 for fail. Sounds good. I'

Re: [suggestion]support UNICODE host variables in ECPG

2018-12-21 Thread Tom Lane
"Nagaura, Ryohei" writes: > Tsunakawa-san >> * What's the benefit of supporting UTF16 in host variables? > 1) As byte per character is constant in UTF16 encoding, it can process > strings more efficiently than other encodings. I don't think I buy that argument; it falls down as soon as you cons

Re: Tid scan improvements

2018-12-21 Thread Tom Lane
Edmund Horner writes: > Ok. I think that will simplify things. So if I follow you correctly, > we should do: > 1. If has_useful_pathkeys is true: generate pathkeys (for CTID ASC), > and use truncate_useless_pathkeys on them. > 2. If we have tid quals or pathkeys, emit a TID scan path. Check.

Re: A few new options for vacuumdb

2018-12-21 Thread Robert Haas
On Thu, Dec 20, 2018 at 11:48 AM Bossart, Nathan wrote: > Either way, we'll still have to decide whether to fail or to silently > skip the option if you do something like specify --min-mxid-age for a > 9.4 server. +1 for fail. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterpr

Re: Improve selectivity estimate for range queries

2018-12-21 Thread Tom Lane
"Yuzuko Hosoya" writes: > From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] >> At Thu, 20 Dec 2018 17:21:29 +0900, "Yuzuko Hosoya" >> wrote in >> <008701d4983d$02e731c0$08b59540$@lab.ntt.co.jp> >>> To handle such cases I've thought up of an idea based on a previous >>> commit[1] w

Re: A few new options for vacuumdb

2018-12-21 Thread Robert Haas
On Wed, Dec 19, 2018 at 9:05 PM Michael Paquier wrote: > > It does not seem clear whether the user wants us to process mytable > > only if it is at least 1 GB, or if we should process mytable in > > addition to any other relations over 1 GB. Either way, I think trying > > to support these combina

Re: Remove Deprecated Exclusive Backup Mode

2018-12-21 Thread Robert Haas
On Fri, Dec 21, 2018 at 1:18 AM David Steele wrote: > On 12/21/18 2:01 AM, Michael Paquier wrote: > > On Thu, Dec 20, 2018 at 12:29:48PM +0200, David Steele wrote: > >> Cannot move patch to the same commitfest, and multiple future commitfests > >> exist! > > > > I am not sure what it means either.

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-12-21 Thread Robert Haas
On Thu, Dec 20, 2018 at 4:38 PM Robert Haas wrote: > Lowering the lock level might also make something that was previously > safe into something unsafe, because now there's no longer a guarantee > that invalidation messages are received soon enough. With > AccessExclusiveLock, we'll send invalidat

Re: START/END line number for COPY FROM

2018-12-21 Thread Tom Lane
Surafel Temesgen writes: > Currently we can skip header line on COPY FROM but having the ability to > skip and stop copying at any line can use to divide long copy operation and > enable to copy a subset of the file and skipping footer. Attach is a patch > for it I do not think this is a good ide

Re: pgsql: Check for conflicting queries during replay of gistvacuumpage()

2018-12-21 Thread Alvaro Herrera
On 2018-Dec-21, Tom Lane wrote: > Alvaro Herrera writes: > > Hmmm, I'm fairly sure you should have bumped XLOG_PAGE_MAGIC for this > > change. Otherwise, what is going to happen to an unpatched standby (of > > released versions) that receives the new WAL record from a patched > > primary? > > W

Re: pgsql: Check for conflicting queries during replay of gistvacuumpage()

2018-12-21 Thread Tom Lane
Alvaro Herrera writes: >> Hmmm, I'm fairly sure you should have bumped XLOG_PAGE_MAGIC for this >> change. Otherwise, what is going to happen to an unpatched standby (of >> released versions) that receives the new WAL record from a patched >> primary? Oh, and if the answer to your question is no

Re: pgsql: Check for conflicting queries during replay of gistvacuumpage()

2018-12-21 Thread Tom Lane
Alvaro Herrera writes: > Hmmm, I'm fairly sure you should have bumped XLOG_PAGE_MAGIC for this > change. Otherwise, what is going to happen to an unpatched standby (of > released versions) that receives the new WAL record from a patched > primary? We can't change XLOG_PAGE_MAGIC in released bran

Re: Changes to pg_dump/psql following collation "C" in the catalog

2018-12-21 Thread Tom Lane
"Daniel Verite" writes: > One consequence of using the "C" collation in the catalog versus > the db collation is that pg_dump -t with a regexp may not find > the same tables as before. It happens when these conditions are > all met: > - the collation of the database is not "C" > - the regexp has l

Re: pgsql: Check for conflicting queries during replay of gistvacuumpage()

2018-12-21 Thread Alvaro Herrera
Hmmm, I'm fairly sure you should have bumped XLOG_PAGE_MAGIC for this change. Otherwise, what is going to happen to an unpatched standby (of released versions) that receives the new WAL record from a patched primary? -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Develo

Re: tickling the lesser contributor's withering ego

2018-12-21 Thread Erik Rijkers
On 2018-12-21 16:17, Alvaro Herrera wrote: On 2018-Dec-21, Tom Lane wrote: Alvaro Herrera writes: > I propose the following patch, which will make those links stable -- > then we can add the following links to the contributors page: > https://www.postgresql/org/docs/10/release-10.html#RELEASE-

Re: tickling the lesser contributor's withering ego

2018-12-21 Thread Alvaro Herrera
On 2018-Dec-21, Tom Lane wrote: > Alvaro Herrera writes: > > I propose the following patch, which will make those links stable -- > > then we can add the following links to the contributors page: > > https://www.postgresql/org/docs/10/release-10.html#RELEASE-10-ACKNOWLEDGEMENTS > > https://www.po

Changes to pg_dump/psql following collation "C" in the catalog

2018-12-21 Thread Daniel Verite
Hi, One consequence of using the "C" collation in the catalog versus the db collation is that pg_dump -t with a regexp may not find the same tables as before. It happens when these conditions are all met: - the collation of the database is not "C" - the regexp has locale-dependant parts - the nam

Re: [PATCH] Improve tab completion for CREATE TABLE

2018-12-21 Thread Dagfinn Ilmari Mannsåker
I wrote: > Another omission I just realised of is that it doesn't complete the list > of table storage options after after "WITH (". That should be fairly > easy to add (we already have the list for completing after ALTER TABLE > SET|RESET), but it's getting late here now. Here's a patch that do

Re: tickling the lesser contributor's withering ego

2018-12-21 Thread Tom Lane
Alvaro Herrera writes: > I propose the following patch, which will make those links stable -- > then we can add the following links to the contributors page: > https://www.postgresql/org/docs/10/release-10.html#RELEASE-10-ACKNOWLEDGEMENTS > https://www.postgresql/org/docs/11/release-11.html#RELEAS

Re: tickling the lesser contributor's withering ego

2018-12-21 Thread Alvaro Herrera
On 2018-Nov-04, Erik Rijkers wrote: > I wouldn't mind if this page: > https://www.postgresql.org/community/contributors/ > > contained a link to (contributors v11): > https://www.postgresql.org/docs/11/static/release-11.html#id-1.11.6.5.6 > > and to (contributors v10) > https://www.postgresql.

Re: [PATCH] Improve tab completion for CREATE TABLE

2018-12-21 Thread Dagfinn Ilmari Mannsåker
Michael Paquier writes: > On Thu, Dec 20, 2018 at 12:02:52AM +, Dagfinn Ilmari Mannsåker wrote: >> Point, fixed in the attached v4. OTOH, as I mentioned in my other >> email, that runs into the problem that it won't complete the actions >> after e.g. "CREATE TEMP TABLE FOO () WITH () ON COM

Re: Clean up some elog messages and comments for do_pg_stop_backup and do_pg_start_backup

2018-12-21 Thread Alvaro Herrera
On 2018-Dec-21, Michael Paquier wrote: > The thing is that the current messages are actually misleading, because > for base backups taken by the replication protocol pg_stop_backup is > never called, which is I think confusing. While looking around I have > also noticed that the top comments of d

Re: could recovery_target_timeline=latest be the default in standby mode?

2018-12-21 Thread Sergei Kornilov
Hello I am +1 for recovery_target_timeline=latest by default. This is common case in my opinion. And first release without recovery.conf is reasonable time for change default value. But i doubt if we can ignore recovery_target_timeline in standby and always use latest in standby. I have no use

pg_upgrade: Pass -j down to vacuumdb

2018-12-21 Thread Jesper Pedersen
Hi Hackers, Here is a patch that passes the -j option from pg_upgrade down to vacuumdb if supported. I'll add it to the January 'Fest. Thanks for considering ! Best regards, Jesper >From ea941f942830589469281e0d5c17740469c6aebc Mon Sep 17 00:00:00 2001 From: jesperpedersen Date: Fri, 21 De

Re: Progress reporting for pg_verify_checksums

2018-12-21 Thread Michael Banck
Hi, On Wed, Oct 03, 2018 at 11:50:36AM +1300, Thomas Munro wrote: > On Sat, Sep 29, 2018 at 1:07 AM Michael Banck > wrote: > Windows doesn't like sigaction: > > https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.15189 Thanks for the report and sorry for taking so long to rep

Re: Online verification of checksums

2018-12-21 Thread Michael Banck
Hi, On Thu, Dec 20, 2018 at 04:19:11PM +0100, Michael Banck wrote: > Yeah, new rebased version attached. By the way, one thing that this patch also fixes is checksum verification on basebackups (as pointed out the other day by my colleague Bernd Helmele): postgres@kohn:~$ initdb -k data postgres

could recovery_target_timeline=latest be the default in standby mode?

2018-12-21 Thread Peter Eisentraut
Is there ever a reason why you would *not* want recovery_target_timeline=latest in standby mode? pg_basebackup -R doesn't set it. That seems suboptimal. Perhaps this could be the default in standby mode, or even the implicit default, ignoring the recovery_target_timeline setting altogether. How

Re: Speeding up creating UPDATE/DELETE generic plan for partitioned table into a lot

2018-12-21 Thread Amit Langote
Kato-san, On 2018/12/21 15:36, Kato, Sho wrote: > Hi, > I want to speed up the creation of UPDATE/DELETE generic plan for tables > partitioned into a lot. > > Currently, creating a generic plan of UPDATE/DELTE for such table, planner > creates a plan to scan all partitions. > So it takes a very

Performance of SELECT in a table partitioned into a lot

2018-12-21 Thread Kato, Sho
Hi, I compared INSERT/UPDATE/DELETE/SELECT throughput with PostgreSQL and another dbms. For INSERT/DELETE/UPDATE, PostgreSQL performance is superior, but for SELECT, PostgreSQL performance is slightly lower than another dbms. Because information may be missing, it may be difficult, but do you k

RE: [suggestion]support UNICODE host variables in ECPG

2018-12-21 Thread Nagaura, Ryohei
Matsumura-san, Tsunakawa-san Thank you for reply. Tsunakawa-san > * What's the benefit of supporting UTF16 in host variables? There are two benefits. 1) As byte per character is constant in UTF16 encoding, it can process strings more efficiently than other encodings. 2) This enables C programmer

RE: Improve selectivity estimate for range queries

2018-12-21 Thread Yuzuko Hosoya
Hi, Thanks for the comments. I attach the v2 patch. > From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > Sent: Friday, December 21, 2018 12:25 PM > > Hello. > > At Thu, 20 Dec 2018 17:21:29 +0900, "Yuzuko Hosoya" > wrote in > <008701d4983d$02e731c0$08b59540$@lab.ntt.co.jp> > >

Re: Tid scan improvements

2018-12-21 Thread Edmund Horner
On Fri, 21 Dec 2018 at 16:31, Tom Lane wrote: > > Edmund Horner writes: > > For the forward scan, I seem to recall, from your merge join example, > > that it's useful to set the pathkeys even when there are no > > query_pathkeys. We just have to unconditionally set them so that the > > larger pl