Re: In-place index updates and HOT (Was: [HACKERS] Patch: Write Amplification Reduction Method (WARM))

2017-07-28 Thread Claudio Freire
On Fri, Jul 28, 2017 at 8:32 PM, Peter Geoghegan wrote: > README.HOT says that that cost is not worth the benefit of > preventing a new index write, but I think that it ought to take into > account that not all index writes are equal. There is an appreciable > difference between inserting a new tu

Re: [HACKERS] AlterUserStmt anmd RoleSpec rules in grammar.y

2017-07-28 Thread Robert Haas
On Thu, Jul 27, 2017 at 2:52 AM, Pavel Golub wrote: > One more notice. ALTER USER ALL works in EnterpriseDB 10beta2 > installer. That's weird. I thought EnterpriseDB uses official sources. I find it really hard to believe that we're doing anything else. It wouldn't make any sense to patc

[HACKERS] [PATCH] Vacuum: Update FSM more frequently

2017-07-28 Thread Claudio Freire
As discussed in the vacuum ring buffer thread: Make Vacuum update the FSM more frequently, to avoid the case where autovacuum fails to reach the point where it updates the FSM in highly contended tables. Introduce a tree pruning threshold to FreeSpaceMapVacuum that avoids recursing into branches

In-place index updates and HOT (Was: [HACKERS] Patch: Write Amplification Reduction Method (WARM))

2017-07-28 Thread Peter Geoghegan
Pavan Deolasee wrote: One good thing is that the patch is ready and fully functional. So that allows those who are keen to run real performance tests and see the actual impact of the patch. Very true. I see your point. But I would like to think this way: does the technology significantly hel

Re: [HACKERS] psql's \d and \dt are sending their complaints to different output files

2017-07-28 Thread Tom Lane
Daniel Gustafsson writes: > On 27 Jul 2017, at 19:40, Tom Lane wrote: >> listTables and listDbRoleSettings print a custom message rather than >> an empty table for no matches (but in QUIET mode they just do the >> latter). I think that's actually a good decision for listDbRoleSettings, >> becaus

Re: [HACKERS] Adding support for Default partition in partitioning

2017-07-28 Thread Robert Haas
On Fri, Jul 28, 2017 at 9:30 AM, Ashutosh Bapat wrote: > 0004 patch > The patch adds another column partdefid to catalog pg_partitioned_table. The > column gives OID of the default partition for a given partitioned table. This > means that the default partition's OID is stored at two places 1. in

Re: [HACKERS] PL_stashcache, or, what's our minimum Perl version?

2017-07-28 Thread Tom Lane
Andrew Dunstan writes: >> On 07/27/2017 11:58 PM, Tom Lane wrote: >>> I wonder if it'd be worth getting the buildfarm >>> to log the output of "perl -V" so we could get a clearer picture >>> of what's being tested. > Looks like this, bit it's rather tedious. I think I might back it out. I > guess

Re: [HACKERS] pl/perl extension fails on Windows

2017-07-28 Thread Tom Lane
Ashutosh Sharma writes: > I quickly ran the some basic test-cases on my Windows machine (machine > where i have been regenerating the issue) and they are all passing. > Also, all the automated test-cases for contrib module hstore_plperl > are passing. Cool, thanks. I hope people will set up some

Re: [HACKERS] pl/perl extension fails on Windows

2017-07-28 Thread Tom Lane
Christoph Berg writes: > The plperl segfault on Debian's kfreebsd port I reported back in 2013 > is also still present: > https://www.postgresql.org/message-id/20130515064201.GC704%40msgid.df7cb.de > https://buildd.debian.org/status/fetch.php?pkg=postgresql-10&arch=kfreebsd-amd64&ver=10~beta2-1&st

Re: [HACKERS] map_partition_varattnos() and whole-row vars

2017-07-28 Thread Robert Haas
On Fri, Jul 28, 2017 at 3:26 PM, Tom Lane wrote: >> (Boy, our implementation of DROP COLUMN is painful! If we really got >> rid of columns when they were dropped we could've avoided this whole >> mess.) > > I think the pain arises mostly from the decision to allow partitions > to not all have ide

Re: [HACKERS] pl/perl extension fails on Windows

2017-07-28 Thread Ashutosh Sharma
On Sat, Jul 29, 2017 at 12:05 AM, Tom Lane wrote: > Ashutosh Sharma writes: >> On Fri, Jul 28, 2017 at 10:05 PM, Tom Lane wrote: >>> Uh-huh. So the issue is indeed that they're injecting PERL_IMPLICIT_SYS >>> via a command-line #define rather than putting it into perl's config.h, >>> and that r

Re: [HACKERS] map_partition_varattnos() and whole-row vars

2017-07-28 Thread Tom Lane
Robert Haas writes: > If we're remapping the varattnos, I don't see how we can just pass > whole-row references through. I mean, if the partition and the parent > have different varattnos, then a whole-row attribute for one is a > different thing from a whole-row attribute for the other; the > He

Re: [HACKERS] [patch] pg_dump/pg_restore zerror() and strerror() mishap

2017-07-28 Thread Vladimir Kunschikov
I haven't seen that but can't guarantee that such case does not exist 28 июля 2017 г. 9:19 PM пользователь "Alvaro Herrera" < alvhe...@2ndquadrant.com> написал: > Vladimir Kunschikov wrote: > > >This "maxlen" business and the fallback error message are > > >strange. We have roughly equivalent co

Re: [HACKERS] map_partition_varattnos() and whole-row vars

2017-07-28 Thread Peter Geoghegan
Robert Haas wrote: (Boy, our implementation of DROP COLUMN is painful! If we really got rid of columns when they were dropped we could've avoided this whole mess.) I tend to agree. I can recall several cases where it led to bugs that went undetected for quite a while. -- Peter Geoghegan --

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2017-07-28 Thread Mark Rofail
On Fri, Jul 28, 2017 at 1:19 PM, Erik Rijkers wrote: > One small thing while building docs: > > $ cd doc/src/sgml && make html > osx -wall -wno-unused-param -wno-empty -wfully-tagged -D . -D . -x lower > postgres.sgml >postgres.xml.tmp > osx:ref/create_table.sgml:960:100:E: document type does no

Re: [HACKERS] PL_stashcache, or, what's our minimum Perl version?

2017-07-28 Thread Andrew Dunstan
On 07/28/2017 08:22 AM, Andrew Dunstan wrote: > > On 07/27/2017 11:58 PM, Tom Lane wrote: >> I kinda suspect we're not actively testing non-MULTIPLICITY builds >> either. The 5.8.7 test I just ran was with a non-MULTIPLICITY build, >> so the case doesn't seem actively broken, but I doubt there i

Re: [HACKERS] map_partition_varattnos() and whole-row vars

2017-07-28 Thread Robert Haas
On Fri, Jul 28, 2017 at 1:06 AM, Noah Misch wrote: > [Action required within three days. This is a generic notification.] > > The above-described topic is currently a PostgreSQL 10 open item. Robert, > since you committed the patch believed to have created it, you own this open > item. If some

Re: [HACKERS] [GSOC] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

2017-07-28 Thread Robert Haas
On Wed, Jul 26, 2017 at 11:41 AM, Mengxing Liu < liu-m...@mails.tsinghua.edu.cn> wrote: > Hi, all. There was a very strange phenomenon I couldn't explain. So I was > wondering if you can help me. > > I was trying to replace the linked list with a skip list in serializable > transaction object for

Re: [HACKERS] segfault in HEAD when too many nested functions call

2017-07-28 Thread Robert Haas
On Fri, Jul 28, 2017 at 12:29 PM, Noah Misch wrote: > Your colleagues achieve compliance despite uncertainty; for inspiration, I > recommend examining Alvaro's status updates as examples of this. The policy > currently governs your open items even if you disagree with it. I emphatically agree wi

Re: [HACKERS] pl/perl extension fails on Windows

2017-07-28 Thread Tom Lane
Ashutosh Sharma writes: > On Fri, Jul 28, 2017 at 10:05 PM, Tom Lane wrote: >> Uh-huh. So the issue is indeed that they're injecting PERL_IMPLICIT_SYS >> via a command-line #define rather than putting it into perl's config.h, >> and that results in a change in the apparent size of the PerlInterp

Re: [HACKERS] [patch] pg_dump/pg_restore zerror() and strerror() mishap

2017-07-28 Thread Alvaro Herrera
Vladimir Kunschikov wrote: > >This "maxlen" business and the fallback error message are > >strange. We have roughly equivalent code in pg_basebackup.c > >which has been working since 2011 > >Perhaps you can drop the memchr/fallback tricks and adopt the > >pg_basebackup coding? Or is there a speci

Re: [HACKERS] [patch] pg_dump/pg_restore zerror() and strerror() mishap

2017-07-28 Thread Vladimir Kunschikov
>This "maxlen" business and the fallback error message are >strange. We have roughly equivalent code in pg_basebackup.c >which has been working since 2011 >Perhaps you can drop the memchr/fallback tricks and adopt the >pg_basebackup coding? Or is there a specific reason to have >the memchr check?

Re: [HACKERS] segfault in HEAD when too many nested functions call

2017-07-28 Thread Tom Lane
Andres Freund writes: > Anyway, I'll commit it after another pass in ~1 week if it doesn't get a > review till then, but I assume it'll. FWIW, I intend to review it today, or tomorrow at the very latest. (Right now I'm buried in perl droppings.) regards, tom lane -- Se

Re: [HACKERS] UPDATE of partition key

2017-07-28 Thread Amit Khandekar
On 28 July 2017 at 20:10, Robert Haas wrote: > On Wed, Jul 26, 2017 at 2:13 AM, Amit Langote > wrote: >> Sorry to be responding this late to the Amit's make_resultrel_ordered >> patch itself, but I agree that we should teach the planner to *always* >> expand partitioned tables in the partition bo

[HACKERS] Red-Black tree traversal tests

2017-07-28 Thread Victor Drobny
Hello, Postgres now has its own red-black tree implementation. This tree has 4 types of traversals. In the attachment, you can find module test that checks the correctness of tree traversal strategies. I hope that someone can find it useful. Thank you for attention! -- -- Victor Drobn

Re: [HACKERS] pl/perl extension fails on Windows

2017-07-28 Thread Ashutosh Sharma
On Fri, Jul 28, 2017 at 10:05 PM, Tom Lane wrote: > Ashutosh Sharma writes: >> On Fri, Jul 28, 2017 at 7:22 PM, Tom Lane wrote: >>> Assuming that the Perl crew know what they're doing and this list is >>> complete, I notice that not one of the symbols they show as relevant >>> starts with an und

Re: [HACKERS] segfault in HEAD when too many nested functions call

2017-07-28 Thread Andres Freund
On 2017-07-28 09:29:58 -0700, Noah Misch wrote: > On Thu, Jul 27, 2017 at 10:08:57PM -0700, Andres Freund wrote: > > For me that means the policy isn't quite right. It's not like I can > > force Tom to review the patch at a specific date. But the thread has > > been progressing steadily over the l

Re: [HACKERS] pl/perl extension fails on Windows

2017-07-28 Thread Tom Lane
Ashutosh Sharma writes: > On Fri, Jul 28, 2017 at 7:22 PM, Tom Lane wrote: >> Assuming that the Perl crew know what they're doing and this list is >> complete, I notice that not one of the symbols they show as relevant >> starts with an underscore. So I'm thinking that my previous semi- >> jokin

Re: [HACKERS] segfault in HEAD when too many nested functions call

2017-07-28 Thread Noah Misch
On Thu, Jul 27, 2017 at 10:08:57PM -0700, Andres Freund wrote: > On 2017-07-27 22:04:59 -0700, Noah Misch wrote: > > On Thu, Jul 27, 2017 at 09:49:18PM -0700, Andres Freund wrote: > > > On 2017-07-27 21:46:57 -0700, Noah Misch wrote: > > > > On Thu, Jul 27, 2017 at 02:29:32AM +, Noah Misch wrot

Re: LP_DEAD hinting and not holding on to a buffer pin on leaf page (Was: [HACKERS] [WIP] Zipfian distribution in pgbench)

2017-07-28 Thread Peter Geoghegan
Amit Kapila wrote: Isn't it possible to confirm if the problem is due to commit 2ed5b87f9? Basically, if we have unlogged tables, then it won't release the pin. So if the commit in question is the culprit, then the same workload should not lead to bloat. That's a great idea. Alik? -- Peter

Re: [HACKERS] [patch] pg_dump/pg_restore zerror() and strerror() mishap

2017-07-28 Thread Alvaro Herrera
Kunshchikov Vladimir wrote: > Hello Alvaro, > > here goes v4 version: removed unused header. > > Compilation of this code snippet with -Wall -Wexter -std=c89 doesn't produce > any warnings. Great, thanks. +const char * +get_cfp_error(cfp* fp) +{ +#ifdef HAVE_LIBZ + if(fp->compressedfp){

Re: [HACKERS] tab complete for psql pset pager values

2017-07-28 Thread Jeff Janes
On Wed, Jul 26, 2017 at 8:02 AM, Pavel Stehule wrote: > Hi > > attached trivial patch for missing tab complete for \pset pager setting > > Looks good to me. I've set it ready for committer. Cheers, Jeff

Re: [HACKERS] pl/perl extension fails on Windows

2017-07-28 Thread Ashutosh Sharma
On Fri, Jul 28, 2017 at 7:22 PM, Tom Lane wrote: > Ashutosh Sharma writes: >> Thanks for the patch. I am seeing some compilation errors on Windows >> with the patch. Below are the errors observed, >> ... >> I did spent some time to find reason for these compilation errors and >> could eventually

[HACKERS] Re: [bug fix] PG10: libpq doesn't connect to alternative hosts when some errors occur

2017-07-28 Thread Robert Haas
On Fri, Jul 28, 2017 at 1:30 AM, Noah Misch wrote: > On Fri, May 19, 2017 at 11:08:41AM +0900, Michael Paquier wrote: >> On Fri, May 19, 2017 at 11:01 AM, Tsunakawa, Takayuki >> wrote: >> > From: pgsql-hackers-ow...@postgresql.org >> >> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Pet

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-07-28 Thread Robert Haas
On Fri, Jul 28, 2017 at 12:39 AM, Pavan Deolasee wrote: > I see your point. But I would like to think this way: does the technology > significantly help many common use cases, that are currently not addressed > by HOT? It probably won't help all workloads, that's given. Also, we don't > have any c

Re: [HACKERS] UPDATE of partition key

2017-07-28 Thread Robert Haas
On Wed, Jul 26, 2017 at 2:13 AM, Amit Langote wrote: > Sorry to be responding this late to the Amit's make_resultrel_ordered > patch itself, but I agree that we should teach the planner to *always* > expand partitioned tables in the partition bound order. Sounds like we have unanimous agreement o

[HACKERS] Clarification in pg10's pgupgrade.html step 10 (upgrading standby servers)

2017-07-28 Thread Andreas Joseph Krogh
Hi -hackers.   I'm reading https://www.postgresql.org/docs/10/static/pgupgrade.html to try to understand how to upgrade standby-servers using pg_upgrade with pg10.   The text in step 10 sais: "You will not be running pg_upgrade on the standby servers, but rather rsync", which to me sounds like rs

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2017-07-28 Thread Michael Paquier
On Fri, Jul 28, 2017 at 7:28 AM, Masahiko Sawada wrote: > That also requires to share the same XID space with all remote nodes. You are putting your finger on the main bottleneck with global consistency that XC and XL has because of that. And the source feeding the XIDs is a SPOF. > Perhaps the

Re: [HACKERS] pl/perl extension fails on Windows

2017-07-28 Thread Tom Lane
Ashutosh Sharma writes: > Thanks for the patch. I am seeing some compilation errors on Windows > with the patch. Below are the errors observed, > ... > I did spent some time to find reason for these compilation errors and > could eventually find that some of the Windows specific functions > inside

Re: [HACKERS] Adding support for Default partition in partitioning

2017-07-28 Thread Ashutosh Bapat
On Wed, Jul 26, 2017 at 5:44 PM, Jeevan Ladhe wrote: > Hi, > > I have rebased the patches on the latest commit. > Thanks for rebasing the patches. The patches apply and compile cleanly. make check passes. Here are some review comments 0001 patch Most of this patch is same as 0002 patch posted in

Re: LP_DEAD hinting and not holding on to a buffer pin on leaf page (Was: [HACKERS] [WIP] Zipfian distribution in pgbench)

2017-07-28 Thread Amit Kapila
On Wed, Jul 26, 2017 at 3:32 AM, Peter Geoghegan wrote: > On Fri, Jul 14, 2017 at 5:06 PM, Peter Geoghegan wrote: >> I think that what this probably comes down to, more than anything >> else, is that you have leftmost hot/bloated leaf pages like this: >> >> >> idx | level | l_i

Re: [HACKERS] PL_stashcache, or, what's our minimum Perl version?

2017-07-28 Thread Andrew Dunstan
On 07/27/2017 11:58 PM, Tom Lane wrote: > > I kinda suspect we're not actively testing non-MULTIPLICITY builds > either. The 5.8.7 test I just ran was with a non-MULTIPLICITY build, > so the case doesn't seem actively broken, but I doubt there is any > buildfarm coverage. I wonder if it'd be wo

Re: [HACKERS] Update comments in nodeModifyTable.c

2017-07-28 Thread Etsuro Fujita
On 2017/07/26 22:39, Robert Haas wrote: On Wed, Jun 14, 2017 at 10:40 PM, Etsuro Fujita wrote: Attached is an updated version of the patch. Well, now I'm confused: * Initialize the junk filter(s) if needed. INSERT queries need a filter * if there are any junk attrs in the tlis

Re: [HACKERS] psql's \d and \dt are sending their complaints to different output files

2017-07-28 Thread Daniel Gustafsson
> On 27 Jul 2017, at 19:40, Tom Lane wrote: > > Daniel Gustafsson writes: >>> On 19 Jun 2017, at 17:32, Tom Lane wrote: >>> So, if we're getting into enforcing consistency in describe.c, there's >>> lots to do. > >> Addressed in attached patch, see list of patches below. > > I've pushed most

Re: [HACKERS] [BUGS] BUG #14759: insert into foreign data partitions fail

2017-07-28 Thread Etsuro Fujita
On 2017/07/26 15:29, Amit Langote wrote: On 2017/07/25 9:43, David G. Johnston wrote: On Mon, Jul 24, 2017 at 5:19 PM, Amit Langote wrote: On 2017/07/25 6:28, mtun...@gmail.com wrote: The following bug has been logged on the website: Bug reference: 14759 Logged by: Murat Tuncer

Re: [HACKERS] map_partition_varattnos() and whole-row vars

2017-07-28 Thread Amit Khandekar
On 28 July 2017 at 11:17, Amit Langote wrote: > On 2017/07/26 16:58, Amit Langote wrote: >> Rajkumar Raghuwanshi reported [1] on the "UPDATE partition key" thread >> that whole-row vars don't play nicely with partitioned table operations. >> >> For example, when used to convert WITH CHECK OPTION c

Re: [HACKERS] [patch] pg_dump/pg_restore zerror() and strerror() mishap

2017-07-28 Thread Kunshchikov Vladimir
Hello Alvaro, here goes v4 version: removed unused header. Compilation of this code snippet with -Wall -Wexter -std=c89 doesn't produce any warnings. -- Best regards, Vladimir Kunschikov Lead software developer IDS project InfoTeCS JSC From: Kunshchikov Vladimi

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2017-07-28 Thread Erik Rijkers
On 2017-07-27 21:08, Mark Rofail wrote: On Thu, Jul 27, 2017 at 7:15 PM, Erik Rijkers wrote: It would help (me at least) if you could be more explicit about what exactly each instance is. I apologize, I thought it was clear through the context. Thanks a lot. It's just really easy for tes

Re: [HACKERS] pl/perl extension fails on Windows

2017-07-28 Thread Ashutosh Sharma
Hi, On Fri, Jul 28, 2017 at 4:20 AM, Tom Lane wrote: > > Andrew Dunstan writes: > > On 07/27/2017 04:33 PM, Tom Lane wrote: > >> So I was trying to figure a way to not include XSUB.h except in a very > >> limited part of plperl, like ideally just the .xs files. It's looking > >> like that would

Re: [HACKERS] asynchronous execution

2017-07-28 Thread Kyotaro HORIGUCHI
Thank you for the comment. At Wed, 26 Jul 2017 17:16:43 -0400, Robert Haas wrote in > But if we do, then I fear we'll just be reintroducing the same > performance regression that we introduced by switching to this > framework from the previous one - or maybe a different one, but a > regression