Re: [HACKERS] Cannot shutdown subscriber after DROP SUBSCRIPTION

2017-02-01 Thread Michael Paquier
On Thu, Feb 2, 2017 at 2:14 AM, Fujii Masao wrote: > The lwlock would be released when an exception occurs, so I don't think > that TRY-CATCH is necessary here. Or it's necessary for another reason? +PG_CATCH(); +{ +LWLockRelease(LogicalRepLauncherLock); +PG_RE_THROW(); +

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Jim Nasby
On 2/1/17 3:36 PM, Michael Paquier wrote: On Thu, Feb 2, 2017 at 7:01 AM, Jim Nasby wrote: On 2/1/17 10:27 AM, Robert Haas wrote: This looks fine to me. This could go without the comments, they are likely going to be forgotten if any updates happen in the future. I'm confused... I put the c

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Michael Paquier
On Thu, Feb 2, 2017 at 9:17 AM, Jim Nasby wrote: > Speaking of which... I have a meeting in 15 minutes to discuss moving to a > server with 4TB of memory. With current limits shared buffers maxes at 16TB, > which isn't all that far in the future. While 16TB of shared buffers might > not be a good

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Andres Freund
On 2017-02-02 09:22:46 +0900, Michael Paquier wrote: > On Thu, Feb 2, 2017 at 9:17 AM, Jim Nasby wrote: > > Speaking of which... I have a meeting in 15 minutes to discuss moving to a > > server with 4TB of memory. With current limits shared buffers maxes at 16TB, > > which isn't all that far in th

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Andres Freund
On 2016-11-28 11:40:53 -0800, Jim Nasby wrote: > With current limits, the most bgwriter can do (with 8k pages) is 1000 pages > * 100 times/sec = 780MB/s. It's not hard to exceed that with modern > hardware. Should we increase the limit on bgwriter_lru_maxpages? FWIW, I think working on replacing b

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Robert Haas
On Wed, Feb 1, 2017 at 7:28 PM, Andres Freund wrote: > On 2016-11-28 11:40:53 -0800, Jim Nasby wrote: >> With current limits, the most bgwriter can do (with 8k pages) is 1000 pages >> * 100 times/sec = 780MB/s. It's not hard to exceed that with modern >> hardware. Should we increase the limit on b

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Andres Freund
On 2017-02-01 20:30:30 -0500, Robert Haas wrote: > On Wed, Feb 1, 2017 at 7:28 PM, Andres Freund wrote: > > On 2016-11-28 11:40:53 -0800, Jim Nasby wrote: > >> With current limits, the most bgwriter can do (with 8k pages) is 1000 pages > >> * 100 times/sec = 780MB/s. It's not hard to exceed that w

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Robert Haas
On Wed, Feb 1, 2017 at 8:35 PM, Andres Freund wrote: > On 2017-02-01 20:30:30 -0500, Robert Haas wrote: >> On Wed, Feb 1, 2017 at 7:28 PM, Andres Freund wrote: >> > On 2016-11-28 11:40:53 -0800, Jim Nasby wrote: >> >> With current limits, the most bgwriter can do (with 8k pages) is 1000 >> >> pa

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Andres Freund
On 2017-02-01 20:38:58 -0500, Robert Haas wrote: > On Wed, Feb 1, 2017 at 8:35 PM, Andres Freund wrote: > > On 2017-02-01 20:30:30 -0500, Robert Haas wrote: > >> On Wed, Feb 1, 2017 at 7:28 PM, Andres Freund wrote: > >> > On 2016-11-28 11:40:53 -0800, Jim Nasby wrote: > >> >> With current limits,

Re: [HACKERS] WAL consistency check facility

2017-02-01 Thread Robert Haas
On Tue, Jan 31, 2017 at 9:31 PM, Michael Paquier wrote: > On Wed, Feb 1, 2017 at 1:06 AM, Robert Haas wrote: >> On Thu, Jan 5, 2017 at 12:54 AM, Kuntal Ghosh >> wrote: >>> I've attached the patch with the modified changes. PFA. >> >> Can this patch check contrib/bloom? > > Only full pages are ap

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-02-01 Thread Kyotaro HORIGUCHI
Thank you for the comment. At Thu, 2 Feb 2017 01:26:03 +0900, Fujii Masao wrote in > > The attached patch does that. Usually it reads page headers only > > on segment boundaries, but once continuation record found (or > > failed to read the next page header, that is, the first record on > > the

Re: [HACKERS] [Bug fix] PQsendQuery occurs error when target_session_attrs is set to read-write

2017-02-01 Thread Higuchi, Daisuke
From: Ashutosh Bapat [mailto:ashutosh.ba...@enterprisedb.com] > Per the documentation [1], "PQgetResult must be called repeatedly > until it returns a null pointer, indicating that the command is > done.". The code in PQgetResult() under CONNECTION_CHECK_WRITABLE > case, violates that. The patch f

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Jim Nasby
On 2/1/17 4:27 PM, Andres Freund wrote: On 2017-02-02 09:22:46 +0900, Michael Paquier wrote: On Thu, Feb 2, 2017 at 9:17 AM, Jim Nasby wrote: Speaking of which... I have a meeting in 15 minutes to discuss moving to a server with 4TB of memory. With current limits shared buffers maxes at 16TB,

Re: [HACKERS] sequence data type

2017-02-01 Thread Michael Paquier
On Wed, Feb 1, 2017 at 10:02 AM, Michael Paquier wrote: > On Wed, Feb 1, 2017 at 1:11 AM, Peter Eisentraut > wrote: >> And here is a rebased patch for the original feature. I think this >> addresses all raised concerns and suggestions now. > > Thanks for the new version. That looks good to me af

Re: [HACKERS] Cannot shutdown subscriber after DROP SUBSCRIPTION

2017-02-01 Thread Kyotaro HORIGUCHI
At Thu, 2 Feb 2017 08:46:11 +0900, Michael Paquier wrote in > On Thu, Feb 2, 2017 at 2:14 AM, Fujii Masao wrote: > > The lwlock would be released when an exception occurs, so I don't think > > that TRY-CATCH is necessary here. Or it's necessary for another reason? > > +PG_CATCH(); > +{

Re: [HACKERS] pg_background contrib module proposal

2017-02-01 Thread Michael Paquier
On Fri, Jan 27, 2017 at 11:38 PM, Robert Haas wrote: > On Fri, Jan 27, 2017 at 9:14 AM, Peter Eisentraut > wrote: >> On 1/19/17 12:47 PM, Andrey Borodin wrote: >>> 4. There is some controversy on where implemented feature shall be: in >>> separate extension (as in this patch), in db_link, in som

Re: [HACKERS] autonomous transactions

2017-02-01 Thread Michael Paquier
On Sat, Jan 7, 2017 at 5:30 PM, Andrey Borodin wrote: > The new status of this patch is: Ready for Committer I don't think that this thread has reached a conclusion yet. From what I can see the last patch does not apply, so I have moved the patch to next CF with "waiting on author". -- Michael

Re: [HACKERS] [PATCH] Add GUCs for predicate lock promotion thresholds

2017-02-01 Thread Michael Paquier
On Tue, Jan 24, 2017 at 3:32 AM, Kevin Grittner wrote: > I will need some time to consider that Moved to CF 2017-03 for now. The last patch sent still applies. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www

Re: [HACKERS] Add pgstathashindex() to get hash index table statistics.

2017-02-01 Thread Michael Paquier
On Tue, Jan 24, 2017 at 8:36 PM, Kuntal Ghosh wrote: > Nothing else to add from my side. I'm marking this 'Ready for commiter'. Moved to CF 2017-03 with the same status. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Parallel Index Scans

2017-02-01 Thread Michael Paquier
On Wed, Feb 1, 2017 at 10:20 PM, Amit Kapila wrote: > makes sense, so changed accordingly. I have moved this patch to CF 2017-03. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hac

Re: [HACKERS] Performance improvement for joins where outer side is unique

2017-02-01 Thread Michael Paquier
On Tue, Jan 31, 2017 at 9:13 AM, David Rowley wrote: > On 31 January 2017 at 13:10, David Rowley > wrote: >> I've attached a patch which implements this. > > Please disregards previous patch. (I forgot git commit before git diff > to make the patch) > > I've attached the correct patch. Moved to

Re: [HACKERS] Logical Replication and Character encoding

2017-02-01 Thread Kyotaro HORIGUCHI
Hello. This version makes subscriber automatically set its database encoding to clinet_encoding (if not explicitly set). And changed the behavior when pg_server_to_client returns NULL to ERROR from sending original string. At Wed, 1 Feb 2017 08:39:41 +, "Shinoda, Noriyoshi" wrote in > Tha

Re: [HACKERS] Commit fest 2017-01 will begin soon!

2017-02-01 Thread Michael Paquier
On Tue, Jan 31, 2017 at 4:01 PM, Michael Paquier wrote: > The CF finishing soon, I have done a first pass on the patches > remaining on it, moving patches to the next CF or updating them. There > are still 48 patches still on the tracker when writing this email: > Needs review: 32. > Ready for Com

Re: [HACKERS] TRAP: FailedAssertion("!(hassrf)", File: "nodeProjectSet.c", Line: 180)

2017-02-01 Thread Tom Lane
Andres Freund writes: > On 2017-02-02 00:09:03 +0100, Erik Rijkers wrote: >> Something is broken in HEAD: > Hm. Indeed. > The issue is that we're generating ProjectSet nodes instead of Result > for the top-level nodes - but we currently assume that ProjectSet nodes > actually contain sets. I'm

Re: [HACKERS] Logical Replication and Character encoding

2017-02-01 Thread Euler Taveira
On 01-02-2017 00:05, Kyotaro HORIGUCHI wrote: > - Subscriber connects with client_encoding specification and the > output plugin pgoutput decide whether it accepts the encoding > or not. If the subscriber doesn't, pgoutput send data without > conversion. > I don't think storage without conve

Re: pg_authid.rolpassword format (was Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol)

2017-02-01 Thread David Rowley
On 2 February 2017 at 00:13, Heikki Linnakangas wrote: > Ok, I'll drop the second patch for now. I committed the first patch after > fixing the things you and Michael pointed out. Thanks for the review! dbd69118 caused small compiler warning for me. The attached fixed it. -- David Rowley

Re: [HACKERS] parallelize queries containing subplans

2017-02-01 Thread Amit Kapila
On Wed, Feb 1, 2017 at 10:04 PM, Robert Haas wrote: > On Tue, Jan 31, 2017 at 6:01 AM, Amit Kapila wrote: >> Moved this patch to next CF. > > So here is what seems to be the key hunk from this patch: > > /* > - * Since we don't have the ability to push subplans down to workers at > -

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-02-01 Thread Rafia Sabih
On Thu, Feb 2, 2017 at 1:19 AM, Thomas Munro wrote: > On Thu, Feb 2, 2017 at 3:34 AM, Rafia Sabih > wrote: >> 9 | 62928.88 | 59077.909 > > Thanks Rafia. At first glance this plan is using the Parallel Shared > Hash in one place where it should pay off, that is loading the orders > tabl

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-02-01 Thread Thomas Munro
On Thu, Feb 2, 2017 at 4:57 PM, Rafia Sabih wrote: > Apart from the previously reported regression, there appear one more > issue in this set of patches. At times, running a query using parallel > hash it hangs up and all the workers including the master shows the > following backtrace, Ugh, than

Re: [HACKERS] Commit fest 2017-01 will begin soon!

2017-02-01 Thread Amit Kapila
On Thu, Feb 2, 2017 at 8:50 AM, Michael Paquier wrote: > On Tue, Jan 31, 2017 at 4:01 PM, Michael Paquier > wrote: >> The CF finishing soon, I have done a first pass on the patches >> remaining on it, moving patches to the next CF or updating them. There >> are still 48 patches still on the track

Re: [HACKERS] TRAP: FailedAssertion("!(hassrf)", File: "nodeProjectSet.c", Line: 180)

2017-02-01 Thread Tom Lane
I wrote: > Andres Freund writes: >> Tom, do you have an opinion? > Yes, it's broken. split_pathtarget_at_srfs seems to be doing the right > thing, but then something later is recombining the last two steps. Ah, no, I take that back: split_pathtarget_at_srfs is doing the wrong thing. It's gener

Re: [HACKERS] [Bug fix] PQsendQuery occurs error when target_session_attrs is set to read-write

2017-02-01 Thread Ashutosh Bapat
On Thu, Feb 2, 2017 at 8:11 AM, Higuchi, Daisuke wrote: > From: Ashutosh Bapat [mailto:ashutosh.ba...@enterprisedb.com] >> Per the documentation [1], "PQgetResult must be called repeatedly >> until it returns a null pointer, indicating that the command is >> done.". The code in PQgetResult() under

Re: [HACKERS] [Bug fix] PQsendQuery occurs error when target_session_attrs is set to read-write

2017-02-01 Thread Ashutosh Bapat
Sorry, attached wrong patch. Here's the right one. On Thu, Feb 2, 2017 at 10:04 AM, Ashutosh Bapat wrote: > On Thu, Feb 2, 2017 at 8:11 AM, Higuchi, Daisuke > wrote: >> From: Ashutosh Bapat [mailto:ashutosh.ba...@enterprisedb.com] >>> Per the documentation [1], "PQgetResult must be called repeat

Re: [HACKERS] Commit fest 2017-01 will begin soon!

2017-02-01 Thread Michael Paquier
On Thu, Feb 2, 2017 at 1:23 PM, Amit Kapila wrote: > Many thanks to you for running the show. I think it might be okay if > one consolidated mail is sent for all the patches that are marked > "Returned with Feedback" or "Rejected" or "Moved to next CF". OTOH, > there is some value in sending a s

Re: [HACKERS] TRAP: FailedAssertion("!(hassrf)", File: "nodeProjectSet.c", Line: 180)

2017-02-01 Thread Andres Freund
On 2017-02-01 23:27:36 -0500, Tom Lane wrote: > I wrote: > > Andres Freund writes: > >> Tom, do you have an opinion? > > > Yes, it's broken. split_pathtarget_at_srfs seems to be doing the right > > thing, but then something later is recombining the last two steps. > > Ah, no, I take that back:

Re: [HACKERS] Parallel bitmap heap scan

2017-02-01 Thread Dilip Kumar
On Wed, Feb 1, 2017 at 11:09 PM, Robert Haas wrote: Hi Robert, Thanks for the review. > When Andres wrote in > https://www.postgresql.org/message-id/20161017201558.cr34stc6zvxy3...@alap3.anarazel.de > that you should try to share only the iteration arrays, I believe that > he meant the results

Re: [HACKERS] Enabling replication connections by default in pg_hba.conf

2017-02-01 Thread Peter Eisentraut
On 1/22/17 10:29 PM, Michael Paquier wrote: > As now wal_level = replica has become the default for Postgres 10, > could we consider as well making replication connections enabled by > default in pg_hba.conf? This requires just uncommenting a couple of > lines in pg_hba.conf.sample. Yes, I think t

Re: [HACKERS] [Bug fix] PQsendQuery occurs error when target_session_attrs is set to read-write

2017-02-01 Thread Michael Paquier
On Thu, Feb 2, 2017 at 1:34 PM, Ashutosh Bapat wrote: > On Thu, Feb 2, 2017 at 10:04 AM, Ashutosh Bapat > wrote: >> [1] https://www.postgresql.org/docs/devel/static/libpq-exec.html: >> PQgetvalue(). > The code expects that there will be two PQgetResult() calls required. > One to fetch the result

Re: [HACKERS] Should `pg_upgrade --check` check relation filenodes are present?

2017-02-01 Thread Peter Eisentraut
On 1/31/17 4:57 PM, Craig de Stigter wrote: > However, this problem was not caught by the `--check` command. I'm > looking at the source code and it appears that pg_upgrade does not > attempt to verify relation filenodes actually exist before proceeding, > whether using --check or not. The purpose

Re: [HACKERS] Cannot shutdown subscriber after DROP SUBSCRIPTION

2017-02-01 Thread Tom Lane
Kyotaro HORIGUCHI writes: > Then, the reason for the TRY-CATCH cluase is that I found that > some functions called from there can throw exceptions. Yes, but all LWLocks should be released by normal error recovery. It should not be necessary for this code to clean that up by hand. If it were neces

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-02-01 Thread Pavan Deolasee
On Thu, Feb 2, 2017 at 3:49 AM, Tom Lane wrote: > Alvaro Herrera writes: > > Tom Lane wrote: > >> I think what we ought to do about this is invent additional API > >> functions, say > >> > >> Oid CatalogTupleInsertWithInfo(Relation heapRel, HeapTuple tup, > >> CatalogIndexState indstate); > >> v

Re: [HACKERS] [Bug fix] PQsendQuery occurs error when target_session_attrs is set to read-write

2017-02-01 Thread Higuchi, Daisuke
From: Ashutosh Bapat [mailto:ashutosh.ba...@enterprisedb.com] >Sorry, attached wrong patch. Here's the right one. > The code expects that there will be two PQgetResult() calls required. > One to fetch the result of SHOW command and the other to extract NULL. > If we require more calls or unexpecte

Re: [HACKERS] Cannot shutdown subscriber after DROP SUBSCRIPTION

2017-02-01 Thread Michael Paquier
On Thu, Feb 2, 2017 at 2:13 PM, Tom Lane wrote: > Kyotaro HORIGUCHI writes: >> Then, the reason for the TRY-CATCH cluase is that I found that >> some functions called from there can throw exceptions. > > Yes, but all LWLocks should be released by normal error recovery. > It should not be necessar

[HACKERS] Provide list of subscriptions and publications in psql's completion

2017-02-01 Thread Michael Paquier
Hi all, While testing a bit the logical replication facility, I have bumped on the fast that psql's completion does not show the list of things already created. Attached is a patch. Thanks, -- Michael subs-psql-completion.patch Description: Binary data -- Sent via pgsql-hackers mailing list (

Re: [HACKERS] [Bug fix] PQsendQuery occurs error when target_session_attrs is set to read-write

2017-02-01 Thread Higuchi, Daisuke
From: Michael Paquier [mailto:michael.paqu...@gmail.com] >This has not been added yet to the next CF. As Ashutosh mentioned >things tend to be easily forgotten. So I have added it here: >https://commitfest.postgresql.org/13/982/ Thank you for adding this problem to CF. > I have noticed this typo

Re: [HACKERS] Speedup twophase transactions

2017-02-01 Thread Nikhil Sontakke
>> Yeah. Was thinking about this yesterday. How about adding entries in >> TwoPhaseState itself (which become valid later)? Only if it does not >> cause a lot of code churn. > > That's possible as well, yes. PFA a patch which does the above. It re-uses the TwoPhaseState gxact entries to track 2PC

Re: [HACKERS] [BUGS] Bug in Physical Replication Slots (at least 9.5)?

2017-02-01 Thread Michael Paquier
On Thu, Feb 2, 2017 at 1:26 AM, Fujii Masao wrote: > I'm afraid that many WAL segments would start with a continuation record > when there are the workload of short transactions (e.g., by pgbench), and > which would make restart_lsn go behind very much. No? I don't quite understand this argument.

<    1   2