Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Pavan Deolasee
On Thu, Oct 20, 2016 at 11:34 AM, Michael Paquier wrote: > On Thu, Oct 20, 2016 at 2:50 PM, Pavan Deolasee > wrote: > > Actually, if we could add an API which can truncate FSM to the given heap > > block, then the user may not even need to run VACUUM, which could be > costly > > for very large t

Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Michael Paquier
On Thu, Oct 20, 2016 at 2:50 PM, Pavan Deolasee wrote: > Actually, if we could add an API which can truncate FSM to the given heap > block, then the user may not even need to run VACUUM, which could be costly > for very large tables. FreeSpaceMapTruncateRel()? > Also, AFAICS we will need to back

Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Pavan Deolasee
On Thu, Oct 20, 2016 at 10:50 AM, Michael Paquier wrote: > > For VMs a good way would > be to use pg_visibility's pg_truncate_visibility_map(), but only for > 9.6~. Ah ok.. > For FSM there is no real solution, and actually a > pg_truncate_fsm would prove to be useful here. Right, that's wh

Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Michael Paquier
On Thu, Oct 20, 2016 at 2:11 PM, Pavan Deolasee wrote: > 4. Remove FSM and VM files for the affected tables (I don't think if it's > safe to do this on a running server) Definitely not while the server is running... For VMs a good way would be to use pg_visibility's pg_truncate_visibility_map(),

Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Pavan Deolasee
On Wed, Oct 19, 2016 at 6:54 PM, Tom Lane wrote: > > > Can we document an existing procedure for repairing FSM corruption? > (VACUUM, maybe?) I'm afraid it may not be easy to repair the corruption with existing facilities. Most often the corruption will be on the standby and a VACUUM may not ac

Re: [HACKERS] Parallel bitmap heap scan

2016-10-19 Thread Amit Kapila
On Wed, Oct 19, 2016 at 9:23 PM, Dilip Kumar wrote: > On Wed, Oct 19, 2016 at 12:39 PM, Andres Freund wrote: >> Try measuring with something more heavy on bitmap scan time >> itself. E.g. >> SELECT SUM(l_extendedprice) FROM lineitem WHERE (l_shipdate >= >> '1995-01-01'::date) AND (l_shipdate <=

Re: [HACKERS] Avoiding pin scan during btree vacuum

2016-10-19 Thread Amit Kapila
On Thu, Oct 20, 2016 at 4:00 AM, Alvaro Herrera wrote: > Robert Haas wrote: >> On Mon, Jan 4, 2016 at 10:30 AM, Tom Lane wrote: >> >> This seems like a might subtle thing to backpatch. If we really want to >> >> go there, ISTM that the relevant code should stew in an unreleased >> >> branch for a

Re: [HACKERS] Parallel Index Scans

2016-10-19 Thread Peter Geoghegan
On Wed, Oct 19, 2016 at 8:07 PM, Amit Kapila wrote: > I have also checked and found that you are right. In SQL Server, they > are using max degree of parallelism (MAXDOP) parameter which is I > think is common for all the sql statements. It's not just that one that does things this way, for what

Re: [HACKERS] Parallel Index Scans

2016-10-19 Thread Amit Kapila
On Thu, Oct 20, 2016 at 7:39 AM, Peter Geoghegan wrote: > On Mon, Oct 17, 2016 at 8:08 PM, Amit Kapila wrote: >> Create Index With (parallel_workers = 4); >> >> If above syntax looks sensible, then we might need to think what >> should be used for parallel index build. It seems to me that p

Re: [HACKERS] Disable autovacuum guc?

2016-10-19 Thread Michael Paquier
On Thu, Oct 20, 2016 at 11:55 AM, Craig Ringer wrote: > On 20 October 2016 at 09:27, Joshua D. Drake wrote: >> 1. It does not hurt anyone > > I disagree. > > It's important for a number of tests, and can be very useful when > doing diagnostic work. It's also important for some kinds of data > rec

Re: [HACKERS] Query cancel seems to be broken in master since Oct 17

2016-10-19 Thread Noah Misch
On Tue, Oct 18, 2016 at 10:03:39AM -0400, Tom Lane wrote: > Heikki Linnakangas writes: > > On 10/18/2016 04:13 PM, Tom Lane wrote: > >> There's a smoking gun in the postmaster log: > >> 2016-10-18 09:10:34.547 EDT [18502] LOG: wrong key in cancel request for > >> process 18491 > > > Ok, I've re

Re: [HACKERS] Disable autovacuum guc?

2016-10-19 Thread Craig Ringer
On 20 October 2016 at 09:27, Joshua D. Drake wrote: > Hello, > > After all these years, we are still regularly running into people who say, > "performance was bad so we disabled autovacuum". I am not talking about once > in a while, it is often. I would like us to consider removing the autovacuum

Re: [HACKERS] Disable autovacuum guc?

2016-10-19 Thread Josh Berkus
On 10/19/2016 06:27 PM, Joshua D. Drake wrote: > Hello, > > After all these years, we are still regularly running into people who > say, "performance was bad so we disabled autovacuum". I am not talking > about once in a while, it is often. I would like us to consider removing > the autovacuum opt

Re: [HACKERS] Parallel Index Scans

2016-10-19 Thread Peter Geoghegan
On Mon, Oct 17, 2016 at 8:08 PM, Amit Kapila wrote: > Create Index With (parallel_workers = 4); > > If above syntax looks sensible, then we might need to think what > should be used for parallel index build. It seems to me that parallel > tuple sort patch [1] proposed by Peter G. is using ab

[HACKERS] Disable autovacuum guc?

2016-10-19 Thread Joshua D. Drake
Hello, After all these years, we are still regularly running into people who say, "performance was bad so we disabled autovacuum". I am not talking about once in a while, it is often. I would like us to consider removing the autovacuum option. Here are a few reasons: 1. It does not hurt anyo

Re: [HACKERS] Remove vacuum_defer_cleanup_age

2016-10-19 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 08:17:46PM -0400, Robert Haas wrote: > On Wed, Oct 19, 2016 at 12:59 PM, Bruce Momjian wrote: > > Uh, vacuum_defer_cleanup_age sets an upper limit on how long, in terms > > of xids, that a standby query can run before cancel, like > > old_snapshot_threshold, no? > > No, no

Re: [HACKERS] On conflict update & hint bits

2016-10-19 Thread Peter Geoghegan
On Sat, Oct 1, 2016 at 5:15 AM, Peter Geoghegan wrote: > On Fri, Sep 30, 2016 at 5:33 PM, Konstantin Knizhnik > wrote: >> So the question is whether it is correct that ExecOnConflictUpdate tries to >> access and update tuple without holding lock on the buffer? > > You're right -- this is a bug in

Re: [HACKERS] Remove vacuum_defer_cleanup_age

2016-10-19 Thread Robert Haas
On Wed, Oct 19, 2016 at 12:59 PM, Bruce Momjian wrote: > Uh, vacuum_defer_cleanup_age sets an upper limit on how long, in terms > of xids, that a standby query can run before cancel, like > old_snapshot_threshold, no? No, not really. It affects the behavior of the master, not the standby. -- R

Re: [HACKERS] Patch: Implement failover on libpq connect level.

2016-10-19 Thread Robert Haas
On Wed, Oct 19, 2016 at 12:04 PM, Robert Haas wrote: > On Thu, Oct 13, 2016 at 5:53 AM, Victor Wagner wrote: >> On Thu, 13 Oct 2016 12:30:59 +0530 >> Mithun Cy wrote: >>> On Fri, Sep 30, 2016 at 2:14 PM, Victor Wagner >>> wrote: >>> Okay but for me consistency is also important. Since we agree

Re: [HACKERS] Patch: Implement failover on libpq connect level.

2016-10-19 Thread Peter van Hardenberg
On Wed, Oct 19, 2016 at 3:08 PM, Robert Haas wrote: > On Mon, Oct 26, 2015 at 4:25 PM, Peter Eisentraut wrote: > > On 10/14/15 6:41 AM, Victor Wagner wrote: > All in all, I'm still feeling pretty good about trying to support the > same syntax that our JDBC driver already does. It's certainly no

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Alvaro Herrera
Alexander Korotkov wrote: Hi, > Thank you for your proposal. One question about vacuum excites me most. > Imagine another situation: PK column was not updated, but indirect indexed > column was updated. > Thus, for single heap tuple we would have single PK tuple and two indirect > index tuples

Re: [HACKERS] Avoiding pin scan during btree vacuum

2016-10-19 Thread Alvaro Herrera
Robert Haas wrote: > On Mon, Jan 4, 2016 at 10:30 AM, Tom Lane wrote: > >> This seems like a might subtle thing to backpatch. If we really want to > >> go there, ISTM that the relevant code should stew in an unreleased > >> branch for a while, before being backpatched. > > > > I'm definitely -1 on

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Alexander Korotkov
On Wed, Oct 19, 2016 at 12:53 PM, Simon Riggs wrote: > On 18 October 2016 at 23:46, Alexander Korotkov > wrote: > > > Then vacuum removes (0;1) from heap, reference to (0;1) from tbl_pk_idx. > > But how will it remove (1,1) tuple from tbl_val_indirect_idx? Thus, > before > > vacuuming tbl_val_i

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Claudio Freire
On Wed, Oct 19, 2016 at 2:04 PM, Bruce Momjian wrote: >> What we should ask is what is the difference between indirect indexes >> and WARM and to what extent they overlap. >> >> My current understanding is that WARM won't help you if you update >> parts of a JSON document and/or use GIN indexes, b

Re: [HACKERS] Patch: Implement failover on libpq connect level.

2016-10-19 Thread Robert Haas
On Mon, Oct 26, 2015 at 4:25 PM, Peter Eisentraut wrote: > On 10/14/15 6:41 AM, Victor Wagner wrote: >> 1. It is allowed to specify several hosts in the connect string, either >> in URL-style (separated by comma) or in param=value form (several host >> parameters). > > I'm not fond of having URLs

Re: [HACKERS] emergency outage requiring database restart

2016-10-19 Thread Merlin Moncure
On Wed, Oct 19, 2016 at 9:56 AM, Bruce Momjian wrote: > On Wed, Oct 19, 2016 at 08:54:48AM -0500, Merlin Moncure wrote: >> > Yeah. Believe me -- I know the drill. Most or all the damage seemed >> > to be to the system catalogs with at least two critical tables dropped >> > or inaccessible in som

Re: [HACKERS] Remove vacuum_defer_cleanup_age

2016-10-19 Thread Josh Berkus
On 10/19/2016 09:59 AM, Bruce Momjian wrote: > On Wed, Oct 19, 2016 at 09:00:06AM -0400, Robert Haas wrote: >> On Wed, Oct 19, 2016 at 8:47 AM, Bruce Momjian wrote: >>> On Wed, Oct 19, 2016 at 08:33:20AM -0400, Robert Haas wrote: Actually, I think vacuum_defer_cleanup_age is, and always has

Re: [HACKERS] Gather Merge

2016-10-19 Thread Peter Geoghegan
On Tue, Oct 4, 2016 at 11:05 PM, Rushabh Lathia wrote: > Query 4: With GM 7901.480 -> Without GM 9064.776 > Query 5: With GM 53452.126 -> Without GM 55059.511 > Query 9: With GM 52613.132 -> Without GM 98206.793 > Query 15: With GM 68051.058 -> Without GM 68918.378 > Query 17: With GM 129236.07

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-10-19 Thread Peter Geoghegan
On Wed, Oct 19, 2016 at 7:39 AM, Robert Haas wrote: > Gather Merge can't emit a tuple unless it has buffered at least one > tuple from every producer; otherwise, the next tuple it receives from > one of those producers might proceed whichever tuple it chooses to > emit. However, it doesn't need t

Re: [HACKERS] Patch: Implement failover on libpq connect level.

2016-10-19 Thread Thom Brown
On 13 October 2016 at 10:53, Victor Wagner wrote: > On Thu, 13 Oct 2016 12:30:59 +0530 > Mithun Cy wrote: > >> On Fri, Sep 30, 2016 at 2:14 PM, Victor Wagner >> wrote: > >> Okay but for me consistency is also important. Since we agree to >> disagree on some of the comments and others have not ex

Re: [HACKERS] Portable check for unportable macro usage

2016-10-19 Thread Tom Lane
Andres Freund writes: > Hm. I'd be kind of inclined to instead do something akin to > #include > #define system_isupper(c) isupper(c) > #undef isupper Note that that doesn't do what you are probably thinking it does. What is actually happening there, I believe, is that you're forcing a fallbac

Re: [HACKERS] incorrect libpq comment

2016-10-19 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 01:16:28PM -0400, Robert Haas wrote: > Bruce's commit 5d305d86bd917723f09ab4f15c075d90586a210a back in April > of 2014 includes this change: > > /* See PQconnectPoll() for how we use 'int' and not 'pgsocket'. */ > -int sock; /* Unix FD for socket,

Re: [HACKERS] Move pg_largeobject to a different tablespace *without* turning on system_table_mods.

2016-10-19 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 01:25:33PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Wed, Oct 19, 2016 at 09:44:05AM -0700, David G. Johnston wrote: > >> ​I think the theory of having all system tables except LO on SSD storage, > >> and > >> having LO on a less performant device, makes sense.

Re: [HACKERS] Portable check for unportable macro usage

2016-10-19 Thread Andres Freund
On 2016-10-19 12:14:50 -0400, Tom Lane wrote: > A portability hazard that we fight constantly is that the > macros such as isalpha() can't safely be called on "char" values. > Per POSIX, you have to make sure the input is cast to "unsigned char". > That's far too easy to forget, but on most machin

Re: [HACKERS] Move pg_largeobject to a different tablespace *without* turning on system_table_mods.

2016-10-19 Thread Tom Lane
Bruce Momjian writes: > On Wed, Oct 19, 2016 at 09:44:05AM -0700, David G. Johnston wrote: >> ​I think the theory of having all system tables except LO on SSD storage, and >> having LO on a less performant device, makes sense. > OK, so is this a TODO item? According to https://www.postgresql.org

[HACKERS] incorrect libpq comment

2016-10-19 Thread Robert Haas
Bruce's commit 5d305d86bd917723f09ab4f15c075d90586a210a back in April of 2014 includes this change: /* See PQconnectPoll() for how we use 'int' and not 'pgsocket'. */ -int sock; /* Unix FD for socket, -1 if not connected */ +pgsocketsock; /* FD for sock

Re: [HACKERS] Move pg_largeobject to a different tablespace *without* turning on system_table_mods.

2016-10-19 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 09:44:05AM -0700, David G. Johnston wrote: > On Wed, Oct 19, 2016 at 9:29 AM, Bruce Momjian wrote: > > On Tue, Oct 18, 2016 at 04:51:54PM +0200, Andreas Joseph Krogh wrote: > >     > 2. Being able to move pg_largeobject to a different tablespace > >     >    *w

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-10-19 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 11:47:25AM -0400, Robert Haas wrote: > It seems to me that the only way to really make this feature robust is > to have a background worker as part of the equation. The background > worker launches at startup and looks around for local state that tells > it whether there ar

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 06:58:05PM +0200, Simon Riggs wrote: > >> I agree. Also, I think the recheck mechanism will have to be something like > >> what I wrote for WARM i.e. only checking for index quals won't be enough > >> and we > >> would actually need to verify that the heap tuple satisfies t

Re: [HACKERS] Remove vacuum_defer_cleanup_age

2016-10-19 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 09:00:06AM -0400, Robert Haas wrote: > On Wed, Oct 19, 2016 at 8:47 AM, Bruce Momjian wrote: > > On Wed, Oct 19, 2016 at 08:33:20AM -0400, Robert Haas wrote: > >> On Tue, Oct 18, 2016 at 4:33 PM, Josh Berkus wrote: > >> > Based on that argument, we would never be able to r

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Simon Riggs
On 19 October 2016 at 18:40, Bruce Momjian wrote: > On Wed, Oct 19, 2016 at 07:23:28PM +0530, Pavan Deolasee wrote: >> AFAICS, even without considering VACUUM, indirect indexes would be always >> used with recheck. >> As long as they don't contain visibility information. When indirect

Re: [HACKERS] Move pg_largeobject to a different tablespace *without* turning on system_table_mods.

2016-10-19 Thread Andreas Joseph Krogh
På onsdag 19. oktober 2016 kl. 18:44:24, skrev Tom Lane mailto:t...@sss.pgh.pa.us>>: Andreas Joseph Krogh writes: > På onsdag 19. oktober 2016 kl. 18:29:31, skrev Bruce Momjian   I think an open question is why you would not want to move the other >  system tables at the same time you move pg_l

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Claudio Freire
On Wed, Oct 19, 2016 at 10:21 AM, Simon Riggs wrote: >> Simon objected that putting the PK >> into the index tuple would disable HOT, but I don't think that's a >> valid objection. > > Just to be clear, that's not what I objected to. Claudio appeared to > be suggesting that an indirect index is th

Re: [HACKERS] [COMMITTERS] packing/alignment annotation for ItemPointerData redux

2016-10-19 Thread Tom Lane
Greg Stark writes: > Sorry -- with the obvious error fixed: You didn't show -E output from this version, but the other one had > __attribute__((packed)) > __attribute__((aligned(2))) so it appears that clang 4.0 does accept these attributes but then produces the warning anyway. I suggest fil

Re: [HACKERS] Move pg_largeobject to a different tablespace *without* turning on system_table_mods.

2016-10-19 Thread Andreas Joseph Krogh
På onsdag 19. oktober 2016 kl. 18:42:11, skrev Bruce Momjian mailto:br...@momjian.us>>: On Wed, Oct 19, 2016 at 06:33:55PM +0200, Andreas Joseph Krogh wrote: > På onsdag 19. oktober 2016 kl. 18:29:31, skrev Bruce Momjian >: > >     On Tue, Oct 18, 2016 at 04:51:54PM +0200, Andreas Joseph Krogh

Re: [HACKERS] Move pg_largeobject to a different tablespace *without* turning on system_table_mods.

2016-10-19 Thread Tom Lane
Andreas Joseph Krogh writes: > På onsdag 19. oktober 2016 kl. 18:29:31, skrev Bruce Momjian > I think an open question is why you would not want to move the other > system tables at the same time you move pg_largeobject. > Are you saying that if I move all system-tables to the tablespace I mo

Re: [HACKERS] Move pg_largeobject to a different tablespace *without* turning on system_table_mods.

2016-10-19 Thread David G. Johnston
On Wed, Oct 19, 2016 at 9:29 AM, Bruce Momjian wrote: > On Tue, Oct 18, 2016 at 04:51:54PM +0200, Andreas Joseph Krogh wrote: > > > 2. Being able to move pg_largeobject to a different tablespace > > >*without* turning on system_table_mods. This is important for > > >people sto

Re: [HACKERS] Move pg_largeobject to a different tablespace *without* turning on system_table_mods.

2016-10-19 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 06:33:55PM +0200, Andreas Joseph Krogh wrote: > På onsdag 19. oktober 2016 kl. 18:29:31, skrev Bruce Momjian >: > > On Tue, Oct 18, 2016 at 04:51:54PM +0200, Andreas Joseph Krogh wrote: > >     > 2. Being able to move pg_largeobject to a different tablespace >

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 07:23:28PM +0530, Pavan Deolasee wrote: > AFAICS, even without considering VACUUM, indirect indexes would be always > used with recheck. > As long as they don't contain visibility information.  When indirect > indexed column was updated, indirect index would

Re: [HACKERS] [COMMITTERS] packing/alignment annotation for ItemPointerData redux

2016-10-19 Thread Greg Stark
Sorry -- with the obvious error fixed: $ /usr/bin/clang-4.0 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -Wall -c clang-bug.c clang-bug.c:55:9: w

Re: [HACKERS] [COMMITTERS] packing/alignment annotation for ItemPointerData redux

2016-10-19 Thread Greg Stark
Ah. Here we go: $ /usr/bin/clang-4.0 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -Wall -c clang-bug.c clang-bug.c:54:9: error: use of undeclared

Re: [HACKERS] Move pg_largeobject to a different tablespace *without* turning on system_table_mods.

2016-10-19 Thread Andreas Joseph Krogh
På onsdag 19. oktober 2016 kl. 18:29:31, skrev Bruce Momjian mailto:br...@momjian.us>>: On Tue, Oct 18, 2016 at 04:51:54PM +0200, Andreas Joseph Krogh wrote: >     > 2. Being able to move pg_largeobject to a different tablespace >     >    *without* turning on system_table_mods. This is important

Re: [HACKERS] [COMMITTERS] packing/alignment annotation for ItemPointerData redux

2016-10-19 Thread Greg Stark
On Wed, Oct 19, 2016 at 5:20 PM, Tom Lane wrote: > Don't know how that version number compares to "3.8.0". Argh. Just to further confuse matters apparently the warnings are from clang 4.0. I had been testing with 3.8.0 earlier but updated at some point. And I'm not being able to reproduce them w

Re: [HACKERS] Move pg_largeobject to a different tablespace *without* turning on system_table_mods.

2016-10-19 Thread Bruce Momjian
On Tue, Oct 18, 2016 at 04:51:54PM +0200, Andreas Joseph Krogh wrote: > > 2. Being able to move pg_largeobject to a different tablespace > >    *without* turning on system_table_mods. This is important for > >    people storing LOTS of large-objects on separate > >    disks (non-SSD

Re: [HACKERS] [COMMITTERS] packing/alignment annotation for ItemPointerData redux

2016-10-19 Thread Tom Lane
[ moved to -hackers ] Greg Stark writes: > Back in 2001 a hack to add __attribute__((packed)) to ItemPtr was > added with a comment "Appropriate whack upside the head for ARM" > (dcbbdb1b3ee). I don't know if this is still a factor in 2016 or not > but it has already resulted in some collateral d

Re: [HACKERS] Question about behavior of snapshot too old feature

2016-10-19 Thread Kevin Grittner
On Wed, Oct 19, 2016 at 11:11 AM, Bruce Momjian wrote: > On Wed, Oct 19, 2016 at 11:08:28AM -0500, Kevin Grittner wrote: >> On Wed, Oct 19, 2016 at 10:04 AM, Bruce Momjian wrote: >> >>> Slide 10 of this presentation has an example showing >>> old_snapshot_threshold set to '1min': >>> >>>

[HACKERS] packing/alignment annotation for ItemPointerData redux

2016-10-19 Thread Greg Stark
[resending to -hackers sorry] Back in 2001 a hack to add __attribute__((packed)) to ItemPtr was added with a comment "Appropriate whack upside the head for ARM" (dcbbdb1b3ee). I don't know if this is still a factor in 2016 or not but it has already resulted in some collateral damage in 2015 when s

[HACKERS] Portable check for unportable macro usage

2016-10-19 Thread Tom Lane
A portability hazard that we fight constantly is that the macros such as isalpha() can't safely be called on "char" values. Per POSIX, you have to make sure the input is cast to "unsigned char". That's far too easy to forget, but on most machines you won't get any warning about it. We have a coup

Re: [HACKERS] Question about behavior of snapshot too old feature

2016-10-19 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 11:08:28AM -0500, Kevin Grittner wrote: > On Wed, Oct 19, 2016 at 10:04 AM, Bruce Momjian wrote: > > > Slide 10 of this presentation has an example showing > > old_snapshot_threshold set to '1min': > > > > http://momjian.us/main/writings/pgsql/features.pdf > > If

Re: [HACKERS] Question about behavior of snapshot too old feature

2016-10-19 Thread Kevin Grittner
On Wed, Oct 19, 2016 at 10:04 AM, Bruce Momjian wrote: > Slide 10 of this presentation has an example showing > old_snapshot_threshold set to '1min': > > http://momjian.us/main/writings/pgsql/features.pdf If the presentation is intending to show reasonable values for production use, that

Re: [HACKERS] Patch: Implement failover on libpq connect level.

2016-10-19 Thread Robert Haas
On Thu, Oct 13, 2016 at 5:53 AM, Victor Wagner wrote: > On Thu, 13 Oct 2016 12:30:59 +0530 > Mithun Cy wrote: >> On Fri, Sep 30, 2016 at 2:14 PM, Victor Wagner >> wrote: >> Okay but for me consistency is also important. Since we agree to >> disagree on some of the comments and others have not ex

Re: [HACKERS] Parallel bitmap heap scan

2016-10-19 Thread Dilip Kumar
On Wed, Oct 19, 2016 at 12:39 PM, Andres Freund wrote: > Try measuring with something more heavy on bitmap scan time > itself. E.g. > SELECT SUM(l_extendedprice) FROM lineitem WHERE (l_shipdate >= > '1995-01-01'::date) AND (l_shipdate <= '1996-12-31'::date); > or similar. The tpch queries don't

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-10-19 Thread Robert Haas
On Thu, Oct 13, 2016 at 7:27 AM, Amit Langote wrote: > However, when I briefly read the description in "Transaction Management in > the R* Distributed Database Management System (C. Mohan et al)" [2], it > seems that what Ashutosh is saying might be a correct way to proceed after > all: I think A

Re: [HACKERS] Question about behavior of snapshot too old feature

2016-10-19 Thread Bruce Momjian
On Mon, Oct 17, 2016 at 08:04:43AM -0500, Kevin Grittner wrote: > > We have regression test for this feature but it sets > > old_snapshot_threshold = 0, I doubt about we can test it properly. > > Am I missing something? > > This is a hard feature to test properly, and certainly hard to test > with

Re: [HACKERS] emergency outage requiring database restart

2016-10-19 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 08:54:48AM -0500, Merlin Moncure wrote: > > Yeah. Believe me -- I know the drill. Most or all the damage seemed > > to be to the system catalogs with at least two critical tables dropped > > or inaccessible in some fashion. A lot of the OIDs seemed to be > > pointing at t

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-10-19 Thread Robert Haas
On Mon, Oct 17, 2016 at 8:36 AM, Amit Kapila wrote: >> This project of mine is about parallelizing tuplesort.c, which isn't >> really what you want for parallel query -- you shouldn't try to scope >> the problem as "make the sort more scalable using parallelism" there. >> Rather, you want to scope

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Robert Haas
On Wed, Oct 19, 2016 at 9:21 AM, Simon Riggs wrote: > The main problem IMV is GIN indexes. It's relatively easy to discuss > variable length PKs with btrees, but the GIN format is designed around > use of 6byte values, so expanding beyond that would require > significant redesign/reimplementation.

Re: [HACKERS] Draft for next update release (scheduled for 27th Oct)

2016-10-19 Thread Umair Shahid
On Wed, Oct 19, 2016 at 6:34 PM, Tom Lane wrote: > Umair Shahid writes: > > We are in the process of writing up draft notes for the upcoming update > > releases. To initiate the process, we need to list out the major fixes > > going into the release. Need help from -hackers to generate that list

Re: [HACKERS] emergency outage requiring database restart

2016-10-19 Thread Merlin Moncure
On Tue, Oct 18, 2016 at 8:45 AM, Merlin Moncure wrote: > On Mon, Oct 17, 2016 at 2:04 PM, Alvaro Herrera > wrote: >> Merlin Moncure wrote: >> >>> castaging=# CREATE OR REPLACE VIEW vw_ApartmentSample AS >>> castaging-# SELECT ... >>> ERROR: 42809: "pg_cast_oid_index" is an index >>> LINE 11:

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Pavan Deolasee
On Wed, Oct 19, 2016 at 7:19 PM, Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > On Wed, Oct 19, 2016 at 3:52 PM, Robert Haas > wrote: > >> The VACUUM problems seem fairly serious. It's true that these indexes >> will be less subject to bloat, because they only need updating when >> the

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Alexander Korotkov
On Wed, Oct 19, 2016 at 3:52 PM, Robert Haas wrote: > The VACUUM problems seem fairly serious. It's true that these indexes > will be less subject to bloat, because they only need updating when > the PK or the indexed columns change, not when other indexed columns > change. On the other hand, t

Re: [HACKERS] Draft for next update release (scheduled for 27th Oct)

2016-10-19 Thread Tom Lane
Umair Shahid writes: > We are in the process of writing up draft notes for the upcoming update > releases. To initiate the process, we need to list out the major fixes > going into the release. Need help from -hackers to generate that list. Why don't you just wait for the release notes to be writ

Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Pavan Deolasee
On Wed, Oct 19, 2016 at 6:44 PM, Heikki Linnakangas wrote: > On 10/19/2016 02:32 PM, Heikki Linnakangas wrote: > >> >>> >> Oh, forgot that this needs to be backported, of course. Will do that >> shortly... >> > > Done. > Thanks! > > This didn't include anything to cope with an already-corrupt

Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Tom Lane
Heikki Linnakangas writes: > This didn't include anything to cope with an already-corrupt FSM, BTW. > Do we still want to try something for that? I think it's good enough if > we prevent the FSM corruption from happening, but not sure what the > consensus on that might be.. Can we document an

Re: [HACKERS] Using pg_ctl promote -w in TAP tests

2016-10-19 Thread Peter Eisentraut
On 10/10/16 8:37 AM, Michael Paquier wrote: > Now that we have support for the wait mode of pg_ctl promote, I think > that it would be a good idea to switch to it in the TAP tests. This > allows avoiding extra logic with poll_query_until() to be sure that a > promoted standby is ready for read-writ

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Simon Riggs
On 19 October 2016 at 14:52, Robert Haas wrote: > On Tue, Oct 18, 2016 at 2:28 PM, Alvaro Herrera > wrote: >> I propose we introduce the concept of "indirect indexes". I have a toy >> implementation and before I go further with it, I'd like this assembly's >> input on the general direction. >> >

Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Heikki Linnakangas
On 10/19/2016 02:32 PM, Heikki Linnakangas wrote: On 10/19/2016 02:29 PM, Heikki Linnakangas wrote: On 10/19/2016 01:07 PM, Pavan Deolasee wrote: Anyways, we seem good to go with the patch. Ok, committed. Thanks for the analysis! Oh, forgot that this needs to be backported, of course. Will

Re: [HACKERS] Remove vacuum_defer_cleanup_age

2016-10-19 Thread Robert Haas
On Wed, Oct 19, 2016 at 8:47 AM, Bruce Momjian wrote: > On Wed, Oct 19, 2016 at 08:33:20AM -0400, Robert Haas wrote: >> On Tue, Oct 18, 2016 at 4:33 PM, Josh Berkus wrote: >> > Based on that argument, we would never be able to remove any >> > configuration parameter ever. >> >> Well... no. Based

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Robert Haas
On Tue, Oct 18, 2016 at 2:28 PM, Alvaro Herrera wrote: > I propose we introduce the concept of "indirect indexes". I have a toy > implementation and before I go further with it, I'd like this assembly's > input on the general direction. > > Indirect indexes are similar to regular indexes, except

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-10-19 Thread Tomas Vondra
On 10/19/2016 12:27 AM, Petr Jelinek wrote: > On 18/10/16 22:25, Robert Haas wrote: >> On Wed, Oct 5, 2016 at 12:22 AM, Tomas Vondra >> wrote: >>> attached is v3 of the patches, with a few minor fixes in Slab, and much >>> larger fixes in GenSlab. >>> >>> Slab (minor fixes) >>> -

Re: [HACKERS] Long options for pg_ctl waiting

2016-10-19 Thread Peter Eisentraut
On 9/7/16 5:08 PM, Vik Fearing wrote: > On 09/07/2016 10:41 PM, Alvaro Herrera wrote: >> Gavin Flower wrote: >> >>> possibly '--nosync' (& any similar) should have a '--no-sync' variation >>> added, with the '--nosync' variation documented as depreciated? >> >> I agree -- I would go as far as just

Re: [HACKERS] Long options for pg_ctl waiting

2016-10-19 Thread Peter Eisentraut
On 9/3/16 4:57 PM, Vik Fearing wrote: > One thing that has been irking me ever since I came to PostgreSQL is the > fact that pg_ctl -w (and -W) don't have longhand equivalents. I like to > use the long version in scripts and such as extra documentation, and > I've never been able to with these. W

Re: [HACKERS] Remove vacuum_defer_cleanup_age

2016-10-19 Thread Bruce Momjian
On Wed, Oct 19, 2016 at 08:33:20AM -0400, Robert Haas wrote: > On Tue, Oct 18, 2016 at 4:33 PM, Josh Berkus wrote: > > Based on that argument, we would never be able to remove any > > configuration parameter ever. > > Well... no. Based on that argument, we should only remove > configuration para

Re: [HACKERS] Draft for next update release (scheduled for 27th Oct)

2016-10-19 Thread Michael Paquier
On Wed, Oct 19, 2016 at 9:35 PM, Heikki Linnakangas wrote: > Off the top of my head, these two recent commits were quite significant: > > 1c02ee314bc17ba0ff226c33a64935d02ff1208a (FSM corruption leading to errors) > a5f0bd77a2fab60a52dc335a63efc21abc806aa7 (use-after-free around DISTINCT > transit

Re: [HACKERS] Draft for next update release (scheduled for 27th Oct)

2016-10-19 Thread Heikki Linnakangas
On 10/19/2016 12:06 PM, Umair Shahid wrote: Hi, We are in the process of writing up draft notes for the upcoming update releases. To initiate the process, we need to list out the major fixes going into the release. Need help from -hackers to generate that list. I think you need to read through

Re: [HACKERS] Remove vacuum_defer_cleanup_age

2016-10-19 Thread Robert Haas
On Tue, Oct 18, 2016 at 4:33 PM, Josh Berkus wrote: > Based on that argument, we would never be able to remove any > configuration parameter ever. Well... no. Based on that argument, we should only remove configuration parameters if we're fairly certain that they are not useful any more, which w

Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Michael Paquier
On Wed, Oct 19, 2016 at 8:29 PM, Heikki Linnakangas wrote: > On 10/19/2016 01:07 PM, Pavan Deolasee wrote: >> >> Anyways, we seem good to go with the patch. > > Ok, committed. Thanks for the analysis! Thanks! I am surprised that you kept the TAP test at the end. -- Michael -- Sent via pgsql-h

Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Heikki Linnakangas
On 10/19/2016 02:29 PM, Heikki Linnakangas wrote: On 10/19/2016 01:07 PM, Pavan Deolasee wrote: Anyways, we seem good to go with the patch. Ok, committed. Thanks for the analysis! Oh, forgot that this needs to be backported, of course. Will do that shortly... - Heikki -- Sent via pgsql

Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Heikki Linnakangas
On 10/19/2016 01:07 PM, Pavan Deolasee wrote: Anyways, we seem good to go with the patch. Ok, committed. Thanks for the analysis! - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hac

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-19 Thread Christoph Berg
Re: Bruce Momjian 2016-10-19 <20161018220909.ga11...@momjian.us> > > There's actually another instance of "rename so people shoot their > > feet less often" here: pg_resetxlog, which is a user-facing tool. > > Folks on #postgresql have repeatedly been joking that it should rather > > be named pg_ea

Re: [HACKERS] LLVM Address Sanitizer (ASAN) and valgrind support

2016-10-19 Thread Greg Stark
On Sat, Feb 6, 2016 at 4:52 AM, Noah Misch wrote: > aset.c relies on the fact that VALGRIND_MEMPOOL_ALLOC() has an implicit > VALGRIND_MAKE_MEM_UNDEFINED() and VALGRIND_MEMPOOL_FREE() has an implicit > VALGRIND_MAKE_MEM_NOACCESS(). #define those two accordingly. If ASAN has no Actually this is

Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Pavan Deolasee
On Wed, Oct 19, 2016 at 2:37 PM, Heikki Linnakangas wrote: > >> > Actually, this is still not 100% safe. Flushing the WAL before modifying > the FSM page is not enough. We also need to WAL-log a full-page image of > the FSM page, otherwise we are still vulnerable to the torn page problem. > > I c

Re: [HACKERS] Indirect indexes

2016-10-19 Thread Simon Riggs
On 18 October 2016 at 23:46, Alexander Korotkov wrote: > Then vacuum removes (0;1) from heap, reference to (0;1) from tbl_pk_idx. > But how will it remove (1,1) tuple from tbl_val_indirect_idx? Thus, before > vacuuming tbl_val_indirect_idx we should know not only values of id which > are being r

Re: [HACKERS] FSM corruption leading to errors

2016-10-19 Thread Heikki Linnakangas
On 10/18/2016 07:01 AM, Pavan Deolasee wrote: On Mon, Oct 17, 2016 at 4:34 PM, Heikki Linnakangas wrote: visibilitymap_truncate is actually also wrong, in a different way. The truncation WAL record is written only after the VM (and FSM) are truncated. But visibilitymap_truncate() has already m

[HACKERS] Draft for next update release (scheduled for 27th Oct)

2016-10-19 Thread Umair Shahid
Hi, We are in the process of writing up draft notes for the upcoming update releases. To initiate the process, we need to list out the major fixes going into the release. Need help from -hackers to generate that list. Thanks! - Umair

Re: [HACKERS] minor issue: \c without parameter disconnect current user

2016-10-19 Thread Pavel Stehule
2016-10-18 20:11 GMT+02:00 Alvaro Herrera : > Robert Haas wrote: > > On Mon, Oct 17, 2016 at 12:49 AM, Pavel Stehule > wrote: > > > I expect so \c without parameters has only informational character. > But \c > > > reset user. > > > > Yeah, I use that feature all the time. > > And so do the regre

Re: [HACKERS] Parallel bitmap heap scan

2016-10-19 Thread Andres Freund
On 2016-10-19 09:43:10 +0530, Dilip Kumar wrote: > On Tue, Oct 18, 2016 at 1:42 AM, Robert Haas wrote: > > But what's the impact on performance? Presumably parallel bitmap heap > > scan was already slower than the non-parallel version, and that commit > > presumably widens the gap. Seems like so