Re: [HACKERS] Enabling parallelism for queries coming from SQL or other PL functions

2017-02-25 Thread Amit Kapila
On Sat, Feb 25, 2017 at 9:47 PM, Dilip Kumar <dilipbal...@gmail.com> wrote: > On Sat, Feb 25, 2017 at 5:12 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Sure, but that should only happen if the function is *not* declared as >> parallel safe (aka in paralle

Re: [HACKERS] Proposal : Parallel Merge Join

2017-02-25 Thread Amit Kapila
On Sat, Feb 25, 2017 at 3:22 PM, Dilip Kumar <dilipbal...@gmail.com> wrote: > On Sat, Feb 25, 2017 at 11:29 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> It seems you have forgotten to change in the below check: >> >> + if (innerpath != NULL

Re: [HACKERS] Enabling parallelism for queries coming from SQL or other PL functions

2017-02-25 Thread Amit Kapila
On Fri, Feb 24, 2017 at 11:30 AM, Dilip Kumar <dilipbal...@gmail.com> wrote: > On Fri, Feb 24, 2017 at 10:06 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> We have a below check in standard_planner() (!IsParallelWorker()) >> which should prohibit generating

Re: [HACKERS] Proposal : Parallel Merge Join

2017-02-24 Thread Amit Kapila
On Fri, Feb 24, 2017 at 4:49 PM, Dilip Kumar <dilipbal...@gmail.com> wrote: > On Fri, Feb 24, 2017 at 3:04 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> What advantage do you see for considering such a path when the cost of >> innerpath is more than cheapest_tota

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-02-24 Thread Amit Kapila
s discussion, AFAICT. I > think that both you and Simon mean "recyclable" pages. > Yes, I think so and I think that is just confusion about terminology. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@

Re: [HACKERS] Proposal : Parallel Merge Join

2017-02-24 Thread Amit Kapila
On Fri, Feb 24, 2017 at 3:42 PM, Dilip Kumar <dilipbal...@gmail.com> wrote: > On Fri, Feb 24, 2017 at 3:04 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > > Thanks for the comments. > >> What advantage do you see for considering such a path when the cos

Re: [HACKERS] Proposal : Parallel Merge Join

2017-02-24 Thread Amit Kapila
On Tue, Feb 14, 2017 at 5:22 PM, Dilip Kumar <dilipbal...@gmail.com> wrote: > On Tue, Feb 14, 2017 at 12:25 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > Apart from this, there was one small problem in an earlier version > which I have corrected in this. > > + /*

Re: [HACKERS] Enabling parallelism for queries coming from SQL or other PL functions

2017-02-23 Thread Amit Kapila
and shutdown there is no > change in the database state. > +1. I also think you are right that there should not be a problem in such a case. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] Enabling parallelism for queries coming from SQL or other PL functions

2017-02-23 Thread Amit Kapila
_workers_per_gather > 0 && !IsParallelWorker() && !IsolationIsSerializable()) { /* all the cheap tests pass, so scan the query tree */ glob->maxParallelHazard = max_parallel_hazard(parse); glob->parallelModeOK = (glob->maxParallelHazard != PROPARALLEL_UNSAFE); } -- Wit

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-02-23 Thread Amit Kapila
On Thu, Feb 23, 2017 at 5:15 PM, Masahiko Sawada <sawada.m...@gmail.com> wrote: > On Wed, Feb 22, 2017 at 12:01 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > >> I understand that there could be some delay in reclaiming dead pages >> but do you think it is such

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-02-21 Thread Amit Kapila
On Tue, Feb 21, 2017 at 1:09 AM, Simon Riggs <si...@2ndquadrant.com> wrote: > On 20 February 2017 at 10:27, Amit Kapila <amit.kapil...@gmail.com> wrote: >> On Mon, Feb 20, 2017 at 3:01 PM, Simon Riggs <si...@2ndquadrant.com> wrote: >>> On 20 February 20

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-02-20 Thread Amit Kapila
ation in shared memory based on which we can decide how much percentage of buckets are allocated in current table half? I think we might be able to construct this information after crash recovery as well. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-02-20 Thread Amit Kapila
On Mon, Feb 20, 2017 at 3:01 PM, Simon Riggs <si...@2ndquadrant.com> wrote: > On 20 February 2017 at 09:15, Amit Kapila <amit.kapil...@gmail.com> wrote: >> On Mon, Feb 20, 2017 at 7:26 AM, Masahiko Sawada <sawada.m...@gmail.com> >> wrote: >>> On Fri, Feb

Re: [HACKERS] Gather Merge

2017-02-20 Thread Amit Kapila
er adding Mark/Restore support will make it any faster than just adding Materialize on top of Gather Merge. So, it seems better not to go there unless we see some use of it. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-02-20 Thread Amit Kapila
erform a complete pass to reclaim dead/deleted pages unless we know someway that there are many such pages. Also, I think we do reclaim the complete page while allocating a new page in btree. > The half-dead pages are never cleaned up if the ratio of pages > containing garbage is always lower

Re: [HACKERS] Instability in select_parallel regression test

2017-02-19 Thread Amit Kapila
On Sun, Feb 19, 2017 at 8:32 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Sun, Feb 19, 2017 at 6:50 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> To close the remaining gap, don't you think we can check slot->in_use >> flag when generation numbe

Re: [HACKERS] Parallel bitmap heap scan

2017-02-19 Thread Amit Kapila
ther worker have finished the work. For Parallel Seq Scan, we do wait for parallel workers to finish before freeing the shared memory. Why the case is different here? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Gather Merge

2017-02-19 Thread Amit Kapila
On Sun, Feb 19, 2017 at 2:22 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Sat, Feb 18, 2017 at 6:43 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> I think there is a value in supporting mark/restore position for any >> node which produces sorted results,

Re: [HACKERS] Instability in select_parallel regression test

2017-02-19 Thread Amit Kapila
ing the notification to the starting process, that closes > this hole. Almost. > To close the remaining gap, don't you think we can check slot->in_use flag when generation number for handle and slot are same. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Gather Merge

2017-02-18 Thread Amit Kapila
On Fri, Feb 17, 2017 at 6:27 PM, Rushabh Lathia <rushabh.lat...@gmail.com> wrote: > On Fri, Feb 17, 2017 at 4:47 PM, Amit Kapila <amit.kapil...@gmail.com> > wrote: >> >> Are you suggesting that commit 0c2070ce has helped to improve >> performance if so, I don't

Re: [HACKERS] Parallel Index-only scan

2017-02-17 Thread Amit Kapila
On Fri, Feb 17, 2017 at 10:35 AM, Rafia Sabih <rafia.sa...@enterprisedb.com> wrote: > > > On Thu, Feb 16, 2017 at 9:25 PM, Amit Kapila <amit.kapil...@gmail.com> > wrote: >> >> >> 4. >> ExecReScanIndexOnlyScan(IndexOnlyScanState *node) >>

Re: [HACKERS] Instability in select_parallel regression test

2017-02-17 Thread Amit Kapila
On Fri, Feb 17, 2017 at 9:15 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: > Amit Kapila <amit.kapil...@gmail.com> writes: >> On Fri, Feb 17, 2017 at 11:22 AM, Tom Lane <t...@sss.pgh.pa.us> wrote: >>> In short, it looks to me like ExecShutdownGatherWorkers doesn't ac

Re: [HACKERS] Instability in select_parallel regression test

2017-02-17 Thread Amit Kapila
nd from tenk2 where thousand > 100); > 2017-02-17 00:28:33.254 EST [18934] LOG: LaunchParallelWorkers: failed to > register 3 of 4 workers > 2017-02-17 00:28:33.254 EST [18934] STATEMENT: select count(*) from tenk1 > where (two, four) not in > (select hundred, th

Re: [HACKERS] Gather Merge

2017-02-17 Thread Amit Kapila
l" to demonstrate the new kind of plan being created; It has added both explain statement test and a test to exercise parallel index scan code. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2017-02-16 Thread Amit Kapila
On Thu, Feb 16, 2017 at 8:16 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Thu, Feb 16, 2017 at 7:15 AM, Robert Haas <robertmh...@gmail.com> wrote: > > Attached are refactoring patches. WAL patch needs some changes based > on above comments, so will post it later.

Re: [HACKERS] Parallel Index-only scan

2017-02-16 Thread Amit Kapila
+ /* + * if we are here to just update the scan keys, then don't reset parallel + * scan + */ + if (node->ioss_NumRuntimeKeys != 0 && !node->ioss_RuntimeKeysReady) + reset_parallel_scan = false; .. } I think here you can update the comment to indicate that for detailed reason refer Exe

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2017-02-16 Thread Amit Kapila
On Thu, Feb 16, 2017 at 7:15 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Feb 13, 2017 at 10:22 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> As discussed, attached are refactoring patches and a patch to enable >> WAL for the hash index on top of

Re: [HACKERS] Possible issue with expanded object infrastructure on Postgres 9.6.1

2017-02-16 Thread Amit Kapila
ls is extremely limited so I don't know > how much help I can be, but we'd like to see this resolved beyond the quick > fix, or at least understand why it happened. > Thanks Justin for the confirmation. Is it possible for you to get a standalone test case with which we can reproduce the problem?

Re: [HACKERS] Parallel Index Scans

2017-02-15 Thread Amit Kapila
On Thu, Feb 16, 2017 at 12:27 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Feb 15, 2017 at 1:39 PM, Robert Haas <robertmh...@gmail.com> wrote: >> On Wed, Feb 15, 2017 at 7:11 AM, Amit Kapila <amit.kapil...@gmail.com> >> wrote:> >>>

Re: [HACKERS] Proposal: GetOldestXminExtend for ignoring arbitrary vacuum flags

2017-02-15 Thread Amit Kapila
On Wed, Feb 15, 2017 at 12:03 PM, Seki, Eiji <seki.e...@jp.fujitsu.com> wrote: > Amit Kapila wrote: >> How will you decide just based on oldest xmin whether the tuple is visible >> or not? How will you take decisions about tuples which have xmax set? > > In our use

Re: [HACKERS] Parallel Index Scans

2017-02-15 Thread Amit Kapila
t; I have rebased the optimizer/executor support related patch. [1] - https://www.postgresql.org/message-id/CAA4eK1%2BTnM4pXQbvn7OXqam%2Bk_HZqb0ROZUMxOiL6DWJYCyYow%40mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com parallel_index_scan_v10.patch Description

Re: [HACKERS] parallelize queries containing subplans

2017-02-14 Thread Amit Kapila
On Wed, Feb 15, 2017 at 4:38 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Feb 14, 2017 at 4:24 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> On further evaluation, it seems this patch has one big problem which >> is that it will allow forming

Re: [HACKERS] parallelize queries containing subplans

2017-02-14 Thread Amit Kapila
On Wed, Feb 15, 2017 at 4:46 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Feb 13, 2017 at 11:07 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> I have removed the check of AlternativeSubPlan so that it can be >> handled by expression_tree_walker. > .

Re: [HACKERS] WIP: Covering + unique indexes.

2017-02-14 Thread Amit Kapila
atch. [1] - https://www.postgresql.org/message-id/3863bca17face15c6acd507e0173a6dc%40xs4all.nl -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal: GetOldestXminExtend for ignoring arbitrary vacuum flags

2017-02-14 Thread Amit Kapila
ecide just based on oldest xmin whether the tuple is visible or not? How will you take decisions about tuples which have xmax set? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] parallelize queries containing subplans

2017-02-14 Thread Amit Kapila
On Tue, Jan 3, 2017 at 4:19 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Wed, Dec 28, 2016 at 11:47 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> >> Now, we can further extend this to parallelize queries containing >> correlated subplans like belo

Re: [HACKERS] Small improvement to parallel query docs

2017-02-14 Thread Amit Kapila
lanning process and will choose how to > +perform the aggregation accordingly. This part of the sentence sounds unclear. It doesn't clearly indicate that planner won't choose a parallel plan in such cases. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -

Re: [HACKERS] Proposal : Parallel Merge Join

2017-02-13 Thread Amit Kapila
On Wed, Jan 4, 2017 at 9:27 PM, Dilip Kumar <dilipbal...@gmail.com> wrote: > On Wed, Jan 4, 2017 at 12:02 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> 3. >> + /* >> + * See comments in try_partial_nestloop_path(). >> + */ >> >>

Re: [HACKERS] parallelize queries containing subplans

2017-02-13 Thread Amit Kapila
On Thu, Feb 2, 2017 at 9:23 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Wed, Feb 1, 2017 at 10:04 PM, Robert Haas <robertmh...@gmail.com> wrote: >> On Tue, Jan 31, 2017 at 6:01 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> Moved this patch to

Re: [HACKERS] Parallel Index Scans

2017-02-13 Thread Amit Kapila
On Mon, Feb 13, 2017 at 5:47 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Sat, Feb 11, 2017 at 6:35 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>>>> Why can't we rely on _bt_walk_left? >>>> The reason is mentioned in comments, but let me tr

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2017-02-11 Thread Amit Kapila
. I find all this highly > confusing. Have I missed some further subtlety that applies with > PARALLEL RESTRICTED? > The basic idea of parallel restricted is that any expression will be considered parallel restricted if that can't be executed in the worker, for example, a reference to i

Re: [HACKERS] Parallel Index Scans

2017-02-11 Thread Amit Kapila
On Fri, Feb 10, 2017 at 11:27 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Feb 1, 2017 at 8:20 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> The hunk in indexam.c looks like a leftover that can be omitted. >> >> It is not a leftover hunk. E

Re: [HACKERS] Parallel Index Scans

2017-02-11 Thread Amit Kapila
On Sat, Feb 11, 2017 at 9:41 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Fri, Feb 10, 2017 at 11:22 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> Why can't we rely on _bt_walk_left? >> >> The reason is mentioned in comments, but let me try to exp

Re: [HACKERS] Parallel Index Scans

2017-02-10 Thread Amit Kapila
On Fri, Feb 10, 2017 at 11:27 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Feb 1, 2017 at 8:20 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> The hunk in indexam.c looks like a leftover that can be omitted. >> >> It is not a leftover hunk. E

Re: [HACKERS] Passing query string to workers

2017-02-10 Thread Amit Kapila
you can remove the macro definition from > parallel.c. Thoughts? > Yes, I also don't see any need of defining it in parallel.c. I think she has kept to report it in pg_stat_activity, but I feel that code can also be moved to execParallel.c. Another question is don't we need to set

Re: [HACKERS] Parameterization of partial path

2017-02-10 Thread Amit Kapila
hat I am missing something, but if someone can present a use case, then I think we can try to support such parallel plans. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] parallelize queries containing initplans

2017-02-10 Thread Amit Kapila
On Tue, Jan 31, 2017 at 4:16 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Wed, Dec 28, 2016 at 5:20 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > >> The drawback of the second approach is >> that we need to evaluate the initplan before it is actually r

Re: [HACKERS] pg_basebackup -R

2017-02-09 Thread Amit Kapila
are > both new in v10 and have non-empty default values, yet neither of them > seems like something that you necessarily want in your > automatically-created recovery.conf file. It seems like it would be > more correct to leave out parameters that are set to the default > value, rather

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-09 Thread Amit Kapila
ped from shared buffers before the server has stopped. The way to extend it could be that while dumping it just dumps the btree internal pages or something like that. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-ha

Re: [HACKERS] Parallel Index Scans

2017-02-09 Thread Amit Kapila
On Thu, Feb 9, 2017 at 12:08 PM, Peter Geoghegan <p...@bowt.ie> wrote: > On Wed, Feb 8, 2017 at 10:33 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> I had some offlist discussion with Robert about the above point and we >> feel that keeping only heap pages for

Re: [HACKERS] Parallel Index Scans

2017-02-08 Thread Amit Kapila
On Sat, Feb 4, 2017 at 7:14 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Sat, Feb 4, 2017 at 5:54 AM, Robert Haas <robertmh...@gmail.com> wrote: >> On Wed, Feb 1, 2017 at 12:58 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > >> On balance, I'm some

Re: [HACKERS] pg_stat_wal_write statistics view

2017-02-08 Thread Amit Kapila
riting before the last reset? I think there should be a separate column for write and sync time. > Or it is possible to integrate the new columns into the existing > pg_stat_bgwriter view also. > I feel separate view is better. -- With Regards, Amit Kapila. EnterpriseDB: http://w

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-08 Thread Amit Kapila
On Tue, Feb 7, 2017 at 5:14 PM, Mithun Cy <mithun...@enterprisedb.com> wrote: > On Tue, Feb 7, 2017 at 12:24 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> On Tue, Feb 7, 2017 at 11:53 AM, Beena Emerson <memissemer...@gmail.com> >> wrote: >>>

Re: [HACKERS] GSoC 2017 - LAST CALL

2017-02-07 Thread Amit Kapila
what we are discussing in thread: https://www.postgresql.org/message-id/CA%2BTgmoaXQEt4tZ03FtQhnzeDEMzBck%2BLrni0UWHVVgOTnA6C1w%40mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] WAL consistency check facility

2017-02-07 Thread Amit Kapila
t makes you think that the else case shouldn't exist? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-06 Thread Amit Kapila
ing, then don't allow to start a new one? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-06 Thread Amit Kapila
re is no explanation in the patch or otherwise when and why this needs a pair of workers. Also, if the dump interval is greater than zero, then do we really need to separately register a dynamic worker? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers

Re: [HACKERS] Add pgstathashindex() to get hash index table statistics.

2017-02-06 Thread Amit Kapila
pages from here onwards. > > Maybe we should call them "unused pages". > +1. If we consider some more names for that column then probably one alternative could be "empty pages". -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

2017-02-06 Thread Amit Kapila
On Mon, Feb 6, 2017 at 10:28 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: > Amit Kapila <amit.kapil...@gmail.com> writes: >> Hmm. Consider that the first time relcahe invalidation occurs while >> computing id_attrs, so now the retry logic will compute the correct >&

Re: [HACKERS] Add pgstathashindex() to get hash index table statistics.

2017-02-05 Thread Amit Kapila
ere will be any > zero pages from here onwards. > Can't we use PageIsEmpty() to show such information? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

2017-02-05 Thread Amit Kapila
On Mon, Feb 6, 2017 at 9:47 AM, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > > > On Mon, Feb 6, 2017 at 9:41 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> >> >> >> Hmm. Consider that the first time relcahe invalidation occurs while

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

2017-02-05 Thread Amit Kapila
On Mon, Feb 6, 2017 at 8:35 AM, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > > > On Mon, Feb 6, 2017 at 8:15 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> >> On Mon, Feb 6, 2017 at 8:01 AM, Pavan Deolasee <pavan.deola...@gmail.com> >> wrot

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

2017-02-05 Thread Amit Kapila
that retry logic is now limited to only when index set changes, which > fits in the situation we're dealing with. > Don't you think it also has the same problem as mentioned by me in Alvaro's patch and you also agreed on it? Do you see any need to fix it locally in RelationGetIndexAttrB

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

2017-02-05 Thread Amit Kapila
e but I think your solution can be extended to make it somewhat similar to what we do with rd_indexvalid (basically, we should set rd_bitmapsvalid to 2 (bitmap is temporarily forced) at rel cache invalidation and then clean it up transaction end). I can try something on those lines. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Draft release notes for next week's releases are up for review

2017-02-05 Thread Amit Kapila
into further-back branches. I'll sort that out when I split > things up. Right now the question is whether individual items are > correctly/adequately documented. > > Fix possible miss of socket read events while waiting on Windows (Amit Kapial) Typo Amit Kapial/Amit Kapila --

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

2017-02-03 Thread Amit Kapila
ote that we get a copy here, so we need not worry about relcache flush * happening midway through. */ hot_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_ALL); key_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_KEY); id_attrs = RelationGetIndexAttrBitmap(relation, IN

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2017-02-03 Thread Amit Kapila
On Sat, Feb 4, 2017 at 4:37 AM, Jeff Janes <jeff.ja...@gmail.com> wrote: > On Thu, Jan 12, 2017 at 7:23 PM, Amit Kapila <amit.kapil...@gmail.com> > wrote: >> >> On Fri, Jan 13, 2017 at 1:04 AM, Jesper Pedersen >> <jesper.peder...@redhat.com> wrote: >

Re: [HACKERS] Parallel Index Scans

2017-02-03 Thread Amit Kapila
On Sat, Feb 4, 2017 at 5:54 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Feb 1, 2017 at 12:58 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Yeah, I understand that point and I can see there is strong argument >> to do that way, but let's wait and see w

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

2017-02-02 Thread Amit Kapila
r the particular session where you are seeing the problem. I could see few places where we are setting it to 0 and clearing rd_indexlist, but not rd_indexattr. I am not sure whether that has anything to do with the problem you are seeing. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Commit fest 2017-01 will begin soon!

2017-02-01 Thread Amit Kapila
all the patches that are marked "Returned with Feedback" or "Rejected" or "Moved to next CF". OTOH, there is some value in sending a separate mail for each patch so that people can argue on the decision by CFM for one particular patch, but not sure if it worth. -- Wit

Re: [HACKERS] parallelize queries containing subplans

2017-02-01 Thread Amit Kapila
On Wed, Feb 1, 2017 at 10:04 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Jan 31, 2017 at 6:01 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Moved this patch to next CF. > > So here is what seems to be the key hunk from this patch: > >

Re: [HACKERS] Parallel Index Scans

2017-02-01 Thread Amit Kapila
On Wed, Feb 1, 2017 at 3:52 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Jan 24, 2017 at 4:51 PM, Robert Haas <robertmh...@gmail.com> wrote: >> On Mon, Jan 23, 2017 at 1:03 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> In spite of

Re: [HACKERS] Parallel Index Scans

2017-01-31 Thread Amit Kapila
of > the number of heap pages scanned. > Yeah, I understand that point and I can see there is strong argument to do that way, but let's wait and see what others including Robert have to say about this point. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent

Re: [HACKERS] parallelize queries containing subplans

2017-01-31 Thread Amit Kapila
On Tue, Jan 24, 2017 at 10:59 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Thu, Jan 19, 2017 at 4:51 PM, Dilip Kumar <dilipbal...@gmail.com> wrote: >> On Thu, Jan 19, 2017 at 3:05 PM, Dilip Kumar <dilipbal...@gmail.com> wrote: >>> During debugging I

Re: [HACKERS] parallelize queries containing initplans

2017-01-31 Thread Amit Kapila
On Wed, Dec 28, 2016 at 5:20 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > To start > with let us see the plan of TPC-H query (Q-22) and understand how it > can be improved. > > Limit >InitPlan 1 (returns $0) > -> Finalize Agg

Re: [HACKERS] Supporting huge pages on Windows

2017-01-30 Thread Amit Kapila
On Mon, Jan 30, 2017 at 7:16 AM, Tsunakawa, Takayuki <tsunakawa.ta...@jp.fujitsu.com> wrote: > From: Amit Kapila [mailto:amit.kapil...@gmail.com] >> Hmm. It doesn't work even on a command prompt with administrative >> privileges. It gives below error: >> >> waiti

Re: [HACKERS] Parallel Index Scans

2017-01-30 Thread Amit Kapila
On Sat, Jan 28, 2017 at 1:34 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Jan 23, 2017 at 1:03 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> parallel_index_opt_exec_support_v6 - Removed the usage of >> GatherSupportsBackwardScan. Expanded the comment

Re: [HACKERS] Microvacuum support for Hash Index

2017-01-28 Thread Amit Kapila
ckends trying > to do so there is high chance of getting deadlock which i assume is > expected behaviour in database. > I agree with your analysis, surely trying to update multiple rows with same values from different backends can lead to deadlock. -- With Regards, Amit Kapila. Enterprise

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-26 Thread Amit Kapila
te of the other instance from that? > > In my experience, that kind of use is just as interesting as preserving > the buffers across a restart. > An interesting use case. I am not sure if somebody has tried that way but it appears to me that the current proposed patch should work for th

Re: [HACKERS] Microvacuum support for Hash Index

2017-01-26 Thread Amit Kapila
the reason. It is quite possible that the test case is such that the deadlock is expected in rare cases, if that is the case then it is okay. I have not spent enough time on that to comment whether it is a test or code issue. [1] - https://www.postgresql.org/message-id/dc6d7247-050f-4014-8c80-a4ee676eb384%40redh

Re: [HACKERS] Parallel Index Scans

2017-01-26 Thread Amit Kapila
On Fri, Jan 27, 2017 at 6:53 AM, Haribabu Kommi <kommi.harib...@gmail.com> wrote: > > > On Mon, Jan 23, 2017 at 5:07 PM, Amit Kapila <amit.kapil...@gmail.com> > wrote: >> >> On Fri, Jan 20, 2017 at 7:29 AM, Haribabu Kommi >> <kommi.harib...@gmail.co

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Amit Kapila
+TO DO: +-- +Add functionality to dump based on timer at regular interval. I think you need to remove above TO DO. 2. + /* Load the page only if there exist a free buffer. We do not want to + * replace an existing buffer. */ This is not a PG style multiline comment. -- With Regards, Amit Kapila.

Re: [HACKERS] parallelize queries containing subplans

2017-01-24 Thread Amit Kapila
On Tue, Jan 24, 2017 at 10:59 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Thu, Jan 19, 2017 at 4:51 PM, Dilip Kumar <dilipbal...@gmail.com> wrote: >> On Thu, Jan 19, 2017 at 3:05 PM, Dilip Kumar <dilipbal...@gmail.com> wrote: >>> During debugging I

Re: [HACKERS] Cache Hash Index meta page.

2017-01-24 Thread Amit Kapila
On Wed, Jan 18, 2017 at 11:51 AM, Mithun Cy <mithun...@enterprisedb.com> wrote: > On Tue, Jan 17, 2017 at 10:07 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > >> (b) Another somewhat bigger problem is that with this new change it >> won't retain the pin on meta pa

Re: [HACKERS] pageinspect: Hash index support

2017-01-23 Thread Amit Kapila
let's keep the interface as proposed by you. TBH, I am not sure how important is to expose this (hash_bitmap_info()) function, but let's keep it and see if committer has any opinion on this API. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers ma

Re: [HACKERS] parallelize queries containing subplans

2017-01-23 Thread Amit Kapila
that is certainly not essential for this project). So, I have set parallel_safe as false for CTE subplans. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com pq_pushdown_subplan_v4.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] Checksums by default?

2017-01-23 Thread Amit Kapila
On Mon, Jan 23, 2017 at 6:57 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: > On 01/23/2017 01:40 PM, Amit Kapila wrote: >> >> On Mon, Jan 23, 2017 at 3:56 PM, Tomas Vondra >> <tomas.von...@2ndquadrant.com> wrote: >>> >>> On 01/23/2017

Re: [HACKERS] Checksums by default?

2017-01-23 Thread Amit Kapila
On Mon, Jan 23, 2017 at 3:56 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: > On 01/23/2017 09:57 AM, Amit Kapila wrote: >> >> On Mon, Jan 23, 2017 at 1:18 PM, Tomas Vondra >> <tomas.von...@2ndquadrant.com> wrote: >>> >>> On 01/23/2017

Re: [HACKERS] pageinspect: Hash index support

2017-01-23 Thread Amit Kapila
ave a prototype of a function as hash_bitmap_info(index_oid, page bytea) which will be quite similar to other API's exposed by this module. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Fix a comment in feelist.c

2017-01-23 Thread Amit Kapila
he comment. >> >> commit d72731a70450b5e7084991b9caa15cb58a2820df >> Author: Andres Freund <and...@anarazel.de> >> Date: Thu Dec 25 18:24:20 2014 +0100 >> >> Lockless StrategyGetBuffer clock sweep hot path. > > Looks good to me. > +1. -- With Regards, Amit Kap

Re: [HACKERS] Checksums by default?

2017-01-23 Thread Amit Kapila
On Mon, Jan 23, 2017 at 1:18 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: > On 01/23/2017 08:30 AM, Amit Kapila wrote: >> >> >> I think if we can get data for pgbench read-write workload when data >> doesn't fit in shared buffers but fit in RAM, that ca

Re: [HACKERS] Checksums by default?

2017-01-22 Thread Amit Kapila
t next read. I think it also makes sense to check the WAL data size for each of those runs. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgres

Re: [HACKERS] Parallel Index Scans

2017-01-22 Thread Amit Kapila
On Fri, Jan 20, 2017 at 7:29 AM, Haribabu Kommi <kommi.harib...@gmail.com> wrote: > > On Thu, Jan 19, 2017 at 1:18 AM, Amit Kapila <amit.kapil...@gmail.com> > wrote: >> > >> > +extern BlockNumber _bt_parallel_seize(IndexScanDesc scan, bool >> >

Re: [HACKERS] Parallel Index Scans

2017-01-22 Thread Amit Kapila
On Sat, Jan 21, 2017 at 12:23 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Sat, Jan 21, 2017 at 1:15 AM, Robert Haas <robertmh...@gmail.com> wrote: >> >> I think it's a good idea to put all three of those functions together >> in th

Re: [HACKERS] new autovacuum criterion for visible pages

2017-01-21 Thread Amit Kapila
UPDATEs and DELETEs. > > Tracking the INSERTs as a reason to VACUUM is also very natural when you > consider the need to update BRIN indexes. > Another possible advantage of tracking INSERTs is for hash indexes where after split we need to remove tuples from buckets that underwent sp

Re: [HACKERS] Parallel Index Scans

2017-01-20 Thread Amit Kapila
On Sat, Jan 21, 2017 at 1:15 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Fri, Jan 20, 2017 at 9:29 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Sure, if scan keys have changed then we can't continue, but this is >> the case where runtime keys

Re: [HACKERS] Parallel Index Scans

2017-01-20 Thread Amit Kapila
On Fri, Jan 20, 2017 at 3:41 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Jan 18, 2017 at 8:03 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> Why do we need separate AM support for index_parallelrescan()? Why >>> can't index_rescan()

Re: [HACKERS] Parallel Index Scans

2017-01-20 Thread Amit Kapila
On Fri, Jan 20, 2017 at 12:59 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Thu, Jan 19, 2017 at 4:26 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Fixed. > > > If all of that were no issue, the considerations in > TargetListSupportsBackwardScan coul

Re: [HACKERS] Parallel Index Scans

2017-01-19 Thread Amit Kapila
On Thu, Jan 19, 2017 at 12:28 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Jan 16, 2017 at 7:11 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Fixed. > > With respect to the second patch > (parallel_index_opt_exec_support_v4.patch), I'm hoping it c

Re: [HACKERS] Parallel Index Scans

2017-01-19 Thread Amit Kapila
On Wed, Jan 18, 2017 at 6:25 AM, Haribabu Kommi <kommi.harib...@gmail.com> wrote: > > > On Mon, Jan 16, 2017 at 11:11 PM, Amit Kapila <amit.kapil...@gmail.com> > wrote: >> >> >> Changed as per suggestion. >> >> >&

<    1   2   3   4   5   6   7   8   9   10   >