Re: [HACKERS] Parallel Seq Scan

On Mon, Jul 6, 2015 at 10:54 PM, Jeff Davis wrote: > > On Mon, 2015-07-06 at 10:37 +0530, Amit Kapila wrote: > > > Or the other way to look at it could be separate out fields which are > > required for parallel scan which is done currently by forming a > > separate structure ParallelHeapScanDescDa

Re: [HACKERS] Parallel Seq Scan

On Fri, Jul 3, 2015 at 10:05 PM, Amit Kapila wrote: > Attached, find the rebased version of patch. > > Note - You need to first apply the assess-parallel-safety patch which you > can find at: > http://www.postgresql.org/message-id/CAA4eK1JjsfE_dOsHTr_z1P_cBKi_X4C4X3d7Nv=vwx9fs7q...@mail.gmail.com

Re: [HACKERS] Parallel Seq Scan

On Mon, 2015-07-06 at 10:37 +0530, Amit Kapila wrote: > Or the other way to look at it could be separate out fields which are > required for parallel scan which is done currently by forming a > separate structure ParallelHeapScanDescData. > I was suggesting that you separate out both the normal s

Re: [HACKERS] Parallel Seq Scan

On Mon, Jul 6, 2015 at 3:26 AM, Jeff Davis wrote: > > On Fri, 2015-07-03 at 17:35 +0530, Amit Kapila wrote: > > > Attached, find the rebased version of patch. > > > > Comments: > > > * The heapam.c changes seem a little ad-hoc. Conceptually, which > portions should be affected by parallelism? How

Re: [HACKERS] Parallel Seq Scan

On Fri, 2015-07-03 at 17:35 +0530, Amit Kapila wrote: > Attached, find the rebased version of patch. > Comments: * The heapam.c changes seem a little ad-hoc. Conceptually, which portions should be affected by parallelism? How do we know we didn't miss something? * Why is initscan getting the n

Re: [HACKERS] Parallel Seq Scan

On 01/07/15 17:37, Amit Kapila wrote: On Tue, Jun 30, 2015 at 4:00 AM, Jeff Davis > wrote: > > [Jumping in without catching up on entire thread. [...] . > 2. Where is the speedup coming from? How much of it is CPU and IO > overlapping (i.e. not leaving disk or CPU id

Re: [HACKERS] Parallel Seq Scan

On Wed, Jul 1, 2015 at 1:21 PM, Jeff Davis wrote: > > On Wed, 2015-07-01 at 11:07 +0530, Amit Kapila wrote: > > > For what you are asking to change name for? > > There are still some places, at least in the comments, that call it a > parallel sequential scan. > In the initial version of patch, th

Re: [HACKERS] Parallel Seq Scan

t > Langote; Amit Langote; Fabrízio Mello; Thom Brown; Stephen Frost; > pgsql-hackers > Subject: Re: [HACKERS] Parallel Seq Scan > > On Wed, 2015-07-01 at 11:07 +0530, Amit Kapila wrote: > > > For what you are asking to change name for? > > There are still some place

Re: [HACKERS] Parallel Seq Scan

On Wed, 2015-07-01 at 11:07 +0530, Amit Kapila wrote: > For what you are asking to change name for? There are still some places, at least in the comments, that call it a parallel sequential scan. > a. Infrastructure for parallel execution, like some of the stuff in > execparallel.c, heapam.c,tq

Re: [HACKERS] Parallel Seq Scan

On 2015-07-01 PM 02:37, Amit Kapila wrote: > > In terms of completeness, I think we should add some documentation > for this patch, one way is to update about the execution mechanism in > src/backend/access/transam/README.parallel and then explain about > new configuration knobs in documentation (

Re: [HACKERS] Parallel Seq Scan

On Tue, Jun 30, 2015 at 4:00 AM, Jeff Davis wrote: > > [Jumping in without catching up on entire thread. No problem. > Please let me know > if these questions have already been covered.] > > 1. Can you change the name to something like ParallelHeapScan? > Parallel Sequential is a contradiction.

Re: [HACKERS] Parallel Seq Scan

[Jumping in without catching up on entire thread. Please let me know if these questions have already been covered.] 1. Can you change the name to something like ParallelHeapScan? Parallel Sequential is a contradiction. (I know this is bikeshedding and I won't protest further if you keep the name.)

Re: [HACKERS] Parallel Seq Scan

On Tue, May 19, 2015 at 8:45 AM, Amit Kapila wrote: > On Mon, May 11, 2015 at 3:00 AM, Robert Haas wrote: >> I think it might be better to try to solve this problem in a more >> localized way. Can we arrange for planstate->instrumentation to point >> directory into the DSM, instead of copying th

Re: [HACKERS] Parallel Seq Scan

On Mon, May 11, 2015 at 3:00 AM, Robert Haas wrote: > > > I think it might be better to try to solve this problem in a more > localized way. Can we arrange for planstate->instrumentation to point > directory into the DSM, instead of copying the data over later? Yes, we can do that but I am not s

Re: [HACKERS] Parallel Seq Scan

On Mon, May 18, 2015 at 6:28 AM, Haribabu Kommi wrote: > > On Wed, Apr 22, 2015 at 10:48 PM, Amit Kapila wrote: > > parallel_seqscan_v14.patch (Attached with this mail) > > This patch is not applying/working with the latest head after parallel > mode patch got committed. > can you please rebase t

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 22, 2015 at 10:48 PM, Amit Kapila wrote: > parallel_seqscan_v14.patch (Attached with this mail) This patch is not applying/working with the latest head after parallel mode patch got committed. can you please rebase the patch. Regards, Hari Babu Fujitsu Australia -- Sent via pgsql-

Re: [HACKERS] Parallel Seq Scan

On Thu, May 7, 2015 at 3:23 AM, Amit Kapila wrote: >> > I observed one issue while working on this review comment. When we >> > try to destroy the parallel setup via ExecEndNode (as due to Limit >> > Node, it could not destroy after consuming all tuples), it waits for >> > parallel >> > workers t

Re: [HACKERS] Parallel Seq Scan

On Wed, May 6, 2015 at 7:10 PM, Robert Haas wrote: > > On Wed, May 6, 2015 at 7:55 AM, Amit Kapila wrote: > >>> - I believe the separation of concerns between ExecFunnel() and > >>> ExecEndFunnel() is not quite right. If the scan is shut down before > >>> it runs to completion (e.g. because of L

Re: [HACKERS] Parallel Seq Scan

On Wed, May 6, 2015 at 7:55 AM, Amit Kapila wrote: >>> - I believe the separation of concerns between ExecFunnel() and >>> ExecEndFunnel() is not quite right. If the scan is shut down before >>> it runs to completion (e.g. because of LIMIT), then I think we'll call >>> ExecEndFunnel() before Exec

Re: [HACKERS] Parallel Seq Scan

On Tue, Apr 28, 2015 at 5:37 PM, Robert Haas wrote: > > On Fri, Apr 24, 2015 at 8:32 AM, Amit Kapila wrote: >> - I believe the separation of concerns between ExecFunnel() and >> ExecEndFunnel() is not quite right. If the scan is shut down before >> it runs to completion (e.g. because of LIMIT),

Re: [HACKERS] Parallel Seq Scan

On Fri, Apr 24, 2015 at 8:32 AM, Amit Kapila wrote: >> - InitializeParallelWorkers() still mixes together general parallel >> executor concerns with concerns specific to parallel sequential scan >> (e.g. EstimatePartialSeqScanSpace). > > Here we are doing 2 things, first one is for planned stateme

Re: [HACKERS] Parallel Seq Scan

On Thu, Apr 23, 2015 at 2:26 AM, Robert Haas wrote: > > On Wed, Apr 22, 2015 at 8:48 AM, Amit Kapila wrote: > > I have implemented this idea (note that I have to expose a new API > > shm_mq_from_handle as TupleQueueFunnel stores shm_mq_handle* and > > we sum_mq* to call shm_mq_detach) and apart t

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 22, 2015 at 8:48 AM, Amit Kapila wrote: > I have implemented this idea (note that I have to expose a new API > shm_mq_from_handle as TupleQueueFunnel stores shm_mq_handle* and > we sum_mq* to call shm_mq_detach) and apart this I have fixed other > problems reported on this thread: > >

Re: [HACKERS] Parallel Seq Scan

On Tue, Apr 21, 2015 at 9:38 AM, Amit Kapila wrote: > On Mon, Apr 20, 2015 at 10:08 PM, Robert Haas wrote: >> >> On Tue, Apr 7, 2015 at 11:58 PM, Amit Kapila >> wrote: >> > One disadvantage of retaining parallel-paths could be that it can >> > increase the number of combinations planner might ne

Re: [HACKERS] Parallel Seq Scan

On Mon, Apr 20, 2015 at 10:08 PM, Robert Haas wrote: > > On Tue, Apr 7, 2015 at 11:58 PM, Amit Kapila wrote: > > One disadvantage of retaining parallel-paths could be that it can > > increase the number of combinations planner might need to evaluate > > during planning (in particular during join

Re: [HACKERS] Parallel Seq Scan

On Tue, Apr 21, 2015 at 6:34 AM, Amit Langote wrote: > On 2015-04-21 AM 03:29, Robert Haas wrote: > > On Wed, Apr 8, 2015 at 3:38 AM, Amit Langote wrote: > >> On 08-04-2015 PM 12:46, Amit Kapila wrote: > >>> Going forward, I think we can improve the same if we decide not to shutdown > >>> parallel

Re: [HACKERS] Parallel Seq Scan

On Tue, Apr 21, 2015 at 2:29 PM, David Rowley wrote: > > I've also been thinking about how, instead of having to have a special > PartialSeqScan node which contains a bunch of code to store tuples in a > shared memory queue, could we not have a "TupleBuffer", or > "ParallelTupleReader" node, one o

Re: [HACKERS] Parallel Seq Scan

On 21 April 2015 at 06:26, Robert Haas wrote: > On Wed, Apr 8, 2015 at 3:34 AM, David Rowley wrote: > > In summary it sounds like with my idea we get: > > > > Pros > > * Optimal plan if no workers are available at execution time. > > * Parallelism possible if the chosen optimal plan happens to s

Re: [HACKERS] Parallel Seq Scan

On 2015-04-21 AM 03:29, Robert Haas wrote: > On Wed, Apr 8, 2015 at 3:38 AM, Amit Langote wrote: >> On 08-04-2015 PM 12:46, Amit Kapila wrote: >>> Going forward, I think we can improve the same if we decide not to shutdown >>> parallel workers till postmaster shutdown once they are started and >>>

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 8, 2015 at 3:38 AM, Amit Langote wrote: > On 08-04-2015 PM 12:46, Amit Kapila wrote: >> Going forward, I think we can improve the same if we decide not to shutdown >> parallel workers till postmaster shutdown once they are started and >> then just allocate them during executor-start ph

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 8, 2015 at 3:34 AM, David Rowley wrote: > On 8 April 2015 at 14:24, Robert Haas wrote: >> I think one of the philosophical questions that has to be answered >> here is "what does it mean to talk about the cost of a parallel >> plan?". For a non-parallel plan, the cost of the plan mea

Re: [HACKERS] Parallel Seq Scan

On Tue, Apr 7, 2015 at 11:58 PM, Amit Kapila wrote: > One disadvantage of retaining parallel-paths could be that it can > increase the number of combinations planner might need to evaluate > during planning (in particular during join path evaluation) unless we > do some special handling to avoid e

Re: [HACKERS] Parallel Seq Scan

On 9 April 2015 at 00:12, Amit Kapila wrote: > On Wed, Apr 8, 2015 at 3:30 PM, David Rowley wrote: > > > > On 8 April 2015 at 15:46, Amit Kapila wrote: > >> > >> I think there is always a chance that resources (like parallel-workers) > >> won't be available at run-time even if we decide about t

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 8, 2015 at 3:30 PM, David Rowley wrote: > > On 8 April 2015 at 15:46, Amit Kapila wrote: >> >> I think there is always a chance that resources (like parallel-workers) >> won't be available at run-time even if we decide about them at >> executor-start phase unless we block it for that

Re: [HACKERS] Parallel Seq Scan

On 8 April 2015 at 15:46, Amit Kapila wrote: > On Wed, Apr 8, 2015 at 1:53 AM, Kevin Grittner wrote: > > > > David Rowley wrote: > > > > > If we attempt to do this parallel stuff at plan time, and we > > > happen to plan at some quiet period, or perhaps worse, some > > > application's start-up

Re: [HACKERS] Parallel Seq Scan

On 08-04-2015 PM 12:46, Amit Kapila wrote: > Going forward, I think we can improve the same if we decide not to shutdown > parallel workers till postmaster shutdown once they are started and > then just allocate them during executor-start phase. > I wonder if it makes sense to invent the notion o

Re: [HACKERS] Parallel Seq Scan

On 8 April 2015 at 14:24, Robert Haas wrote: > I think one of the philosophical questions that has to be answered > here is "what does it mean to talk about the cost of a parallel > plan?". For a non-parallel plan, the cost of the plan means both "the > amount of effort we will spend executing t

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 8, 2015 at 7:54 AM, Robert Haas wrote: > > I agree that this is an area that needs more thought. I don't > (currently, anyway) agree that the planner shouldn't know anything > about parallelism. The problem with that is that there's lots of > relevant stuff that can only be known at

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 8, 2015 at 1:53 AM, Kevin Grittner wrote: > > David Rowley wrote: > > > If we attempt to do this parallel stuff at plan time, and we > > happen to plan at some quiet period, or perhaps worse, some > > application's start-up process happens to PREPARE a load of > > queries when the dat

Re: [HACKERS] Parallel Seq Scan

On Sat, Apr 4, 2015 at 5:19 AM, David Rowley wrote: > Going over the previous emails in this thread I see that it has been a long > time since anyone discussed anything around how we might decide at planning > time how many workers should be used for the query, and from the emails I > don't recall

Re: [HACKERS] Parallel Seq Scan

David Rowley wrote: > If we attempt to do this parallel stuff at plan time, and we > happen to plan at some quiet period, or perhaps worse, some > application's start-up process happens to PREPARE a load of > queries when the database is nice and quite, then quite possibly > we'll end up with som

Re: [HACKERS] Parallel Seq Scan

So I've just finished reading the impressive 244 emails (so far) about Parallel Seq scan, and I've had a quick skim over the latest patch. Its quite exciting to think that one day we'll have parallel query in PostgreSQL, but I have to say, that I think that there's a major point about the proposed

Re: [HACKERS] Parallel Seq Scan

On Thu, Apr 2, 2015 at 3:07 AM, Amit Kapila wrote: > On Wed, Apr 1, 2015 at 6:11 PM, Robert Haas wrote: >> On Wed, Apr 1, 2015 at 7:30 AM, Amit Kapila >> wrote: >> >> Also, the new code to propagate >> >> XactLastRecEnd won't work right, either. >> > >> > As we are generating FATAL error on term

Re: [HACKERS] Parallel Seq Scan

On Thu, Apr 2, 2015 at 2:36 AM, Amit Kapila wrote: >> If I'm not confused, it would be the other way around. We would run >> the initPlan in the master backend *first* and then the rest in the >> workers. > > Either one of us is confused, let me try to describe my understanding in > somewhat more

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 1, 2015 at 6:11 PM, Robert Haas wrote: > > On Wed, Apr 1, 2015 at 7:30 AM, Amit Kapila wrote: > >> Also, the new code to propagate > >> XactLastRecEnd won't work right, either. > > > > As we are generating FATAL error on termination of worker > > (bgworker_die()), so won't it be handl

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 1, 2015 at 8:18 PM, Robert Haas wrote: > > >> I don't think you need to do anything that complicated. I'm not > >> proposing to *run* the initPlan in the workers, just to pass the > >> parameter values down. > > > > Sorry, but I am not able to understand how it will help if just param

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 1, 2015 at 10:28 AM, Amit Kapila wrote: >> Well, if we *don't* handle it, we're going to need to insert some hack >> to ensure that the planner doesn't create plans. And that seems >> pretty unappealing. Maybe it'll significantly compromise plan >> quality, and maybe it won't, but at

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 1, 2015 at 6:03 PM, Robert Haas wrote: > > On Tue, Mar 31, 2015 at 8:53 AM, Amit Kapila wrote: > >> It looks to me like the is an InitPlan, not a subplan. There > >> shouldn't be any problem with a Funnel node having an InitPlan; it > >> looks to me like all of the InitPlan stuff is

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 1, 2015 at 7:30 AM, Amit Kapila wrote: >> > Patch fixes the problem and now for Rescan, we don't need to Wait >> > for workers to finish. >> >> I realized that there is a problem with this. If an error occurs in >> one of the workers just as we're deciding to kill them all, then the >

Re: [HACKERS] Parallel Seq Scan

On Wed, Apr 1, 2015 at 6:30 AM, Amit Kapila wrote: > On Mon, Mar 30, 2015 at 8:35 PM, Robert Haas wrote: >> So, suppose we have a plan like this: >> >> Append >> -> Funnel >> -> Partial Seq Scan >> -> Funnel >> -> Partial Seq Scan >> (repeated many times) >> >> In earlier versions of this pat

Re: [HACKERS] Parallel Seq Scan

On Tue, Mar 31, 2015 at 8:53 AM, Amit Kapila wrote: >> It looks to me like the is an InitPlan, not a subplan. There >> shouldn't be any problem with a Funnel node having an InitPlan; it >> looks to me like all of the InitPlan stuff is handled by common code >> within the executor (grep for initPl

Re: [HACKERS] Parallel Seq Scan

On Mon, Mar 30, 2015 at 8:31 PM, Robert Haas wrote: > > On Wed, Mar 18, 2015 at 11:43 PM, Amit Kapila wrote: > >> I think I figured out the problem. That fix only helps in the case > >> where the postmaster noticed the new registration previously but > >> didn't start the worker, and then later

Re: [HACKERS] Parallel Seq Scan

On Mon, Mar 30, 2015 at 8:35 PM, Robert Haas wrote: > > On Wed, Mar 25, 2015 at 6:27 AM, Amit Kapila wrote: > > Apart from that I have moved the Initialization of dsm segement from > > InitNode phase to ExecFunnel() (on first execution) as per suggestion > > from Robert. The main idea is that as

Re: [HACKERS] Parallel Seq Scan

On Mon, Mar 30, 2015 at 8:11 PM, Robert Haas wrote: > > On Fri, Mar 27, 2015 at 2:34 AM, Amit Kapila wrote: > > The reason of this problem is that above tab-completion is executing > > query [1] which contains subplan for the funnel node and currently > > we don't have capability (enough infrastr

Re: [HACKERS] Parallel Seq Scan

On Wed, Mar 25, 2015 at 6:27 AM, Amit Kapila wrote: > Apart from that I have moved the Initialization of dsm segement from > InitNode phase to ExecFunnel() (on first execution) as per suggestion > from Robert. The main idea is that as it creates large shared memory > segment, so do the work when

Re: [HACKERS] Parallel Seq Scan

On Wed, Mar 18, 2015 at 11:43 PM, Amit Kapila wrote: >> I think I figured out the problem. That fix only helps in the case >> where the postmaster noticed the new registration previously but >> didn't start the worker, and then later notices the termination. >> What's much more likely to happen i

Re: [HACKERS] Parallel Seq Scan

On Fri, Mar 27, 2015 at 2:34 AM, Amit Kapila wrote: > The reason of this problem is that above tab-completion is executing > query [1] which contains subplan for the funnel node and currently > we don't have capability (enough infrastructure) to support execution > of subplans by parallel workers.

Re: [HACKERS] Parallel Seq Scan

On Wed, Mar 25, 2015 at 9:53 PM, Thom Brown wrote: > > On 25 March 2015 at 15:49, Amit Kapila wrote: >> >> On Wed, Mar 25, 2015 at 5:16 PM, Thom Brown wrote: >> > Okay, with my pgbench_accounts partitioned into 300, I ran: >> > >> > SELECT DISTINCT bid FROM pgbench_accounts; >> > >> > The query

Re: [HACKERS] Parallel Seq Scan

On 25 March 2015 at 15:49, Amit Kapila wrote: > On Wed, Mar 25, 2015 at 5:16 PM, Thom Brown wrote: > > > > On 25 March 2015 at 10:27, Amit Kapila wrote: > >> > >> Fixed the reported issue on assess-parallel-safety thread and another > >> bug caught while testing joins and integrated with latest

Re: [HACKERS] Parallel Seq Scan

On Wed, Mar 25, 2015 at 5:16 PM, Thom Brown wrote: > > On 25 March 2015 at 10:27, Amit Kapila wrote: >> >> Fixed the reported issue on assess-parallel-safety thread and another >> bug caught while testing joins and integrated with latest version of >> parallel-mode patch (parallel-mode-v9 patch).

Re: [HACKERS] Parallel Seq Scan

On 25 March 2015 at 11:46, Thom Brown wrote: > > > Still not sure why 8 workers are needed for each partial scan. I would > expect 8 workers to be used for 8 separate scans. Perhaps this is just my > misunderstanding of how this feature works. > Another issue: SELECT * FROM pgb *crash* Logs:

Re: [HACKERS] Parallel Seq Scan

On 25 March 2015 at 10:27, Amit Kapila wrote: > On Fri, Mar 20, 2015 at 5:36 PM, Amit Kapila > wrote: > > > > > > So the patches have to be applied in below sequence: > > HEAD Commit-id : 8d1f2390 > > parallel-mode-v8.1.patch [2] > > assess-parallel-safety-v4.patch [1] > > parallel-heap-scan.pat

Re: [HACKERS] Parallel Seq Scan

On Wed, Mar 25, 2015 at 4:08 PM, Rajeev rastogi wrote: > > On 25 March 2015 16:00, Amit Kapila Wrote: > > > Which version of patch you are looking at? > > > I am seeing below code in ExecInitFunnel() in Version-11 to which > > > you have replied. > > > > > + /* Funnel node doesn't have innerPlan

Re: [HACKERS] Parallel Seq Scan

ogi Cc: Amit Langote; Robert Haas; Andres Freund; Kouhei Kaigai; Amit Langote; Fabrízio Mello; Thom Brown; Stephen Frost; pgsql-hackers Subject: Re: [HACKERS] Parallel Seq Scan On Wed, Mar 25, 2015 at 3:47 PM, Rajeev rastogi mailto:rajeev.rast...@huawei.com>> wrote: > > On 20 Mar

Re: [HACKERS] Parallel Seq Scan

On Wed, Mar 25, 2015 at 3:47 PM, Rajeev rastogi wrote: > > On 20 March 2015 17:37, Amit Kapila Wrote: > > > So the patches have to be applied in below sequence: > > HEAD Commit-id : 8d1f2390 > > parallel-mode-v8.1.patch [2] > > assess-parallel-safety-v4.patch [1] > > parallel-heap-scan.patch [3]

Re: [HACKERS] Parallel Seq Scan

On 20 March 2015 17:37, Amit Kapila Wrote: > So the patches have to be applied in below sequence: > HEAD Commit-id : 8d1f2390 > parallel-mode-v8.1.patch [2] > assess-parallel-safety-v4.patch [1] > parallel-heap-scan.patch [3] > parallel_seqscan_v11.patch (Attached with this mail) While I was goin

Re: [HACKERS] Parallel Seq Scan

On 20-03-2015 PM 09:06, Amit Kapila wrote: > On Mon, Mar 16, 2015 at 12:58 PM, Amit Langote < > langote_amit...@lab.ntt.co.jp> wrote: >> Actually I meant "currently the last" or: >> >> funnel->nextqueue == funnel->nqueue - 1 >> >> So the code you quote would only take care of subset of the case

Re: [HACKERS] Parallel Seq Scan

On Wed, Mar 18, 2015 at 11:43 PM, Amit Kapila wrote: > Patch fixes the problem and now for Rescan, we don't need to Wait > for workers to finish. > >> Assuming this actually fixes the problem, I think we should back-patch >> it into 9.4. > > +1 OK, done. -- Robert Haas EnterpriseDB: http://www.

Re: [HACKERS] Parallel Seq Scan

On Wed, Mar 18, 2015 at 9:14 PM, Robert Haas wrote: > > On Wed, Mar 18, 2015 at 2:22 AM, Amit Kapila wrote: > > > > It didn't fix the problem. IIUC, you have done this to ensure that > > if worker is not already started, then update it's pid, so that we > > can get the required status in WaitFor

Re: [HACKERS] Parallel Seq Scan

On Wed, Mar 18, 2015 at 10:45 PM, Robert Haas wrote: > > On Sat, Mar 14, 2015 at 1:04 AM, Amit Kapila wrote: > >> # EXPLAIN SELECT DISTINCT bid FROM pgbench_accounts; > >> ERROR: too many dynamic shared memory segments > > > > This happens because we have maximum limit on the number of > > dynam

Re: [HACKERS] Parallel Seq Scan

On Sat, Mar 14, 2015 at 1:04 AM, Amit Kapila wrote: >> # EXPLAIN SELECT DISTINCT bid FROM pgbench_accounts; >> ERROR: too many dynamic shared memory segments > > This happens because we have maximum limit on the number of > dynamic shared memory segments in the system. > > In function dsm_postmas

Re: [HACKERS] Parallel Seq Scan

On Wed, Mar 18, 2015 at 2:22 AM, Amit Kapila wrote: >> Can you try this: >> >> diff --git a/src/backend/postmaster/bgworker.c >> b/src/backend/postmaster/bgworker.c >> index f80141a..39b919f 100644 >> --- a/src/backend/postmaster/bgworker.c >> +++ b/src/backend/postmaster/bgworker.c >> @@ -244,6 +

Re: [HACKERS] Parallel Seq Scan

On Tue, Mar 17, 2015 at 7:54 PM, Robert Haas wrote: > > On Tue, Mar 17, 2015 at 1:42 AM, Amit Kapila wrote: > > The problem occurs in second loop inside DestroyParallelContext() > > where it calls WaitForBackgroundWorkerShutdown(). Basically > > WaitForBackgroundWorkerShutdown() just checks for

Re: [HACKERS] Parallel Seq Scan

On Tue, Mar 17, 2015 at 1:42 AM, Amit Kapila wrote: > The problem occurs in second loop inside DestroyParallelContext() > where it calls WaitForBackgroundWorkerShutdown(). Basically > WaitForBackgroundWorkerShutdown() just checks for BGWH_STOPPED > status, refer below code in parallel-mode patch:

Re: [HACKERS] Parallel Seq Scan

On Fri, Mar 13, 2015 at 7:00 PM, Robert Haas wrote: > > On Fri, Mar 13, 2015 at 8:59 AM, Amit Kapila wrote: > > We can't directly call DestroyParallelContext() to terminate workers as > > it can so happen that by that time some of the workers are still not > > started. > > That shouldn't be a pro

Re: [HACKERS] Parallel Seq Scan

On 16-03-2015 PM 04:14, Amit Kapila wrote: > On Mon, Mar 16, 2015 at 9:40 AM, Amit Langote > wrote: >> Or if the just-detached queue happens to be the last one, we'll make >> shm_mq_receive() to read from a potentially already-detached queue in the >> immediately next iteration. > > Won't the las

Re: [HACKERS] Parallel Seq Scan

On Fri, Mar 13, 2015 at 7:06 PM, Alvaro Herrera wrote: > > Amit Kapila wrote: > > > I think this can happen if funnel->nextqueue is greater > > than funnel->nqueues. > > Please see if attached patch fixes the issue, else could you share the > > scenario in more detail where you hit this issue. > >

Re: [HACKERS] Parallel Seq Scan

On Mon, Mar 16, 2015 at 9:40 AM, Amit Langote wrote: > > On 13-03-2015 PM 11:03, Amit Kapila wrote: > > On Fri, Mar 13, 2015 at 7:15 PM, Robert Haas wrote: > >> > >> I don't think this is the right fix; the point of that code is to > >> remove a tuple queue from the funnel when it gets detached,

Re: [HACKERS] Parallel Seq Scan

On 13-03-2015 PM 11:03, Amit Kapila wrote: > On Fri, Mar 13, 2015 at 7:15 PM, Robert Haas wrote: >> >> I don't think this is the right fix; the point of that code is to >> remove a tuple queue from the funnel when it gets detached, which is a >> correct thing to want to do. funnel->nextqueue shou

Re: [HACKERS] Parallel Seq Scan

On Thu, Mar 12, 2015 at 9:50 PM, Thom Brown wrote: > > On 12 March 2015 at 15:29, Amit Kapila wrote: > > Please note that parallel_setup_cost and parallel_startup_cost are > > still set to zero by default, so you need to set it to higher values > > if you don't want the parallel plans once parall

Re: [HACKERS] Parallel Seq Scan

On Fri, Mar 13, 2015 at 11:03 PM, Amit Kapila wrote: > On Fri, Mar 13, 2015 at 7:15 PM, Robert Haas wrote: >> >> On Fri, Mar 13, 2015 at 7:01 AM, Amit Kapila >> wrote: >> > I think this can happen if funnel->nextqueue is greater than >> > funnel->nqueues. >> > Please see if attached patch fixes

Re: [HACKERS] Parallel Seq Scan

On Fri, Mar 13, 2015 at 7:15 PM, Robert Haas wrote: > > On Fri, Mar 13, 2015 at 7:01 AM, Amit Kapila wrote: > > I think this can happen if funnel->nextqueue is greater than > > funnel->nqueues. > > Please see if attached patch fixes the issue, else could you share the > > scenario in more detail

Re: [HACKERS] Parallel Seq Scan

On Fri, Mar 13, 2015 at 7:01 AM, Amit Kapila wrote: > I think this can happen if funnel->nextqueue is greater than > funnel->nqueues. > Please see if attached patch fixes the issue, else could you share the > scenario in more detail where you hit this issue. Speaking as the guy who wrote the firs

Re: [HACKERS] Parallel Seq Scan

On Fri, Mar 13, 2015 at 8:59 AM, Amit Kapila wrote: > We can't directly call DestroyParallelContext() to terminate workers as > it can so happen that by that time some of the workers are still not > started. That shouldn't be a problem. TerminateBackgroundWorker() not only kills an existing work

Re: [HACKERS] Parallel Seq Scan

On Fri, Mar 13, 2015 at 9:01 AM, Amit Kapila wrote: > On Thu, Mar 12, 2015 at 3:44 AM, Haribabu Kommi > wrote: >> In create_parallelscan_paths() function the funnel path is added once >> the partial seq scan >> path is generated. I feel the funnel path can be added once on top of >> the total pos

Re: [HACKERS] Parallel Seq Scan

On Thu, Mar 12, 2015 at 3:44 AM, Haribabu Kommi wrote: > > In create_parallelscan_paths() function the funnel path is added once > the partial seq scan > path is generated. I feel the funnel path can be added once on top of > the total possible > parallel path in the entire query path. > > Is this

Re: [HACKERS] Parallel Seq Scan

On Tue, Mar 10, 2015 at 12:26 PM, Amit Kapila wrote: > > On Tue, Mar 10, 2015 at 10:23 AM, Haribabu Kommi wrote: > > > > On Tue, Mar 10, 2015 at 3:09 PM, Amit Kapila wrote: > > > > > I have currently modelled it based on existing rescan for seqscan > > > (ExecReScanSeqScan()) which means it will

Re: [HACKERS] Parallel Seq Scan

On Thu, Mar 12, 2015 at 10:35 PM, Thom Brown wrote: > > > Another problem. I restarted the instance (just in case), and get this error: > > # \df+ *.* > ERROR: cannot retain locks acquired while in parallel mode > This problem occurs because above statement is trying to execute parallel_unsafe

Re: [HACKERS] Parallel Seq Scan

On Fri, Mar 13, 2015 at 2:12 PM, Amit Langote wrote: > > On 13-03-2015 PM 05:32, Amit Langote wrote: > > On 12-03-2015 PM 11:46, Amit Kapila wrote: > >> [parallel_seqscan_v10.patch] > > > > There may be a bug in TupleQueueFunnelNext(). > > > > 1) I observed a hang with stack looking like: > > > >

Re: [HACKERS] Parallel Seq Scan

On 13-03-2015 PM 05:32, Amit Langote wrote: > On 12-03-2015 PM 11:46, Amit Kapila wrote: >> [parallel_seqscan_v10.patch] > > There may be a bug in TupleQueueFunnelNext(). > > 1) I observed a hang with stack looking like: > > #0 0x0039696df098 in poll () from /lib64/libc.so.6 > #1 0x000

Re: [HACKERS] Parallel Seq Scan

On 12-03-2015 PM 11:46, Amit Kapila wrote: > [parallel_seqscan_v10.patch] There may be a bug in TupleQueueFunnelNext(). 1) I observed a hang with stack looking like: #0 0x0039696df098 in poll () from /lib64/libc.so.6 #1 0x006f1c6a in WaitLatchOrSocket (latch=0x7f29dc3c73b4, wakeEve

Re: [HACKERS] Parallel Seq Scan

On 13-03-2015 PM 01:37, Amit Langote wrote: > By the way, is it right that TupleQueueFunnel.queue has one shm_mq_handle per > initialized parallel worker? If so, how does TupleQueueFunnel.maxqueues relate > to ParallelContext.nworkers (of the corresponding parallel context)? > > Why I asked this i

Re: [HACKERS] Parallel Seq Scan

On 13-03-2015 AM 10:24, Amit Kapila wrote: > On Thu, Mar 12, 2015 at 4:22 PM, Amit Langote >> From Robert's description[1], it looked like the NestLoop with Funnel > would >> have Funnel as either outer plan or topmost plan node or NOT a > parameterised >> plan. In that case, would this case arise

Re: [HACKERS] Parallel Seq Scan

On Thu, Mar 12, 2015 at 4:22 PM, Amit Langote wrote: > > On 10-03-2015 PM 01:09, Amit Kapila wrote: > > On Tue, Mar 10, 2015 at 6:50 AM, Haribabu Kommi < kommi.harib...@gmail.com> > >> Is this patch handles the cases where the re-scan starts without > >> finishing the earlier scan? > >> > > > > Do

Re: [HACKERS] Parallel Seq Scan

On 12 March 2015 at 16:20, Thom Brown wrote: > On 12 March 2015 at 15:29, Amit Kapila wrote: >> On Thu, Mar 12, 2015 at 8:33 PM, Thom Brown wrote: >>> >>> On 12 March 2015 at 14:46, Amit Kapila wrote: >>> > One additional change (we need to SetLatch() in >>> > HandleParallelMessageInterrupt) >>

Re: [HACKERS] Parallel Seq Scan

On 12 March 2015 at 15:29, Amit Kapila wrote: > On Thu, Mar 12, 2015 at 8:33 PM, Thom Brown wrote: >> >> On 12 March 2015 at 14:46, Amit Kapila wrote: >> > One additional change (we need to SetLatch() in >> > HandleParallelMessageInterrupt) >> > is done to handle the hang issue reported on paral

Re: [HACKERS] Parallel Seq Scan

On Thu, Mar 12, 2015 at 8:33 PM, Thom Brown wrote: > > On 12 March 2015 at 14:46, Amit Kapila wrote: > > One additional change (we need to SetLatch() in > > HandleParallelMessageInterrupt) > > is done to handle the hang issue reported on parallel-mode thread. > > Without this change it is difficu

Re: [HACKERS] Parallel Seq Scan

On 12 March 2015 at 14:46, Amit Kapila wrote: > One additional change (we need to SetLatch() in > HandleParallelMessageInterrupt) > is done to handle the hang issue reported on parallel-mode thread. > Without this change it is difficult to verify the patch (will remove this > change > once new ver

Re: [HACKERS] Parallel Seq Scan

On 10-03-2015 PM 01:09, Amit Kapila wrote: > On Tue, Mar 10, 2015 at 6:50 AM, Haribabu Kommi >> Is this patch handles the cases where the re-scan starts without >> finishing the earlier scan? >> > > Do you mean to say cases like ANTI, SEMI Join (in nodeNestLoop.c) > where we scan the next outer t

Re: [HACKERS] Parallel Seq Scan

On Wed, Mar 11, 2015 at 6:31 AM, Robert Haas wrote: > On Tue, Mar 3, 2015 at 7:47 PM, Amit Kapila wrote: >> I have modified the patch to introduce a Funnel node (and left child >> as PartialSeqScan node). Apart from that, some other noticeable >> changes based on feedback include: >> a) Master b

<    1   2   3   4   5   >