On 30 September 2017 at 19:21, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Wed, Sep 20, 2017 at 10:59 AM, Amit Khandekar <amitdkhan...@gmail.com> > wrote: >> On 16 September 2017 at 10:42, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> >>> At a broader level, the idea is good, but I think it won't turn out >>> exactly like that considering your below paragraph which indicates >>> that it is okay if leader picks a partial path that is costly among >>> other partial paths as a leader won't be locked with that. >>> Considering this is a good design for parallel append, the question is >>> do we really need worker and leader to follow separate strategy for >>> choosing next path. I think the patch will be simpler if we can come >>> up with a way for the worker and leader to use the same strategy to >>> pick next path to process. How about we arrange the list of paths >>> such that first, all partial paths will be there and then non-partial >>> paths and probably both in decreasing order of cost. Now, both leader >>> and worker can start from the beginning of the list. In most cases, >>> the leader will start at the first partial path and will only ever >>> need to scan non-partial path if there is no other partial path left. >>> This is not bulletproof as it is possible that some worker starts >>> before leader in which case leader might scan non-partial path before >>> all partial paths are finished, but I think we can avoid that as well >>> if we are too worried about such cases. >> >> If there are no partial subpaths, then again the leader is likely to >> take up the expensive subpath. And this scenario would not be >> uncommon. >> > > While thinking about how common the case of no partial subpaths would > be, it occurred to me that as of now we always create a partial path > for the inheritance child if it is parallel-safe and the user has not > explicitly set the value of parallel_workers to zero (refer > compute_parallel_worker). So, unless you are planning to change that, > I think it will be quite uncommon to have no partial subpaths.
There are still some paths that can have non-partial paths cheaper than the partial paths. Also, there can be UNION ALL queries which could have non-partial subpaths. I guess this has already been discussed in the other replies. > > Few nitpicks in your latest patch: > 1. > @@ -298,6 +366,292 @@ ExecReScanAppend(AppendState *node) > if (subnode->chgParam == NULL) > ExecReScan(subnode); > } > + > > Looks like a spurious line. > > 2. > @@ -1285,7 +1291,11 @@ add_paths_to_append_rel(PlannerInfo *root, > RelOptInfo *rel, > .. > + if (chosen_path && chosen_path != cheapest_partial_path) > + pa_all_partial_subpaths = false; > > It will keep on setting pa_all_partial_subpaths as false for > non-partial paths which don't seem to be the purpose of this variable. > I think you want it to be set even when there is one non-partial path, > so isn't it better to write as below or something similar: > if (pa_nonpartial_subpaths && pa_all_partial_subpaths) > pa_all_partial_subpaths = false; Ok. How about removing pa_all_partial_subpaths altogether , and instead of the below condition : /* * If all the child rels have partial paths, and if the above Parallel * Append path has a mix of partial and non-partial subpaths, then consider * another Parallel Append path which will have *all* partial subpaths. * If enable_parallelappend is off, make this one non-parallel-aware. */ if (partial_subpaths_valid && !pa_all_partial_subpaths) ...... Use this condition : if (partial_subpaths_valid && pa_nonpartial_subpaths != NIL) ...... ---- Regarding a mix of partial and non-partial paths, I feel it always makes sense for the leader to choose the partial path. If it chooses a non-partial path, no other worker would be able to help finish that path. Among the partial paths, whether it chooses the cheapest one or expensive one does not matter, I think. We have the partial paths unordered. So whether it starts from the last partial path or the first partial path should not matter. Regarding scenario where all paths are non-partial, here is an e.g. : Suppose we have 4 child paths with costs : 10 5 5 3, and with 2 workers plus one leader. And suppose the leader takes additionally 1/4th of these costs to process the returned tuples. If leader takes least expensive one (3) : 2 workers will finish 10, 5, 5 in 10 units, and leader simultaneously chooses the plan with cost 3, and so it takes 3 + (1/4)(10 + 5 + 5 + 3) = 9 units. So the total time taken by Append is : 10. Whereas if leader takes most expensive one (10) : 10 + .25 (total) = 10 + 6 = 16 The 2 workers will finish 2nd, 3rd and 4th plan (5,5,3) in 8 units. and simultaneously leader will finish 1st plan (10) in 10 units, plus tuple processing cost i.e. 10 + (1/4)(10 + 5 + 5 + 3) = 15 units. So the total time taken by Append is : 15. -- Thanks, -Amit Khandekar EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
