[
https://issues.apache.org/jira/browse/DRILL-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164606#comment-16164606
]
ASF GitHub Bot commented on DRILL-1162:
---------------------------------------
Github user vvysotskyi commented on the issue:
https://github.com/apache/drill/pull/905
@jinfengni, the idea is very interesting for me. More appropriate memory
cost estimation will cause to choosing better plans, that will cause the less
time execution and memory costs.
Writing about the comparing the ratio of column count vs ratio of row count
I meant that the multiplying of row count and column count does not help to
choose which input needs more memory in the case when the ratio of **actual**
row count is much greater than the ratio of column count.
`getCumulativeMemCost` implementation will not help for this case.
To show that, let's consider an example. We have three tables:
- a(has 5 columns and 5 rows with the same values);
- b(has 5 columns and 10 rows with the same values as in the table a);
- c(has 5 columns and 35 rows with the same values as in the table a).
For the query
```
select count(*) from a
inner join b on a.col1=b.col1
inner join c on b.col1=c.col1;
```
Drill will build plan
```
HahJoin[1]
/ \
c HahJoin[2]
/ \
b a
```
`getCumulativeMemCost` for HahJoin[1] will return a value proportional to
`(HahJoin[2] row count) * (HahJoin[2] column count) +
HahJoin[2].getCumulativeMemCost() = Max(aRowCount, bRowCount) * (aColumnCount +
bColumnCount) + aRowCount * aColumnCount = 5 * (5 + 5) + 5 * 5 = 75`.
Actual row count for build side of HahJoin[1] in this case is 50.
For the plan, that will be more suitable for this particular case:
```
HahJoin[1]
/ \
HahJoin[2] c
/ \
b a
```
`getCumulativeMemCost` for HahJoin[1] will return a value proportional to
`cRowCount * cColumnCount = 35 * 5 = 175`.
Actual row count for build side of HahJoin[1] in this case is 35.
Smal information about this pull request.
This pull request addresses only the case of large row count. It checks
that OOM may happen and if swap allows avoiding this potential OOM, the swap
will happen.
> 25 way join ended up with OOM
> -----------------------------
>
> Key: DRILL-1162
> URL: https://issues.apache.org/jira/browse/DRILL-1162
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow, Query Planning & Optimization
> Reporter: Rahul Challapalli
> Assignee: Volodymyr Vysotskyi
> Priority: Critical
> Fix For: Future
>
> Attachments: error.log, oom_error.log
>
>
> git.commit.id.abbrev=e5c2da0
> The below query results in 0 results being returned
> {code:sql}
> select count(*) from `lineitem1.parquet` a
> inner join `part.parquet` j on a.l_partkey = j.p_partkey
> inner join `orders.parquet` k on a.l_orderkey = k.o_orderkey
> inner join `supplier.parquet` l on a.l_suppkey = l.s_suppkey
> inner join `partsupp.parquet` m on j.p_partkey = m.ps_partkey and l.s_suppkey
> = m.ps_suppkey
> inner join `customer.parquet` n on k.o_custkey = n.c_custkey
> inner join `lineitem2.parquet` b on a.l_orderkey = b.l_orderkey
> inner join `lineitem2.parquet` c on a.l_partkey = c.l_partkey
> inner join `lineitem2.parquet` d on a.l_suppkey = d.l_suppkey
> inner join `lineitem2.parquet` e on a.l_extendedprice = e.l_extendedprice
> inner join `lineitem2.parquet` f on a.l_comment = f.l_comment
> inner join `lineitem2.parquet` g on a.l_shipdate = g.l_shipdate
> inner join `lineitem2.parquet` h on a.l_commitdate = h.l_commitdate
> inner join `lineitem2.parquet` i on a.l_receiptdate = i.l_receiptdate
> inner join `lineitem2.parquet` o on a.l_receiptdate = o.l_receiptdate
> inner join `lineitem2.parquet` p on a.l_receiptdate = p.l_receiptdate
> inner join `lineitem2.parquet` q on a.l_receiptdate = q.l_receiptdate
> inner join `lineitem2.parquet` r on a.l_receiptdate = r.l_receiptdate
> inner join `lineitem2.parquet` s on a.l_receiptdate = s.l_receiptdate
> inner join `lineitem2.parquet` t on a.l_receiptdate = t.l_receiptdate
> inner join `lineitem2.parquet` u on a.l_receiptdate = u.l_receiptdate
> inner join `lineitem2.parquet` v on a.l_receiptdate = v.l_receiptdate
> inner join `lineitem2.parquet` w on a.l_receiptdate = w.l_receiptdate
> inner join `lineitem2.parquet` x on a.l_receiptdate = x.l_receiptdate;
> {code}
> However when we remove the last 'inner join' and run the query it returns
> '716372534'. Since the last inner join is similar to the one's before it, it
> should match some records and return the data appropriately.
> The logs indicated that it actually returned 0 results. Attached the log file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
