[
https://issues.apache.org/jira/browse/DRILL-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551909#comment-16551909
]
ASF GitHub Bot commented on DRILL-6622:
---------------------------------------
sachouche opened a new pull request #1391: DRILL-6622: Fixed a
NullPointerException in a query with Union
URL: https://github.com/apache/drill/pull/1391
@Ben-Zvi, can you please review the fix?
Thanks!
There were two bugs in the Aggregator batch sizing logic:
Issue I
- The aggregator runs in a loop to consume all input batches
- The loop was updating the batch sizing stats after they were consumed
- Assume output-row-count is 1 and we receive a batch with at least 32k + 1
records
- The code would create 32k output batches (one per incoming record) and
then fails because of overflow
- Fix - Now updating the batch sizing logic when a non-empty batch is
received and before the processing loop
Issue II
- The Aggregator has two main modules: AggregatorBatch and Aggregator objects
- Both share the same "incoming" record batch instance
- Though there is logic to spill incoming batches when under pressure
- The batch sizing logic was not aware that when batches are spilled the
shared "incoming" object instance will diverge; that is, the Aggregator object
will mutate the incoming object
- The batch sizer was being invoked with a stale "incoming" object (the one
from the AggregatorBatch)
- Fix - Update the Aggregator code to always pass the active incoming
object explicitly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> UNION on tpcds sf100 tables hit SYSTEM ERROR: NullPointerException
> -------------------------------------------------------------------
>
> Key: DRILL-6622
> URL: https://issues.apache.org/jira/browse/DRILL-6622
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
> Affects Versions: 1.14.0
> Reporter: Vitalii Diravka
> Assignee: salim achouche
> Priority: Blocker
> Fix For: 1.14.0
>
> Attachments:
> MD4208_id_05_1_id_24b2a6f9-ed66-b97e-594d-f116cd3fdd23.json,
> MD4208_id_05_3_id_24b2ad9c-4568-a476-bbf6-2e17441078b1.json
>
>
> {code}
> SELECT c_customer_id FROM customer
> UNION
> SELECT ca_address_id FROM customer_address
> UNION
> SELECT cd_credit_rating FROM customer_demographics
> UNION
> SELECT hd_buy_potential FROM household_demographics
> UNION
> SELECT i_item_id FROM item
> UNION
> SELECT p_promo_id FROM promotion
> UNION
> SELECT t_time_id FROM time_dim
> UNION
> SELECT d_date_id FROM date_dim
> UNION
> SELECT s_store_id FROM store
> UNION
> SELECT w_warehouse_id FROM warehouse
> UNION
> SELECT sm_ship_mode_id FROM ship_mode
> UNION
> SELECT r_reason_id FROM reason
> UNION
> SELECT cc_call_center_id FROM call_center
> UNION
> SELECT web_site_id FROM web_site
> UNION
> SELECT wp_web_page_id FROM web_page
> UNION
> SELECT cp_catalog_page_id FROM catalog_page;
> {code}
> hit the following error:
> {code}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:96)
> ~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.HashTableGen3$BatchHolder.isKeyMatchInternalBuild(BatchHolder.java:171)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.isKeyMatch(HashTableTemplate.java:218)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.access$1000(HashTableTemplate.java:120)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate.put(HashTableTemplate.java:650)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.HashAggregatorGen0.checkGroupAndAggrValues(HashAggTemplate.java:1372)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.HashAggregatorGen0.doWork(HashAggTemplate.java:599)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:268)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.union.UnionAllRecordBatch$UnionInputIterator.next(UnionAllRecordBatch.java:381)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> {code}
> [~dechanggu] found that the issue is absent in Drill 1.13.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
