[jira] [Commented] (DRILL-2362) Drill should manage Query Profiling archiving

[Rahul Raj (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146664#comment-16146664
 ] 

Rahul Raj commented on DRILL-2362:
--

We had more than 200,000 profiles stored in the default location and it made 
the profile page access extremely slow. Also while accessing this page, client 
connections were disconnected (channel closed exception) - could be related to 
a large GC causing missed heart beats.

-Rahul

> Drill should manage Query Profiling archiving
> -
>
> Key: DRILL-2362
> URL: https://issues.apache.org/jira/browse/DRILL-2362
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 0.7.0
>Reporter: Chris Westin
> Fix For: Future
>
>
> We collect query profile information for analysis purposes, but we keep it 
> forever. At this time, for a few queries, it isn't a problem. But as users 
> start putting Drill into production, automated use via other applications 
> will make this grow quickly. We need to come up with a retention policy 
> mechanism, with suitable settings administrators can use, and implement it so 
> that this data can be cleaned up.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146659#comment-16146659
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135974905
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
 ---
@@ -768,4 +765,73 @@ else if (exprHasPrefix && refHasPrefix) {
   }
 }
   }
+
+  /**
+   * handle FAST NONE specially when Project for query output. This 
happens when input returns a
+   * FAST NONE directly ( input does not return any batch with 
schema/data).
+   *
+   * Project operator has to return a batch with schema derived using the 
following 3 rules:
+   *  Case 1:  *  ==>  expand into an empty list of columns.
+   *  Case 2:  regular column reference ==> treat as nullable-int column
+   *  Case 3:  expressions => Call ExpressionTreeMaterialization over an 
empty vector contain.
+   *   Once the expression is materialized without error, use the 
output type of materialized
+   *   expression.
+   * The batch is constructed with the above rules, and recordCount = 0.
+   * Returned with OK_NEW_SCHEMA to down-stream operator.
+   */
+  @Override
+  protected IterOutcome handleFastNone() {
+if (! popConfig.isOutputProj()) {
+  return super.handleFastNone();
+}
+
+allocationVectors = new ArrayList<>();
+final List exprs = getExpressionList();
+final ErrorCollector collector = new ErrorCollectorImpl();
+VectorContainer fakedIncomingVC = new VectorContainer();
+
+for (NamedExpression namedExpression : exprs) {
+  if (namedExpression.getExpr() instanceof SchemaPath) {
+final NameSegment expr = ((SchemaPath) 
namedExpression.getExpr()).getRootSegment();
+if (expr.getPath().contains(StarColumnHelper.STAR_COLUMN)) {
--- End diff --

This code has been removed in revised patch.


> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146661#comment-16146661
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135974928
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
 ---
@@ -768,4 +765,73 @@ else if (exprHasPrefix && refHasPrefix) {
   }
 }
   }
+
+  /**
+   * handle FAST NONE specially when Project for query output. This 
happens when input returns a
+   * FAST NONE directly ( input does not return any batch with 
schema/data).
+   *
+   * Project operator has to return a batch with schema derived using the 
following 3 rules:
+   *  Case 1:  *  ==>  expand into an empty list of columns.
+   *  Case 2:  regular column reference ==> treat as nullable-int column
+   *  Case 3:  expressions => Call ExpressionTreeMaterialization over an 
empty vector contain.
+   *   Once the expression is materialized without error, use the 
output type of materialized
+   *   expression.
+   * The batch is constructed with the above rules, and recordCount = 0.
+   * Returned with OK_NEW_SCHEMA to down-stream operator.
+   */
+  @Override
+  protected IterOutcome handleFastNone() {
+if (! popConfig.isOutputProj()) {
+  return super.handleFastNone();
+}
+
+allocationVectors = new ArrayList<>();
+final List exprs = getExpressionList();
+final ErrorCollector collector = new ErrorCollectorImpl();
+VectorContainer fakedIncomingVC = new VectorContainer();
+
+for (NamedExpression namedExpression : exprs) {
+  if (namedExpression.getExpr() instanceof SchemaPath) {
+final NameSegment expr = ((SchemaPath) 
namedExpression.getExpr()).getRootSegment();
+if (expr.getPath().contains(StarColumnHelper.STAR_COLUMN)) {
+  continue; // * would expand into an empty list.
+} else {
+  final TypeProtos.MajorType majorType = 
TypeProtos.MajorType.newBuilder()
+  .setMinorType(MinorType.INT)
+  .setMode(TypeProtos.DataMode.OPTIONAL)
+  .build();
+
+  MaterializedField outputField = 
MaterializedField.create(namedExpression.getRef().getRootSegment().getPath(), 
majorType);
+  final ValueVector vv = container.addOrGet(outputField, callBack);
+  allocationVectors.add(vv);
+}
+continue;
+  }
+
+  final LogicalExpression materializedExpr = 
ExpressionTreeMaterializer.materialize(namedExpression.getExpr(),
+  fakedIncomingVC,
+  collector,
+  context.getFunctionRegistry(),
+  true,
+  unionTypeEnabled);
+
+  if (collector.hasErrors()) {
+throw new IllegalArgumentException(String.format("Failure while 
trying to materialize expressions : %s.  Errors:\n %s.",
+namedExpression.getExpr(),
+collector.toErrorString()));
+  }
--- End diff --

This code has been removed in revised patch.




> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146660#comment-16146660
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135974916
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
 ---
@@ -768,4 +765,73 @@ else if (exprHasPrefix && refHasPrefix) {
   }
 }
   }
+
+  /**
+   * handle FAST NONE specially when Project for query output. This 
happens when input returns a
+   * FAST NONE directly ( input does not return any batch with 
schema/data).
+   *
+   * Project operator has to return a batch with schema derived using the 
following 3 rules:
+   *  Case 1:  *  ==>  expand into an empty list of columns.
+   *  Case 2:  regular column reference ==> treat as nullable-int column
+   *  Case 3:  expressions => Call ExpressionTreeMaterialization over an 
empty vector contain.
+   *   Once the expression is materialized without error, use the 
output type of materialized
+   *   expression.
+   * The batch is constructed with the above rules, and recordCount = 0.
+   * Returned with OK_NEW_SCHEMA to down-stream operator.
+   */
+  @Override
+  protected IterOutcome handleFastNone() {
+if (! popConfig.isOutputProj()) {
+  return super.handleFastNone();
+}
+
+allocationVectors = new ArrayList<>();
+final List exprs = getExpressionList();
+final ErrorCollector collector = new ErrorCollectorImpl();
+VectorContainer fakedIncomingVC = new VectorContainer();
+
+for (NamedExpression namedExpression : exprs) {
+  if (namedExpression.getExpr() instanceof SchemaPath) {
+final NameSegment expr = ((SchemaPath) 
namedExpression.getExpr()).getRootSegment();
+if (expr.getPath().contains(StarColumnHelper.STAR_COLUMN)) {
+  continue; // * would expand into an empty list.
+} else {
+  final TypeProtos.MajorType majorType = 
TypeProtos.MajorType.newBuilder()
+  .setMinorType(MinorType.INT)
+  .setMode(TypeProtos.DataMode.OPTIONAL)
+  .build();
+
+  MaterializedField outputField = 
MaterializedField.create(namedExpression.getRef().getRootSegment().getPath(), 
majorType);
+  final ValueVector vv = container.addOrGet(outputField, callBack);
--- End diff --

This code has been removed in revised patch.


> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146651#comment-16146651
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135974302
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
 ---
@@ -768,4 +765,73 @@ else if (exprHasPrefix && refHasPrefix) {
   }
 }
   }
+
+  /**
+   * handle FAST NONE specially when Project for query output. This 
happens when input returns a
+   * FAST NONE directly ( input does not return any batch with 
schema/data).
+   *
+   * Project operator has to return a batch with schema derived using the 
following 3 rules:
+   *  Case 1:  *  ==>  expand into an empty list of columns.
+   *  Case 2:  regular column reference ==> treat as nullable-int column
+   *  Case 3:  expressions => Call ExpressionTreeMaterialization over an 
empty vector contain.
+   *   Once the expression is materialized without error, use the 
output type of materialized
+   *   expression.
+   * The batch is constructed with the above rules, and recordCount = 0.
+   * Returned with OK_NEW_SCHEMA to down-stream operator.
+   */
+  @Override
+  protected IterOutcome handleFastNone() {
+if (! popConfig.isOutputProj()) {
+  return super.handleFastNone();
+}
+
+allocationVectors = new ArrayList<>();
+final List exprs = getExpressionList();
+final ErrorCollector collector = new ErrorCollectorImpl();
--- End diff --

This code has been removed in revised patch. 


> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146652#comment-16146652
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135974310
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
 ---
@@ -768,4 +765,73 @@ else if (exprHasPrefix && refHasPrefix) {
   }
 }
   }
+
+  /**
+   * handle FAST NONE specially when Project for query output. This 
happens when input returns a
+   * FAST NONE directly ( input does not return any batch with 
schema/data).
+   *
+   * Project operator has to return a batch with schema derived using the 
following 3 rules:
+   *  Case 1:  *  ==>  expand into an empty list of columns.
+   *  Case 2:  regular column reference ==> treat as nullable-int column
+   *  Case 3:  expressions => Call ExpressionTreeMaterialization over an 
empty vector contain.
+   *   Once the expression is materialized without error, use the 
output type of materialized
+   *   expression.
+   * The batch is constructed with the above rules, and recordCount = 0.
+   * Returned with OK_NEW_SCHEMA to down-stream operator.
+   */
+  @Override
+  protected IterOutcome handleFastNone() {
+if (! popConfig.isOutputProj()) {
+  return super.handleFastNone();
+}
+
+allocationVectors = new ArrayList<>();
+final List exprs = getExpressionList();
+final ErrorCollector collector = new ErrorCollectorImpl();
+VectorContainer fakedIncomingVC = new VectorContainer();
+
+for (NamedExpression namedExpression : exprs) {
+  if (namedExpression.getExpr() instanceof SchemaPath) {
+final NameSegment expr = ((SchemaPath) 
namedExpression.getExpr()).getRootSegment();
+if (expr.getPath().contains(StarColumnHelper.STAR_COLUMN)) {
--- End diff --

This code has been removed in revised patch. 


> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146648#comment-16146648
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135974107
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
 ---
@@ -768,4 +765,73 @@ else if (exprHasPrefix && refHasPrefix) {
   }
 }
   }
+
+  /**
+   * handle FAST NONE specially when Project for query output. This 
happens when input returns a
+   * FAST NONE directly ( input does not return any batch with 
schema/data).
+   *
+   * Project operator has to return a batch with schema derived using the 
following 3 rules:
+   *  Case 1:  *  ==>  expand into an empty list of columns.
+   *  Case 2:  regular column reference ==> treat as nullable-int column
+   *  Case 3:  expressions => Call ExpressionTreeMaterialization over an 
empty vector contain.
--- End diff --

I managed to find a way to share the existing code path. With the revised 
patch, both of the first scenario and second one would call same method to 
setup a new schema based on incoming batch. The only difference for the second 
scenario is we have to simulate a record batch with empty vector container, as 
the code for set up new schema requires it.  That is done by leveraging an 
existing class (with some extension). 



> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146649#comment-16146649
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135974133
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
 ---
@@ -768,4 +765,73 @@ else if (exprHasPrefix && refHasPrefix) {
   }
 }
   }
+
+  /**
+   * handle FAST NONE specially when Project for query output. This 
happens when input returns a
+   * FAST NONE directly ( input does not return any batch with 
schema/data).
+   *
+   * Project operator has to return a batch with schema derived using the 
following 3 rules:
+   *  Case 1:  *  ==>  expand into an empty list of columns.
+   *  Case 2:  regular column reference ==> treat as nullable-int column
+   *  Case 3:  expressions => Call ExpressionTreeMaterialization over an 
empty vector contain.
--- End diff --

Done. 


> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-5732) Unable to allocate sv2 for 9039 records, and not enough batchGroups to spill.

[Robert Hou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Hou resolved DRILL-5732.
---
Resolution: Not A Problem

> Unable to allocate sv2 for 9039 records, and not enough batchGroups to spill.
> -
>
> Key: DRILL-5732
> URL: https://issues.apache.org/jira/browse/DRILL-5732
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Robert Hou
>Assignee: Paul Rogers
> Attachments: 26621eb2-daec-cef9-efed-5986e72a750a.sys.drill, 
> drillbit.log.83
>
>
> git commit id:
> {noformat}
> | 1.12.0-SNAPSHOT  | e9065b55ea560e7f737d6fcb4948f9e945b9b14f  | DRILL-5660: 
> Parquet metadata caching improvements  | 15.08.2017 @ 09:31:00 PDT  | 
> r...@qa-node190.qa.lab  | 15.08.2017 @ 13:29:26 PDT  |
> {noformat}
> Query is:
> {noformat}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.disable_exchanges` = true;
> alter session set `planner.memory.max_query_memory_per_node` = 104857600;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.width.max_per_query` = 1;
> select max(col1), max(cs_sold_date_sk), max(cs_sold_time_sk), 
> max(cs_ship_date_sk), max(cs_bill_customer_sk), max(cs_bill_cdemo_sk), 
> max(cs_bill_hdemo_sk), max(cs_bill_addr_sk), max(cs_ship_customer_sk), 
> max(cs_ship_cdemo_sk), max(cs_ship_hdemo_sk), max(cs_ship_addr_sk), 
> max(cs_call_center_sk), max(cs_catalog_page_sk), max(cs_ship_mode_sk), 
> min(cs_warehouse_sk), max(cs_item_sk), max(cs_promo_sk), 
> max(cs_order_number), max(cs_quantity), max(cs_wholesale_cost), 
> max(cs_list_price), max(cs_sales_price), max(cs_ext_discount_amt), 
> min(cs_ext_sales_price), max(cs_ext_wholesale_cost), min(cs_ext_list_price), 
> min(cs_ext_tax), min(cs_coupon_amt), max(cs_ext_ship_cost), max(cs_net_paid), 
> max(cs_net_paid_inc_tax), min(cs_net_paid_inc_ship), 
> min(cs_net_paid_inc_ship_tax), min(cs_net_profit), min(c_customer_sk), 
> min(length(c_customer_id)), max(c_current_cdemo_sk), max(c_current_hdemo_sk), 
> min(c_current_addr_sk), min(c_first_shipto_date_sk), 
> min(c_first_sales_date_sk), min(length(c_salutation)), 
> min(length(c_first_name)), min(length(c_last_name)), 
> min(length(c_preferred_cust_flag)), max(c_birth_day), min(c_birth_month), 
> min(c_birth_year), max(c_last_review_date), c_email_address  from (select 
> cs_sold_date_sk+cs_sold_time_sk col1, * from 
> dfs.`/drill/testdata/resource-manager/md1362` order by c_email_address nulls 
> first) d where d.col1 > 2536816 and c_email_address is not null group by 
> c_email_address;
> ALTER SESSION SET `exec.sort.disable_managed` = true;
> alter session set `planner.disable_exchanges` = false;
> alter session set `planner.memory.max_query_memory_per_node` = 2147483648;
> alter session set `planner.width.max_per_node` = 17;
> alter session set `planner.width.max_per_query` = 1000;
> {noformat}
> Here is the stack trace:
> {noformat}
> 2017-08-18 13:15:27,052 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] DEBUG 
> o.a.d.e.t.g.SingleBatchSorterGen27 - Took 6445 us to sort 9039 records
> 2017-08-18 13:15:27,420 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] DEBUG 
> o.a.d.e.p.i.xsort.ExternalSortBatch - Copier allocator current allocation 0
> 2017-08-18 13:15:27,420 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] DEBUG 
> o.a.d.e.p.i.xsort.ExternalSortBatch - mergeAndSpill: starting total size in 
> memory = 71964288
> 2017-08-18 13:15:27,421 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] INFO  
> o.a.d.e.p.i.xsort.ExternalSortBatch - User Error Occurred: One or more nodes 
> ran out of memory while executing the query.
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
> nodes ran out of memory while executing the query.
> Unable to allocate sv2 for 9039 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 71964288
> allocator limit 52428800
> [Error Id: 7b248f12-2b31-4013-86b6-92e6c842db48 ]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
>  ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.newSV2(ExternalSortBatch.java:637)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:379)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> 

[jira] [Closed] (DRILL-5732) Unable to allocate sv2 for 9039 records, and not enough batchGroups to spill.

[Robert Hou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Hou closed DRILL-5732.
-

I have verified that this query completes.

> Unable to allocate sv2 for 9039 records, and not enough batchGroups to spill.
> -
>
> Key: DRILL-5732
> URL: https://issues.apache.org/jira/browse/DRILL-5732
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Robert Hou
>Assignee: Paul Rogers
> Attachments: 26621eb2-daec-cef9-efed-5986e72a750a.sys.drill, 
> drillbit.log.83
>
>
> git commit id:
> {noformat}
> | 1.12.0-SNAPSHOT  | e9065b55ea560e7f737d6fcb4948f9e945b9b14f  | DRILL-5660: 
> Parquet metadata caching improvements  | 15.08.2017 @ 09:31:00 PDT  | 
> r...@qa-node190.qa.lab  | 15.08.2017 @ 13:29:26 PDT  |
> {noformat}
> Query is:
> {noformat}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.disable_exchanges` = true;
> alter session set `planner.memory.max_query_memory_per_node` = 104857600;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.width.max_per_query` = 1;
> select max(col1), max(cs_sold_date_sk), max(cs_sold_time_sk), 
> max(cs_ship_date_sk), max(cs_bill_customer_sk), max(cs_bill_cdemo_sk), 
> max(cs_bill_hdemo_sk), max(cs_bill_addr_sk), max(cs_ship_customer_sk), 
> max(cs_ship_cdemo_sk), max(cs_ship_hdemo_sk), max(cs_ship_addr_sk), 
> max(cs_call_center_sk), max(cs_catalog_page_sk), max(cs_ship_mode_sk), 
> min(cs_warehouse_sk), max(cs_item_sk), max(cs_promo_sk), 
> max(cs_order_number), max(cs_quantity), max(cs_wholesale_cost), 
> max(cs_list_price), max(cs_sales_price), max(cs_ext_discount_amt), 
> min(cs_ext_sales_price), max(cs_ext_wholesale_cost), min(cs_ext_list_price), 
> min(cs_ext_tax), min(cs_coupon_amt), max(cs_ext_ship_cost), max(cs_net_paid), 
> max(cs_net_paid_inc_tax), min(cs_net_paid_inc_ship), 
> min(cs_net_paid_inc_ship_tax), min(cs_net_profit), min(c_customer_sk), 
> min(length(c_customer_id)), max(c_current_cdemo_sk), max(c_current_hdemo_sk), 
> min(c_current_addr_sk), min(c_first_shipto_date_sk), 
> min(c_first_sales_date_sk), min(length(c_salutation)), 
> min(length(c_first_name)), min(length(c_last_name)), 
> min(length(c_preferred_cust_flag)), max(c_birth_day), min(c_birth_month), 
> min(c_birth_year), max(c_last_review_date), c_email_address  from (select 
> cs_sold_date_sk+cs_sold_time_sk col1, * from 
> dfs.`/drill/testdata/resource-manager/md1362` order by c_email_address nulls 
> first) d where d.col1 > 2536816 and c_email_address is not null group by 
> c_email_address;
> ALTER SESSION SET `exec.sort.disable_managed` = true;
> alter session set `planner.disable_exchanges` = false;
> alter session set `planner.memory.max_query_memory_per_node` = 2147483648;
> alter session set `planner.width.max_per_node` = 17;
> alter session set `planner.width.max_per_query` = 1000;
> {noformat}
> Here is the stack trace:
> {noformat}
> 2017-08-18 13:15:27,052 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] DEBUG 
> o.a.d.e.t.g.SingleBatchSorterGen27 - Took 6445 us to sort 9039 records
> 2017-08-18 13:15:27,420 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] DEBUG 
> o.a.d.e.p.i.xsort.ExternalSortBatch - Copier allocator current allocation 0
> 2017-08-18 13:15:27,420 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] DEBUG 
> o.a.d.e.p.i.xsort.ExternalSortBatch - mergeAndSpill: starting total size in 
> memory = 71964288
> 2017-08-18 13:15:27,421 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] INFO  
> o.a.d.e.p.i.xsort.ExternalSortBatch - User Error Occurred: One or more nodes 
> ran out of memory while executing the query.
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
> nodes ran out of memory while executing the query.
> Unable to allocate sv2 for 9039 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 71964288
> allocator limit 52428800
> [Error Id: 7b248f12-2b31-4013-86b6-92e6c842db48 ]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
>  ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.newSV2(ExternalSortBatch.java:637)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:379)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> 

[jira] [Commented] (DRILL-5732) Unable to allocate sv2 for 9039 records, and not enough batchGroups to spill.

[Robert Hou (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146479#comment-16146479
 ] 

Robert Hou commented on DRILL-5732:
---

This query now completes.

> Unable to allocate sv2 for 9039 records, and not enough batchGroups to spill.
> -
>
> Key: DRILL-5732
> URL: https://issues.apache.org/jira/browse/DRILL-5732
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Robert Hou
>Assignee: Paul Rogers
> Attachments: 26621eb2-daec-cef9-efed-5986e72a750a.sys.drill, 
> drillbit.log.83
>
>
> git commit id:
> {noformat}
> | 1.12.0-SNAPSHOT  | e9065b55ea560e7f737d6fcb4948f9e945b9b14f  | DRILL-5660: 
> Parquet metadata caching improvements  | 15.08.2017 @ 09:31:00 PDT  | 
> r...@qa-node190.qa.lab  | 15.08.2017 @ 13:29:26 PDT  |
> {noformat}
> Query is:
> {noformat}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.disable_exchanges` = true;
> alter session set `planner.memory.max_query_memory_per_node` = 104857600;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.width.max_per_query` = 1;
> select max(col1), max(cs_sold_date_sk), max(cs_sold_time_sk), 
> max(cs_ship_date_sk), max(cs_bill_customer_sk), max(cs_bill_cdemo_sk), 
> max(cs_bill_hdemo_sk), max(cs_bill_addr_sk), max(cs_ship_customer_sk), 
> max(cs_ship_cdemo_sk), max(cs_ship_hdemo_sk), max(cs_ship_addr_sk), 
> max(cs_call_center_sk), max(cs_catalog_page_sk), max(cs_ship_mode_sk), 
> min(cs_warehouse_sk), max(cs_item_sk), max(cs_promo_sk), 
> max(cs_order_number), max(cs_quantity), max(cs_wholesale_cost), 
> max(cs_list_price), max(cs_sales_price), max(cs_ext_discount_amt), 
> min(cs_ext_sales_price), max(cs_ext_wholesale_cost), min(cs_ext_list_price), 
> min(cs_ext_tax), min(cs_coupon_amt), max(cs_ext_ship_cost), max(cs_net_paid), 
> max(cs_net_paid_inc_tax), min(cs_net_paid_inc_ship), 
> min(cs_net_paid_inc_ship_tax), min(cs_net_profit), min(c_customer_sk), 
> min(length(c_customer_id)), max(c_current_cdemo_sk), max(c_current_hdemo_sk), 
> min(c_current_addr_sk), min(c_first_shipto_date_sk), 
> min(c_first_sales_date_sk), min(length(c_salutation)), 
> min(length(c_first_name)), min(length(c_last_name)), 
> min(length(c_preferred_cust_flag)), max(c_birth_day), min(c_birth_month), 
> min(c_birth_year), max(c_last_review_date), c_email_address  from (select 
> cs_sold_date_sk+cs_sold_time_sk col1, * from 
> dfs.`/drill/testdata/resource-manager/md1362` order by c_email_address nulls 
> first) d where d.col1 > 2536816 and c_email_address is not null group by 
> c_email_address;
> ALTER SESSION SET `exec.sort.disable_managed` = true;
> alter session set `planner.disable_exchanges` = false;
> alter session set `planner.memory.max_query_memory_per_node` = 2147483648;
> alter session set `planner.width.max_per_node` = 17;
> alter session set `planner.width.max_per_query` = 1000;
> {noformat}
> Here is the stack trace:
> {noformat}
> 2017-08-18 13:15:27,052 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] DEBUG 
> o.a.d.e.t.g.SingleBatchSorterGen27 - Took 6445 us to sort 9039 records
> 2017-08-18 13:15:27,420 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] DEBUG 
> o.a.d.e.p.i.xsort.ExternalSortBatch - Copier allocator current allocation 0
> 2017-08-18 13:15:27,420 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] DEBUG 
> o.a.d.e.p.i.xsort.ExternalSortBatch - mergeAndSpill: starting total size in 
> memory = 71964288
> 2017-08-18 13:15:27,421 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] INFO  
> o.a.d.e.p.i.xsort.ExternalSortBatch - User Error Occurred: One or more nodes 
> ran out of memory while executing the query.
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
> nodes ran out of memory while executing the query.
> Unable to allocate sv2 for 9039 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 71964288
> allocator limit 52428800
> [Error Id: 7b248f12-2b31-4013-86b6-92e6c842db48 ]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
>  ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.newSV2(ExternalSortBatch.java:637)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:379)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> 

[jira] [Updated] (DRILL-5749) Foreman and Netty threads occure deadlock

[weijie.tong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

weijie.tong updated DRILL-5749:
---
Reviewer: Paul Rogers

> Foreman and Netty threads occure deadlock 
> --
>
> Key: DRILL-5749
> URL: https://issues.apache.org/jira/browse/DRILL-5749
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC
>Affects Versions: 1.10.0, 1.11.0
>Reporter: weijie.tong
>Priority: Critical
>
> when the cluster was in high concurrency query and the reused control 
> connection occured exceptoin, the foreman and netty threads both try to 
> acquire each other's lock then deadlock occured.  The netty thread hold the 
> map (RequestIdMap) lock then try to acquire the ReconnectingConnection lock 
> to send command, while the foreman thread hold the ReconnectingConnection 
> lock then try to acquire the RequestIdMap lock. So the deadlock happend.
> Below is the jstack dump:
> Found one Java-level deadlock:
> =
> "265aa5cb-e5e2-39ed-9c2f-7658b905372e:foreman":
>   waiting to lock monitor 0x7f935b721f48 (object 0x000656affc40, a 
> org.apache.drill.exec.rpc.control.ControlConnectionManager),
>   which is held by "265aa82f-d8c1-5df0-9946-003a4990db7e:foreman"
> "265aa82f-d8c1-5df0-9946-003a4990db7e:foreman":
>   waiting to lock monitor 0x7f90de3b9648 (object 0x0006b524d7e8, a 
> com.carrotsearch.hppc.IntObjectHashMap),
>   which is held by "BitServer-2"
> "BitServer-2":
>   waiting to lock monitor 0x7f935b721f48 (object 0x000656affc40, a 
> org.apache.drill.exec.rpc.control.ControlConnectionManager),
>   which is held by "265aa82f-d8c1-5df0-9946-003a4990db7e:foreman"
> Java stack information for the threads listed above:
> ===
> "265aa5cb-e5e2-39ed-9c2f-7658b905372e:foreman":
>   at 
> org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:72)
>   - waiting to lock <0x000656affc40> (a 
> org.apache.drill.exec.rpc.control.ControlConnectionManager)
>   at 
> org.apache.drill.exec.rpc.control.ControlTunnel.sendFragments(ControlTunnel.java:66)
>   at 
> org.apache.drill.exec.work.foreman.Foreman.sendRemoteFragments(Foreman.java:1210)
>   at 
> org.apache.drill.exec.work.foreman.Foreman.setupNonRootFragments(Foreman.java:1141)
>   at 
> org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:454)
>   at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1045)
>   at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
>   at java.lang.Thread.run(Thread.java:849)
> "265aa82f-d8c1-5df0-9946-003a4990db7e:foreman":
>   at 
> org.apache.drill.exec.rpc.RequestIdMap.createNewRpcListener(RequestIdMap.java:87)
>   - waiting to lock <0x0006b524d7e8> (a 
> com.carrotsearch.hppc.IntObjectHashMap)
>   at 
> org.apache.drill.exec.rpc.AbstractRemoteConnection.createNewRpcListener(AbstractRemoteConnection.java:153)
>   at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:115)
>   at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:89)
>   at 
> org.apache.drill.exec.rpc.control.ControlConnection.send(ControlConnection.java:65)
>   at 
> org.apache.drill.exec.rpc.control.ControlTunnel$SendFragment.doRpcCall(ControlTunnel.java:160)
>   at 
> org.apache.drill.exec.rpc.control.ControlTunnel$SendFragment.doRpcCall(ControlTunnel.java:150)
>   at 
> org.apache.drill.exec.rpc.ListeningCommand.connectionAvailable(ListeningCommand.java:38)
>   at 
> org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:75)
>   - locked <0x000656affc40> (a 
> org.apache.drill.exec.rpc.control.ControlConnectionManager)
>   at 
> org.apache.drill.exec.rpc.control.ControlTunnel.sendFragments(ControlTunnel.java:66)
>   at 
> org.apache.drill.exec.work.foreman.Foreman.sendRemoteFragments(Foreman.java:1210)
>   at 
> org.apache.drill.exec.work.foreman.Foreman.setupNonRootFragments(Foreman.java:1141)
>   at 
> org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:454)
>   at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1045)
>   at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
>   at java.lang.Thread.run(Thread.java:849)
> "BitServer-2":
>   at 
> 

[jira] [Commented] (DRILL-5721) Query with only root fragment and no non-root fragment hangs when Drillbit to Drillbit Control Connection has network issues

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146443#comment-16146443
 ] 

ASF GitHub Bot commented on DRILL-5721:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/919#discussion_r135949946
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -1109,7 +1102,7 @@ private void setupNonRootFragments(final 
Collection fragments) thr
   // nothing to do here
   return;
 }
-/*
+/*z
--- End diff --

It was part for refactor commit only and was later removed in next commit.


> Query with only root fragment and no non-root fragment hangs when Drillbit to 
> Drillbit Control Connection has network issues
> 
>
> Key: DRILL-5721
> URL: https://issues.apache.org/jira/browse/DRILL-5721
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
> Fix For: 1.12.0
>
>
> Recently I found an issue (Thanks to [~knguyen] to create this scenario) 
> related to Fragment Status reporting and would like some feedback on it. 
> When a client submits a query to Foreman, then it is planned by Foreman and 
> later fragments are scheduled to root and non-root nodes. Foreman creates a 
> DriilbitStatusListener and FragmentStatusListener to know about the health of 
> Drillbit node and a fragment respectively. The way root and non-root 
> fragments are setup by Foreman are different: 
> Root fragments are setup without any communication over control channel 
> (since it is executed locally on Foreman)
> Non-root fragments are setup by sending control message 
> (REQ_INITIALIZE_FRAGMENTS_VALUE) over wire. If there is failure in sending 
> any such control message (like due to network hiccup's) during query setup 
> then the query is failed and client is notified. 
> Each fragment is executed on it's node with the help Fragment Executor which 
> has an instance for FragmentStatusReporter. FragmentStatusReporter helps to 
> update the status of a fragment to Foreman node over a control tunnel or 
> connection using RPC message (REQ_FRAGMENT_STATUS) both for root and non-root 
> fragments. 
> Based on above when root fragment is submitted for setup then it is done 
> locally without any RPC communication whereas when status for that fragment 
> is reported by fragment executor that happens over control connection by 
> sending a RPC message. But for non-root fragment setup and status update both 
> happens using RPC message over control connection.
> *Issue 1:*
> What was observed is if for a simple query which has only 1 root fragment 
> running on Foreman node then setup will work fine. But as part of status 
> update when the fragment tries to create a control connection and fails to 
> establish that, then the query hangs. This is because the root fragment will 
> complete execution but will fail to update Foreman about it and Foreman think 
> that the query is running for ever. 
> *Proposed Solution:*
> For root fragment the setup of fragment is happening locally without RPC 
> message, so we can do the same for status update of root fragments. This will 
> avoid RPC communication for status update of fragments running locally on the 
> foreman and hence will resolve issue 1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5721) Query with only root fragment and no non-root fragment hangs when Drillbit to Drillbit Control Connection has network issues

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146428#comment-16146428
 ] 

ASF GitHub Bot commented on DRILL-5721:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/919#discussion_r135948567
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -1073,26 +1070,22 @@ public QueryId getQueryId() {
*/
   private void setupRootFragment(final PlanFragment rootFragment, final 
FragmentRoot rootOperator)
   throws ExecutionSetupException {
-@SuppressWarnings("resource")
 final FragmentContext rootContext = new 
FragmentContext(drillbitContext, rootFragment, queryContext,
 initiatingClient, 
drillbitContext.getFunctionImplementationRegistry());
-@SuppressWarnings("resource")
-final IncomingBuffers buffers = new IncomingBuffers(rootFragment, 
rootContext);
-rootContext.setBuffers(buffers);
-
-queryManager.addFragmentStatusTracker(rootFragment, true);
-
 final ControlTunnel tunnel = 
drillbitContext.getController().getTunnel(queryContext.getCurrentEndpoint());
+final FragmentStatusReporter statusReporter = new 
FragmentStatusReporter(rootContext, tunnel);
 final FragmentExecutor rootRunner = new FragmentExecutor(rootContext, 
rootFragment,
-new FragmentStatusReporter(rootContext, tunnel),
-rootOperator);
-final RootFragmentManager fragmentManager = new 
RootFragmentManager(rootFragment.getHandle(), buffers, rootRunner);
+statusReporter, rootOperator);
 
-if (buffers.isDone()) {
+queryManager.addFragmentStatusTracker(rootFragment, true);
+
+// FragmentManager is setting buffer for FragmentContext
+if (rootContext.isBuffersDone()) {
--- End diff --

Inside AbstractFragmentManager constructor, since that's where the Incoming 
Buffers are created.


https://github.com/apache/drill/pull/919/files#diff-d7418a1fd07314f268049ae9f07b7ed3

```
+  public AbstractFragmentManager(final PlanFragment fragment, final 
FragmentExecutor executor, final FragmentStatusReporter statusReporter, final 
FragmentRoot rootOperator) {
 +this.fragmentHandle = fragment.getHandle();
 +this.fragmentContext = executor.getContext();
 +this.buffers = new IncomingBuffers(fragment, fragmentContext);
 +this.fragmentContext.setBuffers(buffers);
```


> Query with only root fragment and no non-root fragment hangs when Drillbit to 
> Drillbit Control Connection has network issues
> 
>
> Key: DRILL-5721
> URL: https://issues.apache.org/jira/browse/DRILL-5721
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
> Fix For: 1.12.0
>
>
> Recently I found an issue (Thanks to [~knguyen] to create this scenario) 
> related to Fragment Status reporting and would like some feedback on it. 
> When a client submits a query to Foreman, then it is planned by Foreman and 
> later fragments are scheduled to root and non-root nodes. Foreman creates a 
> DriilbitStatusListener and FragmentStatusListener to know about the health of 
> Drillbit node and a fragment respectively. The way root and non-root 
> fragments are setup by Foreman are different: 
> Root fragments are setup without any communication over control channel 
> (since it is executed locally on Foreman)
> Non-root fragments are setup by sending control message 
> (REQ_INITIALIZE_FRAGMENTS_VALUE) over wire. If there is failure in sending 
> any such control message (like due to network hiccup's) during query setup 
> then the query is failed and client is notified. 
> Each fragment is executed on it's node with the help Fragment Executor which 
> has an instance for FragmentStatusReporter. FragmentStatusReporter helps to 
> update the status of a fragment to Foreman node over a control tunnel or 
> connection using RPC message (REQ_FRAGMENT_STATUS) both for root and non-root 
> fragments. 
> Based on above when root fragment is submitted for setup then it is done 
> locally without any RPC communication whereas when status for that fragment 
> is reported by fragment executor that happens over control connection by 
> sending a RPC message. But for non-root fragment setup and status update both 
> happens using RPC message over control connection.
> *Issue 1:*
> What was observed is if for a simple query which has only 1 root fragment 
> running on Foreman node then setup will work fine. But as part of status 
> update when the fragment tries to create a control connection 

[jira] [Commented] (DRILL-5721) Query with only root fragment and no non-root fragment hangs when Drillbit to Drillbit Control Connection has network issues

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146418#comment-16146418
 ] 

ASF GitHub Bot commented on DRILL-5721:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/919#discussion_r135937508
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -1109,7 +1102,7 @@ private void setupNonRootFragments(final 
Collection fragments) thr
   // nothing to do here
   return;
 }
-/*
+/*z
--- End diff --

extra character


> Query with only root fragment and no non-root fragment hangs when Drillbit to 
> Drillbit Control Connection has network issues
> 
>
> Key: DRILL-5721
> URL: https://issues.apache.org/jira/browse/DRILL-5721
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
> Fix For: 1.12.0
>
>
> Recently I found an issue (Thanks to [~knguyen] to create this scenario) 
> related to Fragment Status reporting and would like some feedback on it. 
> When a client submits a query to Foreman, then it is planned by Foreman and 
> later fragments are scheduled to root and non-root nodes. Foreman creates a 
> DriilbitStatusListener and FragmentStatusListener to know about the health of 
> Drillbit node and a fragment respectively. The way root and non-root 
> fragments are setup by Foreman are different: 
> Root fragments are setup without any communication over control channel 
> (since it is executed locally on Foreman)
> Non-root fragments are setup by sending control message 
> (REQ_INITIALIZE_FRAGMENTS_VALUE) over wire. If there is failure in sending 
> any such control message (like due to network hiccup's) during query setup 
> then the query is failed and client is notified. 
> Each fragment is executed on it's node with the help Fragment Executor which 
> has an instance for FragmentStatusReporter. FragmentStatusReporter helps to 
> update the status of a fragment to Foreman node over a control tunnel or 
> connection using RPC message (REQ_FRAGMENT_STATUS) both for root and non-root 
> fragments. 
> Based on above when root fragment is submitted for setup then it is done 
> locally without any RPC communication whereas when status for that fragment 
> is reported by fragment executor that happens over control connection by 
> sending a RPC message. But for non-root fragment setup and status update both 
> happens using RPC message over control connection.
> *Issue 1:*
> What was observed is if for a simple query which has only 1 root fragment 
> running on Foreman node then setup will work fine. But as part of status 
> update when the fragment tries to create a control connection and fails to 
> establish that, then the query hangs. This is because the root fragment will 
> complete execution but will fail to update Foreman about it and Foreman think 
> that the query is running for ever. 
> *Proposed Solution:*
> For root fragment the setup of fragment is happening locally without RPC 
> message, so we can do the same for status update of root fragments. This will 
> avoid RPC communication for status update of fragments running locally on the 
> foreman and hence will resolve issue 1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5721) Query with only root fragment and no non-root fragment hangs when Drillbit to Drillbit Control Connection has network issues

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146417#comment-16146417
 ] 

ASF GitHub Bot commented on DRILL-5721:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/919#discussion_r135937433
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -1073,26 +1070,22 @@ public QueryId getQueryId() {
*/
   private void setupRootFragment(final PlanFragment rootFragment, final 
FragmentRoot rootOperator)
   throws ExecutionSetupException {
-@SuppressWarnings("resource")
 final FragmentContext rootContext = new 
FragmentContext(drillbitContext, rootFragment, queryContext,
 initiatingClient, 
drillbitContext.getFunctionImplementationRegistry());
-@SuppressWarnings("resource")
-final IncomingBuffers buffers = new IncomingBuffers(rootFragment, 
rootContext);
-rootContext.setBuffers(buffers);
-
-queryManager.addFragmentStatusTracker(rootFragment, true);
-
 final ControlTunnel tunnel = 
drillbitContext.getController().getTunnel(queryContext.getCurrentEndpoint());
+final FragmentStatusReporter statusReporter = new 
FragmentStatusReporter(rootContext, tunnel);
 final FragmentExecutor rootRunner = new FragmentExecutor(rootContext, 
rootFragment,
-new FragmentStatusReporter(rootContext, tunnel),
-rootOperator);
-final RootFragmentManager fragmentManager = new 
RootFragmentManager(rootFragment.getHandle(), buffers, rootRunner);
+statusReporter, rootOperator);
 
-if (buffers.isDone()) {
+queryManager.addFragmentStatusTracker(rootFragment, true);
+
+// FragmentManager is setting buffer for FragmentContext
+if (rootContext.isBuffersDone()) {
--- End diff --

I don't see where rootContext.setBuffers is being called to set the buffers


> Query with only root fragment and no non-root fragment hangs when Drillbit to 
> Drillbit Control Connection has network issues
> 
>
> Key: DRILL-5721
> URL: https://issues.apache.org/jira/browse/DRILL-5721
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
> Fix For: 1.12.0
>
>
> Recently I found an issue (Thanks to [~knguyen] to create this scenario) 
> related to Fragment Status reporting and would like some feedback on it. 
> When a client submits a query to Foreman, then it is planned by Foreman and 
> later fragments are scheduled to root and non-root nodes. Foreman creates a 
> DriilbitStatusListener and FragmentStatusListener to know about the health of 
> Drillbit node and a fragment respectively. The way root and non-root 
> fragments are setup by Foreman are different: 
> Root fragments are setup without any communication over control channel 
> (since it is executed locally on Foreman)
> Non-root fragments are setup by sending control message 
> (REQ_INITIALIZE_FRAGMENTS_VALUE) over wire. If there is failure in sending 
> any such control message (like due to network hiccup's) during query setup 
> then the query is failed and client is notified. 
> Each fragment is executed on it's node with the help Fragment Executor which 
> has an instance for FragmentStatusReporter. FragmentStatusReporter helps to 
> update the status of a fragment to Foreman node over a control tunnel or 
> connection using RPC message (REQ_FRAGMENT_STATUS) both for root and non-root 
> fragments. 
> Based on above when root fragment is submitted for setup then it is done 
> locally without any RPC communication whereas when status for that fragment 
> is reported by fragment executor that happens over control connection by 
> sending a RPC message. But for non-root fragment setup and status update both 
> happens using RPC message over control connection.
> *Issue 1:*
> What was observed is if for a simple query which has only 1 root fragment 
> running on Foreman node then setup will work fine. But as part of status 
> update when the fragment tries to create a control connection and fails to 
> establish that, then the query hangs. This is because the root fragment will 
> complete execution but will fail to update Foreman about it and Foreman think 
> that the query is running for ever. 
> *Proposed Solution:*
> For root fragment the setup of fragment is happening locally without RPC 
> message, so we can do the same for status update of root fragments. This will 
> avoid RPC communication for status update of fragments running locally on the 
> foreman and hence will resolve issue 1.



--
This message was sent by 

[jira] [Updated] (DRILL-5443) Managed External Sort fails with OOM while spilling to disk

[Robert Hou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Hou updated DRILL-5443:
--
Attachment: drill5443.drillbit.log
265a014b-8cae-30b5-adab-ff030b6c7086.sys.drill

> Managed External Sort fails with OOM while spilling to disk
> ---
>
> Key: DRILL-5443
> URL: https://issues.apache.org/jira/browse/DRILL-5443
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Rahul Challapalli
>Assignee: Paul Rogers
> Fix For: 1.12.0
>
> Attachments: 265a014b-8cae-30b5-adab-ff030b6c7086.sys.drill, 
> 27016969-ef53-40dc-b582-eea25371fa1c.sys.drill, drill5443.drillbit.log, 
> drillbit.log
>
>
> git.commit.id.abbrev=3e8b01d
> The below query fails with an OOM
> {code}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.disable_exchanges` = true;
> alter session set `planner.width.max_per_query` = 1;
> alter session set `planner.memory.max_query_memory_per_node` = 52428800;
> select s1.type type, flatten(s1.rms.rptd) rptds from (select d.type type, 
> d.uid uid, flatten(d.map.rm) rms from 
> dfs.`/drill/testdata/resource-manager/nested-large.json` d order by d.uid) s1 
> order by s1.rms.mapid;
> {code}
> Exception from the logs
> {code}
> 2017-04-24 17:22:59,439 [27016969-ef53-40dc-b582-eea25371fa1c:frag:0:0] INFO  
> o.a.d.e.p.i.x.m.ExternalSortBatch - User Error Occurred: External Sort 
> encountered an error while spilling to disk (Unable to allocate buffer of 
> size 524288 (rounded from 307197) due to memory limit. Current allocation: 
> 25886728)
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: External 
> Sort encountered an error while spilling to disk
> [Error Id: a64e3790-3a34-42c8-b4ea-4cb1df780e63 ]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
>  ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.doMergeAndSpill(ExternalSortBatch.java:1445)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.mergeAndSpill(ExternalSortBatch.java:1376)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.mergeRuns(ExternalSortBatch.java:1372)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.consolidateBatches(ExternalSortBatch.java:1299)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.mergeSpilledRuns(ExternalSortBatch.java:1195)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load(ExternalSortBatch.java:689)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext(ExternalSortBatch.java:559)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at 
> 

[jira] [Commented] (DRILL-5269) SYSTEM ERROR: JsonMappingException: No suitable constructor found for type [simple type, class org.apache.drill.exec.store.direct.DirectSubScan]

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146380#comment-16146380
 ] 

ASF GitHub Bot commented on DRILL-5269:
---

GitHub user vrozov opened a pull request:

https://github.com/apache/drill/pull/926

DRILL-5269 Make DirectSubScan Jackson JSON deserializable

@amansinha100 Please review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vrozov/drill DRILL-5269

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/926.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #926


commit 685653fed38beb4eea76cfe3460907386eb1a6c0
Author: Vlad Rozov 
Date:   2017-08-30T00:05:24Z

DRILL-5269 Make DirectSubScan Jackson JSON deserializable




> SYSTEM ERROR: JsonMappingException: No suitable constructor found for type 
> [simple type, class org.apache.drill.exec.store.direct.DirectSubScan]
> 
>
> Key: DRILL-5269
> URL: https://issues.apache.org/jira/browse/DRILL-5269
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Anas
>Assignee: Vlad Rozov
>Priority: Critical
> Attachments: tc_sm_parquet.tar.gz
>
>
> I am a query that has nested joins. The query fails with the following 
> exception.
> {code}
> SYSTEM ERROR: JsonMappingException: No suitable constructor found for type 
> [simple type, class org.apache.drill.exec.store.direct.DirectSubScan]: can 
> not instantiate from JSON object (missing default constructor or creator, or 
> perhaps need to add/enable type information?)
>  at [Source: {
>   "pop" : "broadcast-sender",
>   "@id" : 0,
>   "receiver-major-fragment" : 1,
>   "child" : {
> "pop" : "selection-vector-remover",
> "@id" : 1,
> "child" : {
>   "pop" : "filter",
>   "@id" : 2,
>   "child" : {
> "pop" : "project",
> "@id" : 3,
> "exprs" : [ {
>   "ref" : "`__measure__10`",
>   "expr" : "`count`"
> } ],
> "child" : {
>   "pop" : "DirectSubScan",
>   "@id" : 4,
>   "initialAllocation" : 100,
>   "maxAllocation" : 100,
>   "reader" : [ {
> "count" : 633
>   } ],
>   "cost" : 0.0
> },
> "initialAllocation" : 100,
> "maxAllocation" : 100,
> "cost" : 20.0
>   },
>   "expr" : "greater_than(`__measure__10`, 0) ",
>   "initialAllocation" : 100,
>   "maxAllocation" : 100,
>   "cost" : 10.0
> },
> "initialAllocation" : 100,
> "maxAllocation" : 100,
> "cost" : 10.0
>   },
>   "destinations" : [ {
> "minorFragmentId" : 0,
> "endpoint" : "Cg0xOTIuMTY4LjAuMTAwEKLyARij8gEgpPIB"
>   }, {
> "minorFragmentId" : 1,
> "endpoint" : "Cg0xOTIuMTY4LjAuMTAwEKLyARij8gEgpPIB"
>   } ],
>   "initialAllocation" : 100,
>   "maxAllocation" : 100,
>   "cost" : 10.0
> }; line: 20, column: 11] (through reference chain: 
> org.apache.drill.exec.physical.config.BroadcastSender["child"]->org.apache.drill.exec.physical.config.SelectionVectorRemover["child"]->org.apache.drill.exec.physical.config.Filter["child"]->org.apache.drill.exec.physical.config.Project["child"])
> Fragment 3:0
> [Error Id: 9fb4ef4a-f118-4625-94f5-56c96dc7bdb4 on 192.168.0.100:31010]
>   (com.fasterxml.jackson.databind.JsonMappingException) No suitable 
> constructor found for type [simple type, class 
> org.apache.drill.exec.store.direct.DirectSubScan]: can not instantiate from 
> JSON object (missing default constructor or creator, or perhaps need to 
> add/enable type information?)
>  at [Source: {
>   "pop" : "broadcast-sender",
>   "@id" : 0,
>   "receiver-major-fragment" : 1,
>   "child" : {
> "pop" : "selection-vector-remover",
> "@id" : 1,
> "child" : {
>   "pop" : "filter",
>   "@id" : 2,
>   "child" : {
> "pop" : "project",
> "@id" : 3,
> "exprs" : [ {
>   "ref" : "`__measure__10`",
>   "expr" : "`count`"
> } ],
> "child" : {
>   "pop" : "DirectSubScan",
>   "@id" : 4,
>   "initialAllocation" : 100,
>   "maxAllocation" : 100,
>   "reader" : [ {
> "count" : 633
>   } ],
>   "cost" : 0.0
> },
> "initialAllocation" : 100,
> "maxAllocation" : 100,
> "cost" : 20.0
>   },
>   "expr" : "greater_than(`__measure__10`, 0) ",
>   

[jira] [Commented] (DRILL-5443) Managed External Sort fails with OOM while spilling to disk

[Robert Hou (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146377#comment-16146377
 ] 

Robert Hou commented on DRILL-5443:
---

The stack trace is:
{noformat}
2017-08-29 16:55:26,610 [265a014b-8cae-30b5-adab-ff030b6c7086:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes 
ran out of memory while executing the query. (Unable to allocate buffer of size 
2097152 (rounded from 2097120) due to memory limit. Current allocation: 
45088896)
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
nodes ran out of memory while executing the query.

Unable to allocate buffer of size 2097152 (rounded from 2097120) due to memory 
limit. Current allocation: 45088896

[Error Id: 74361b0c-733f-453d-bc82-eda6bad4e64a ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
 ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:244)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_111]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to 
allocate buffer of size 2097152 (rounded from 2097120) due to memory limit. 
Current allocation: 45088896
at 
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:238) 
~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:213) 
~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.BigIntVector.reAlloc(BigIntVector.java:252) 
~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.BigIntVector$Mutator.setSafe(BigIntVector.java:452)
 ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.RepeatedBigIntVector$Mutator.addSafe(RepeatedBigIntVector.java:355)
 ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.RepeatedBigIntVector.copyFromSafe(RepeatedBigIntVector.java:220)
 ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.RepeatedBigIntVector$TransferImpl.copyValueSafe(RepeatedBigIntVector.java:202)
 ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.complex.MapVector$MapTransferPair.copyValueSafe(MapVector.java:225)
 ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.complex.MapVector$MapTransferPair.copyValueSafe(MapVector.java:225)
 ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.complex.MapVector.copyFromSafe(MapVector.java:82) 
~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.test.generated.PriorityQueueCopierGen408.doCopy(PriorityQueueCopierTemplate.java:27)
 ~[na:na]
at 
org.apache.drill.exec.test.generated.PriorityQueueCopierGen408.next(PriorityQueueCopierTemplate.java:77)
 ~[na:na]
at 
org.apache.drill.exec.physical.impl.xsort.managed.PriorityQueueCopierWrapper$BatchMerger.next(PriorityQueueCopierWrapper.java:267)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load(ExternalSortBatch.java:374)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext(ExternalSortBatch.java:303)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 

[jira] [Assigned] (DRILL-4181) Improve memory limit assignment for Sort operators

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers reassigned DRILL-4181:
--

Assignee: Paul Rogers

> Improve memory limit assignment for Sort operators
> --
>
> Key: DRILL-4181
> URL: https://issues.apache.org/jira/browse/DRILL-4181
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Aman Sinha
>Assignee: Paul Rogers
>
> Currently, the max_query_memory_per_node is divided among all the Sort 
> operators in the query plan, regardless of whether the Sort operators are 
> executing at the same time.  Since Sort is a blocking operator, the number of 
> concurrent Sorts running is limited.  Thus, the Sort on both sides of a 
> MergeJoin could be concurrently executing but a Sort that occurs above the 
> MergeJoin cannot start until the MergeJoin produces a row which is only when 
> both the child Sorts have produced a row. 
> Due to the conservative estimate, we have seen queries such as TPC-H Q8 with 
> forced MergeJoin run out-of-memory even with high max_query_memory_per_node.  
> This query plan has 15 Sort operators and with max_width = 23,  
> max_query_memory_per_node = 20GB, each Sort gets only  63MB (20GB/(23*15)).   
>  We should improve the algorithm for computing the memory limit. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5740) hash agg fail to read spill file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146372#comment-16146372
 ] 

ASF GitHub Bot commented on DRILL-5740:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/924


> hash agg fail to read spill file
> 
>
> Key: DRILL-5740
> URL: https://issues.apache.org/jira/browse/DRILL-5740
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.12.0
>Reporter: Chun Chang
>Assignee: Paul Rogers
>Priority: Blocker
>
> -Build: | 1.12.0-SNAPSHOT  | 11008d029bafa36279e3045c4ed1a64366080620
> -Multi-node drill cluster
> Running a query causing hash agg spill fails with the following error. And 
> this seems to be a regression.
> {noformat}
> Execution Failures:
> /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg5.q
> Query:
> select gby_date, gby_int32_rand, sum(int32_field), avg(float_field), 
> min(boolean_field), count(double_rand) from 
> dfs.`/drill/testdata/hagg/PARQUET-500M.parquet` group by gby_date, 
> gby_int32_rand order by gby_date, gby_int32_rand limit 30
> Failed with exception
> java.sql.SQLException: SYSTEM ERROR: FileNotFoundException: File 
> /tmp/drill/spill/10.10.30.168-31010/265f91f9-78d2-78a6-68ad-4709674efe0a_HashAgg_1-4-34/spill3
>  does not exist
> Fragment 1:34
> [Error Id: 291a79f8-9b7a-485d-9404-e7b7fe1d8f1e on 10.10.30.168:31010]
>   (java.lang.RuntimeException) java.io.FileNotFoundException: File 
> /tmp/drill/spill/10.10.30.168-31010/265f91f9-78d2-78a6-68ad-4709674efe0a_HashAgg_1-4-34/spill3
>  does not exist
> 
> org.apache.drill.exec.physical.impl.aggregate.SpilledRecordbatch.():67
> 
> org.apache.drill.exec.test.generated.HashAggregatorGen1891.outputCurrentBatch():980
> org.apache.drill.exec.test.generated.HashAggregatorGen1891.doWork():617
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> 
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5744) External sort fails with OOM error

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146371#comment-16146371
 ] 

Paul Rogers commented on DRILL-5744:


The gist of this issue is this:

{code}
[265b163b-cf44-d2ff-2e70-4cd746b56611:frag:0:0]
Completed load phase: read 977 batches, spilled 19 times, total input bytes: 
1,365,135,360
Starting consolidate phase. Batches = 977, Records = 400, Memory = 
23,101,440, In-memory batches 16, spilled runs 19
Starting merge phase. Runs = 35, Alloc. memory = 2,310,1440

Unable to allocate buffer of size 4,194,304 (rounded from 3276750) due to 
memory limit. Current allocation: 79986944

at 
org.apache.drill.exec.record.VectorInitializer.allocateBatch(VectorInitializer.java:85)
at 
org.apache.drill.exec.physical.impl.xsort.managed.PriorityQueueCopierWrapper$BatchMerger.next(PriorityQueueCopierWrapper.java:262)
 
at 
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load(ExternalSortBatch.java:374)
{code}

Configuration:

{code}
Config: spill file size = 268435456, spill batch size = 1048576, merge batch 
size = 16777216, mSort batch size = 65535
Memory config: Allocator limit = 76,214,400
Actual batch schema & sizes {
...
Records: 4096, Total size: 1,441,792, Data size: 376,615, Gross row width: 352, 
Net row width: 92, Density: 27}
Input Batch Estimates: record size = 123 bytes; net = 10776576 bytes, gross = 
16164864, records = 4096
Spill batch size: net = 1048575 bytes, gross = 1572862 bytes, records = 8525; 
spill file = 268435456 bytes
Output batch size: net = 8060805 bytes, gross = 12091207 bytes, records = 65535
Available memory: 76214400, buffer memory = 72020100, merge memory = 60092790
{code}

Initial analysis suggests that the sort tried to merge too many batches at the 
same time and ran out of memory when trying to read a batch from disk. This 
would be a bug.

> External sort fails with OOM error
> --
>
> Key: DRILL-5744
> URL: https://issues.apache.org/jira/browse/DRILL-5744
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.10.0
>Reporter: Robert Hou
>Assignee: Paul Rogers
> Fix For: 1.12.0
>
> Attachments: 265b163b-cf44-d2ff-2e70-4cd746b56611.sys.drill, 
> q34.drillbit.log
>
>
> Query is:
> {noformat}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.disable_exchanges` = true;
> alter session set `planner.width.max_per_query` = 1;
> alter session set `planner.memory.max_query_memory_per_node` = 152428800;
> select count(*) from (
>   select * from (
> select s1.type type, flatten(s1.rms.rptd) rptds, s1.rms, s1.uid 
> from (
>   select d.type type, d.uid uid, flatten(d.map.rm) rms from 
> dfs.`/drill/testdata/resource-manager/nested-large.json` d order by d.uid
> ) s1
>   ) s2
>   order by s2.rms.mapid
> );
> ALTER SESSION SET `exec.sort.disable_managed` = true;
> alter session set `planner.width.max_per_node` = 17;
> alter session set `planner.disable_exchanges` = false;
> alter session set `planner.width.max_per_query` = 1000;
> alter session set `planner.memory.max_query_memory_per_node` = 2147483648;
> {noformat}
> Stack trace is:
> {noformat}
> 2017-08-23 06:59:42,763 [266275e5-ebdb-14ae-d52d-00fa3a154f6d:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes
>  ran out of memory while executing the query. (Unable to allocate buffer of 
> size 4194304 (rounded from 3276750) due to memory limit. Current allocation: 7
> 9986944)
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
> nodes ran out of memory while executing the query.
> Unable to allocate buffer of size 4194304 (rounded from 3276750) due to 
> memory limit. Current allocation: 79986944
> [Error Id: 4f4959df-0921-4a50-b75e-56488469ab10 ]
>   at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
>  ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:244)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_51]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_51]
>   at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
> Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to 
> allocate buffer of size 4194304 (rounded from 

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146229#comment-16146229
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135925091
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
 ---
@@ -768,4 +765,73 @@ else if (exprHasPrefix && refHasPrefix) {
   }
 }
   }
+
+  /**
+   * handle FAST NONE specially when Project for query output. This 
happens when input returns a
+   * FAST NONE directly ( input does not return any batch with 
schema/data).
+   *
+   * Project operator has to return a batch with schema derived using the 
following 3 rules:
+   *  Case 1:  *  ==>  expand into an empty list of columns.
+   *  Case 2:  regular column reference ==> treat as nullable-int column
+   *  Case 3:  expressions => Call ExpressionTreeMaterialization over an 
empty vector contain.
+   *   Once the expression is materialized without error, use the 
output type of materialized
+   *   expression.
+   * The batch is constructed with the above rules, and recordCount = 0.
+   * Returned with OK_NEW_SCHEMA to down-stream operator.
--- End diff --

It could be Screen/Writer. 


> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146228#comment-16146228
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135925006
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
 ---
@@ -768,4 +765,73 @@ else if (exprHasPrefix && refHasPrefix) {
   }
 }
   }
+
+  /**
+   * handle FAST NONE specially when Project for query output. This 
happens when input returns a
--- End diff --

I change to "handleNullInput" from "handleFastNone". 

The concept of "NullInput" captures the scenarios where input does not 
produce any batch, and returning NONE directly.  The concept of "empty result 
set" could mean the case where we have 0 row, but still we could have a batch. 
I think "empty result set" is not precisely what we want to cover here. 


> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146220#comment-16146220
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135924386
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
 ---
@@ -768,4 +765,73 @@ else if (exprHasPrefix && refHasPrefix) {
   }
 }
   }
+
+  /**
+   * handle FAST NONE specially when Project for query output. This 
happens when input returns a
--- End diff --

Done. 


> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146222#comment-16146222
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135924412
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
 ---
@@ -768,4 +765,73 @@ else if (exprHasPrefix && refHasPrefix) {
   }
 }
   }
+
+  /**
+   * handle FAST NONE specially when Project for query output. This 
happens when input returns a
+   * FAST NONE directly ( input does not return any batch with 
schema/data).
+   *
+   * Project operator has to return a batch with schema derived using the 
following 3 rules:
+   *  Case 1:  *  ==>  expand into an empty list of columns.
+   *  Case 2:  regular column reference ==> treat as nullable-int column
+   *  Case 3:  expressions => Call ExpressionTreeMaterialization over an 
empty vector contain.
--- End diff --

Done. 


> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146221#comment-16146221
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135924402
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
 ---
@@ -768,4 +765,73 @@ else if (exprHasPrefix && refHasPrefix) {
   }
 }
   }
+
+  /**
+   * handle FAST NONE specially when Project for query output. This 
happens when input returns a
+   * FAST NONE directly ( input does not return any batch with 
schema/data).
+   *
--- End diff --

Done.


> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5740) hash agg fail to read spill file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146200#comment-16146200
 ] 

ASF GitHub Bot commented on DRILL-5740:
---

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/924
  
+1

LGTM.


> hash agg fail to read spill file
> 
>
> Key: DRILL-5740
> URL: https://issues.apache.org/jira/browse/DRILL-5740
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.12.0
>Reporter: Chun Chang
>Assignee: Paul Rogers
>Priority: Blocker
>
> -Build: | 1.12.0-SNAPSHOT  | 11008d029bafa36279e3045c4ed1a64366080620
> -Multi-node drill cluster
> Running a query causing hash agg spill fails with the following error. And 
> this seems to be a regression.
> {noformat}
> Execution Failures:
> /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg5.q
> Query:
> select gby_date, gby_int32_rand, sum(int32_field), avg(float_field), 
> min(boolean_field), count(double_rand) from 
> dfs.`/drill/testdata/hagg/PARQUET-500M.parquet` group by gby_date, 
> gby_int32_rand order by gby_date, gby_int32_rand limit 30
> Failed with exception
> java.sql.SQLException: SYSTEM ERROR: FileNotFoundException: File 
> /tmp/drill/spill/10.10.30.168-31010/265f91f9-78d2-78a6-68ad-4709674efe0a_HashAgg_1-4-34/spill3
>  does not exist
> Fragment 1:34
> [Error Id: 291a79f8-9b7a-485d-9404-e7b7fe1d8f1e on 10.10.30.168:31010]
>   (java.lang.RuntimeException) java.io.FileNotFoundException: File 
> /tmp/drill/spill/10.10.30.168-31010/265f91f9-78d2-78a6-68ad-4709674efe0a_HashAgg_1-4-34/spill3
>  does not exist
> 
> org.apache.drill.exec.physical.impl.aggregate.SpilledRecordbatch.():67
> 
> org.apache.drill.exec.test.generated.HashAggregatorGen1891.outputCurrentBatch():980
> org.apache.drill.exec.test.generated.HashAggregatorGen1891.doWork():617
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> 
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5750) NPE on botched Drillbit startup

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5750:
--

 Summary: NPE on botched Drillbit startup
 Key: DRILL-5750
 URL: https://issues.apache.org/jira/browse/DRILL-5750
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.11.0
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: 1.12.0


I am trying to launch a Drillbit as a server from Eclipse. I continue to 
stumble through the issues.

Drill does not help in that it crashes with NPEs due to the misconfiguration.

Crash 1: {{WorkManager}}:

{code}
  public void close() throws Exception {
...
getContext().close();
  }
{code}

{{getContext()}} is null.

Crash 2: {{SystemOptionManager}}:

{code}
  public void close() throws Exception {
options.close();
  }
{code}

{{options}} is null.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5738) Drill query takes 10+ minutes before start executing, excessive Hive metastore queries

[kevin zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevin zou updated DRILL-5738:
-
Attachment: log.tar.gz

Logs in debug mode

> Drill query takes 10+ minutes before start executing, excessive Hive 
> metastore queries
> --
>
> Key: DRILL-5738
> URL: https://issues.apache.org/jira/browse/DRILL-5738
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.6.0, 1.10.0
> Environment: mapr 5.2
>Reporter: kevin zou
>Priority: Critical
> Attachments: log.tar.gz
>
>
> I ve a Drill query on 14 tables in Hive. The query took a few seconds to 
> execute. However, the query would stay in "Starting" state for 10+ minutes 
> before execution. 
>  
> I set up the log to "Debug" mode to figure out what Drill had been doing 
> during the 10+ minutes, only to find out Drill generated excessive number of 
> meta data queries to hive meta store.  
>  
> Although each query took a few micro seconds (meta data cached in memory), 
> the number of queries was 3438793.
> drillbit.log:2017-06-05 18:50:57,201 
> [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG 
> o.a.d.e.s.hive.HiveMetadataProvider
>  - Took 4 µs to get stats from idm_intel_1x.lu_jde_emp_directory
> drillbit.log:2017-06-05 18:50:57,201 
> [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG 
> o.a.drill.exec.store.hive.HiveScan
> - HiveStats: numRows: 15, sizeInBytes: 15
> 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5738) Drill query takes 10+ minutes before start executing, excessive Hive metastore queries

[kevin zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevin zou updated DRILL-5738:
-
Affects Version/s: 1.10.0

> Drill query takes 10+ minutes before start executing, excessive Hive 
> metastore queries
> --
>
> Key: DRILL-5738
> URL: https://issues.apache.org/jira/browse/DRILL-5738
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.6.0, 1.10.0
> Environment: mapr 5.2
>Reporter: kevin zou
>Priority: Critical
>
> I ve a Drill query on 14 tables in Hive. The query took a few seconds to 
> execute. However, the query would stay in "Starting" state for 10+ minutes 
> before execution. 
>  
> I set up the log to "Debug" mode to figure out what Drill had been doing 
> during the 10+ minutes, only to find out Drill generated excessive number of 
> meta data queries to hive meta store.  
>  
> Although each query took a few micro seconds (meta data cached in memory), 
> the number of queries was 3438793.
> drillbit.log:2017-06-05 18:50:57,201 
> [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG 
> o.a.d.e.s.hive.HiveMetadataProvider
>  - Took 4 µs to get stats from idm_intel_1x.lu_jde_emp_directory
> drillbit.log:2017-06-05 18:50:57,201 
> [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG 
> o.a.drill.exec.store.hive.HiveScan
> - HiveStats: numRows: 15, sizeInBytes: 15
> 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5740) hash agg fail to read spill file

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146091#comment-16146091
 ] 

Paul Rogers commented on DRILL-5740:


[~jni], can you do a quick review? If you give a +1 then I can commit the fix.

> hash agg fail to read spill file
> 
>
> Key: DRILL-5740
> URL: https://issues.apache.org/jira/browse/DRILL-5740
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.12.0
>Reporter: Chun Chang
>Assignee: Paul Rogers
>Priority: Blocker
>
> -Build: | 1.12.0-SNAPSHOT  | 11008d029bafa36279e3045c4ed1a64366080620
> -Multi-node drill cluster
> Running a query causing hash agg spill fails with the following error. And 
> this seems to be a regression.
> {noformat}
> Execution Failures:
> /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg5.q
> Query:
> select gby_date, gby_int32_rand, sum(int32_field), avg(float_field), 
> min(boolean_field), count(double_rand) from 
> dfs.`/drill/testdata/hagg/PARQUET-500M.parquet` group by gby_date, 
> gby_int32_rand order by gby_date, gby_int32_rand limit 30
> Failed with exception
> java.sql.SQLException: SYSTEM ERROR: FileNotFoundException: File 
> /tmp/drill/spill/10.10.30.168-31010/265f91f9-78d2-78a6-68ad-4709674efe0a_HashAgg_1-4-34/spill3
>  does not exist
> Fragment 1:34
> [Error Id: 291a79f8-9b7a-485d-9404-e7b7fe1d8f1e on 10.10.30.168:31010]
>   (java.lang.RuntimeException) java.io.FileNotFoundException: File 
> /tmp/drill/spill/10.10.30.168-31010/265f91f9-78d2-78a6-68ad-4709674efe0a_HashAgg_1-4-34/spill3
>  does not exist
> 
> org.apache.drill.exec.physical.impl.aggregate.SpilledRecordbatch.():67
> 
> org.apache.drill.exec.test.generated.HashAggregatorGen1891.outputCurrentBatch():980
> org.apache.drill.exec.test.generated.HashAggregatorGen1891.doWork():617
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> 
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5729) Fix Travis Checks

[Timothy Farkas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Farkas updated DRILL-5729:
--
Labels: ready-to-commit  (was: )

> Fix Travis Checks
> -
>
> Key: DRILL-5729
> URL: https://issues.apache.org/jira/browse/DRILL-5729
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Currently the Travis Checks are failing. The failures are happening because 
> Travis recently switched their default build containers from Ubuntu precise 
> to Ubuntu Trusty and we do not explicitly define the dist we build on in our 
> travis.yml



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5740) hash agg fail to read spill file

[Chun Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chun Chang updated DRILL-5740:
--
Reviewer: Jinfeng Ni  (was: Boaz Ben-Zvi)

> hash agg fail to read spill file
> 
>
> Key: DRILL-5740
> URL: https://issues.apache.org/jira/browse/DRILL-5740
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.12.0
>Reporter: Chun Chang
>Assignee: Paul Rogers
>Priority: Blocker
>
> -Build: | 1.12.0-SNAPSHOT  | 11008d029bafa36279e3045c4ed1a64366080620
> -Multi-node drill cluster
> Running a query causing hash agg spill fails with the following error. And 
> this seems to be a regression.
> {noformat}
> Execution Failures:
> /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg5.q
> Query:
> select gby_date, gby_int32_rand, sum(int32_field), avg(float_field), 
> min(boolean_field), count(double_rand) from 
> dfs.`/drill/testdata/hagg/PARQUET-500M.parquet` group by gby_date, 
> gby_int32_rand order by gby_date, gby_int32_rand limit 30
> Failed with exception
> java.sql.SQLException: SYSTEM ERROR: FileNotFoundException: File 
> /tmp/drill/spill/10.10.30.168-31010/265f91f9-78d2-78a6-68ad-4709674efe0a_HashAgg_1-4-34/spill3
>  does not exist
> Fragment 1:34
> [Error Id: 291a79f8-9b7a-485d-9404-e7b7fe1d8f1e on 10.10.30.168:31010]
>   (java.lang.RuntimeException) java.io.FileNotFoundException: File 
> /tmp/drill/spill/10.10.30.168-31010/265f91f9-78d2-78a6-68ad-4709674efe0a_HashAgg_1-4-34/spill3
>  does not exist
> 
> org.apache.drill.exec.physical.impl.aggregate.SpilledRecordbatch.():67
> 
> org.apache.drill.exec.test.generated.HashAggregatorGen1891.outputCurrentBatch():980
> org.apache.drill.exec.test.generated.HashAggregatorGen1891.doWork():617
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> 
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5740) hash agg fail to read spill file

[Chun Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chun Chang updated DRILL-5740:
--
Priority: Blocker  (was: Critical)

> hash agg fail to read spill file
> 
>
> Key: DRILL-5740
> URL: https://issues.apache.org/jira/browse/DRILL-5740
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.12.0
>Reporter: Chun Chang
>Assignee: Paul Rogers
>Priority: Blocker
>
> -Build: | 1.12.0-SNAPSHOT  | 11008d029bafa36279e3045c4ed1a64366080620
> -Multi-node drill cluster
> Running a query causing hash agg spill fails with the following error. And 
> this seems to be a regression.
> {noformat}
> Execution Failures:
> /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg5.q
> Query:
> select gby_date, gby_int32_rand, sum(int32_field), avg(float_field), 
> min(boolean_field), count(double_rand) from 
> dfs.`/drill/testdata/hagg/PARQUET-500M.parquet` group by gby_date, 
> gby_int32_rand order by gby_date, gby_int32_rand limit 30
> Failed with exception
> java.sql.SQLException: SYSTEM ERROR: FileNotFoundException: File 
> /tmp/drill/spill/10.10.30.168-31010/265f91f9-78d2-78a6-68ad-4709674efe0a_HashAgg_1-4-34/spill3
>  does not exist
> Fragment 1:34
> [Error Id: 291a79f8-9b7a-485d-9404-e7b7fe1d8f1e on 10.10.30.168:31010]
>   (java.lang.RuntimeException) java.io.FileNotFoundException: File 
> /tmp/drill/spill/10.10.30.168-31010/265f91f9-78d2-78a6-68ad-4709674efe0a_HashAgg_1-4-34/spill3
>  does not exist
> 
> org.apache.drill.exec.physical.impl.aggregate.SpilledRecordbatch.():67
> 
> org.apache.drill.exec.test.generated.HashAggregatorGen1891.outputCurrentBatch():980
> org.apache.drill.exec.test.generated.HashAggregatorGen1891.doWork():617
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> 
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5729) Fix Travis Checks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145864#comment-16145864
 ] 

ASF GitHub Bot commented on DRILL-5729:
---

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/913
  
+1


> Fix Travis Checks
> -
>
> Key: DRILL-5729
> URL: https://issues.apache.org/jira/browse/DRILL-5729
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
> Fix For: 1.12.0
>
>
> Currently the Travis Checks are failing. The failures are happening because 
> Travis recently switched their default build containers from Ubuntu precise 
> to Ubuntu Trusty and we do not explicitly define the dist we build on in our 
> travis.yml



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4264) Allow field names to include dots

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145835#comment-16145835
 ] 

ASF GitHub Bot commented on DRILL-4264:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/909#discussion_r135872295
  
--- Diff: 
logical/src/main/java/org/apache/drill/common/expression/SchemaPath.java ---
@@ -115,6 +112,33 @@ public static SchemaPath create(NamePart namePart) {
   }
 
   /**
+   * Parses input string and returns {@code SchemaPath} instance.
+   *
+   * @param expr input string to be parsed
+   * @return {@code SchemaPath} instance
+   */
+  public static SchemaPath parseFromString(String expr) {
--- End diff --

Thanks for the explanation. Can you copy it into the Javadoc for this 
method? Will help others in the future.


> Allow field names to include dots
> -
>
> Key: DRILL-4264
> URL: https://issues.apache.org/jira/browse/DRILL-4264
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Reporter: Alex
>Assignee: Volodymyr Vysotskyi
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> If you have some json data like this...
> {code:javascript}
> {
>   "0.0.1":{
> "version":"0.0.1",
> "date_created":"2014-03-15"
>   },
>   "0.1.2":{
> "version":"0.1.2",
> "date_created":"2014-05-21"
>   }
> }
> {code}
> ... there is no way to select any of the rows since their identifiers contain 
> dots and when trying to select them, Drill throws the following error:
> Error: SYSTEM ERROR: UnsupportedOperationException: Unhandled field reference 
> "0.0.1"; a field reference identifier must not have the form of a qualified 
> name
> This must be fixed since there are many json data files containing dots in 
> some of the keys (e.g. when specifying version numbers etc)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4264) Allow field names to include dots

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145834#comment-16145834
 ] 

ASF GitHub Bot commented on DRILL-4264:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/909#discussion_r135868684
  
--- Diff: 
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/binary/CompareFunctionsProcessor.java
 ---
@@ -82,466 +45,37 @@ public static CompareFunctionsProcessor 
process(FunctionCall call, boolean nullC
 LogicalExpression swapArg = valueArg;
 valueArg = nameArg;
 nameArg = swapArg;
-evaluator.functionName = 
COMPARE_FUNCTIONS_TRANSPOSE_MAP.get(functionName);
+
evaluator.setFunctionName(COMPARE_FUNCTIONS_TRANSPOSE_MAP.get(functionName));
   }
-  evaluator.success = nameArg.accept(evaluator, valueArg);
+  evaluator.setSuccess(nameArg.accept(evaluator, valueArg));
 } else if (nullComparatorSupported && call.args.get(0) instanceof 
SchemaPath) {
-  evaluator.success = true;
-  evaluator.path = (SchemaPath) nameArg;
+  evaluator.setSuccess(true);
+  evaluator.setPath((SchemaPath) nameArg);
 }
 
 return evaluator;
   }
 
-  public CompareFunctionsProcessor(String functionName) {
-this.success = false;
-this.functionName = functionName;
-this.isEqualityFn = 
COMPARE_FUNCTIONS_TRANSPOSE_MAP.containsKey(functionName)
-&& 
COMPARE_FUNCTIONS_TRANSPOSE_MAP.get(functionName).equals(functionName);
-this.isRowKeyPrefixComparison = false;
-this.sortOrderAscending = true;
-  }
-
-  public byte[] getValue() {
-return value;
-  }
-
-  public boolean isSuccess() {
-return success;
-  }
-
-  public SchemaPath getPath() {
-return path;
-  }
-
-  public String getFunctionName() {
-return functionName;
-  }
-
-  public boolean isRowKeyPrefixComparison() {
-   return isRowKeyPrefixComparison;
-  }
-
-  public byte[] getRowKeyPrefixStartRow() {
-return rowKeyPrefixStartRow;
-  }
-
-  public byte[] getRowKeyPrefixStopRow() {
-  return rowKeyPrefixStopRow;
-  }
-
-  public Filter getRowKeyPrefixFilter() {
-  return rowKeyPrefixFilter;
-  }
-
-  public boolean isSortOrderAscending() {
-return sortOrderAscending;
-  }
-
   @Override
-  public Boolean visitCastExpression(CastExpression e, LogicalExpression 
valueArg) throws RuntimeException {
-if (e.getInput() instanceof CastExpression || e.getInput() instanceof 
SchemaPath) {
-  return e.getInput().accept(this, valueArg);
-}
-return false;
-  }
-
-  @Override
-  public Boolean visitConvertExpression(ConvertExpression e, 
LogicalExpression valueArg) throws RuntimeException {
-if (e.getConvertFunction() == ConvertExpression.CONVERT_FROM) {
-
-  String encodingType = e.getEncodingType();
-  int prefixLength= 0;
-
-  // Handle scan pruning in the following scenario:
-  // The row-key is a composite key and the CONVERT_FROM() function 
has byte_substr() as input function which is
-  // querying for the first few bytes of the row-key(start-offset 1)
-  // Example WHERE clause:
-  // CONVERT_FROM(BYTE_SUBSTR(row_key, 1, 8), 'DATE_EPOCH_BE') < DATE 
'2015-06-17'
-  if (e.getInput() instanceof FunctionCall) {
-
-// We can prune scan range only for big-endian encoded data
-if (encodingType.endsWith("_BE") == false) {
-  return false;
-}
-
-FunctionCall call = (FunctionCall)e.getInput();
-String functionName = call.getName();
-if (!functionName.equalsIgnoreCase("byte_substr")) {
-  return false;
-}
-
-LogicalExpression nameArg = call.args.get(0);
-LogicalExpression valueArg1 = call.args.size() >= 2 ? 
call.args.get(1) : null;
-LogicalExpression valueArg2 = call.args.size() >= 3 ? 
call.args.get(2) : null;
-
-if (((nameArg instanceof SchemaPath) == false) ||
- (valueArg1 == null) || ((valueArg1 instanceof IntExpression) 
== false) ||
- (valueArg2 == null) || ((valueArg2 instanceof IntExpression) 
== false)) {
-  return false;
-}
-
-boolean isRowKey = 
((SchemaPath)nameArg).getAsUnescapedPath().equals(DrillHBaseConstants.ROW_KEY);
-int offset = ((IntExpression)valueArg1).getInt();
-
-if (!isRowKey || (offset != 1)) {
-  return false;
-}
-
-this.path= (SchemaPath)nameArg;
  

[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145713#comment-16145713
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/904#discussion_r135851159
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestDateFunctions.java
 ---
@@ -135,11 +139,16 @@ public void testIntervalArithmetic() throws Exception 
{
 
 @Test
 public void testToChar() throws Exception {
-
-String expectedResults[] = {(new LocalDate(2008, 2, 
23)).toString("-MMM-dd"),
+Locale defaultLocale = Locale.getDefault();
+try{
+  Locale.setDefault(new Locale("en", "US"));
--- End diff --

Please replace `new Locale("en", "US")` by `Locale.US`


> change some date time unit cases with specific timezone or Local
> 
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145712#comment-16145712
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/904#discussion_r135850298
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestCastFunctions.java
 ---
@@ -73,16 +74,23 @@ public void testCastByConstantFolding() throws 
Exception {
 
   @Test // DRILL-3769
   public void testToDateForTimeStamp() throws Exception {
-final String query = "select to_date(to_timestamp(-1)) as col \n" +
-"from (values(1))";
+String defaultTimezone = System.getProperty("user.timezone");
+try {
+  System.setProperty("user.timezone","Etc/GMT");
+  final String query = "select to_date(to_timestamp(-1)) as col \n"
++ "from (values(1))";
 
-testBuilder()
+  testBuilder()
 .sqlQuery(query)
 .ordered()
 .baselineColumns("col")
 .baselineValues(new DateTime(1969, 12, 31, 0, 0))
 .build()
 .run();
+}finally {
+  System.setProperty("user.timezone",defaultTimezone);
--- End diff --

space


> change some date time unit cases with specific timezone or Local
> 
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145709#comment-16145709
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/904#discussion_r135850825
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestDateFunctions.java
 ---
@@ -20,7 +20,9 @@
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertTrue;
 
+import java.io.IOException;
--- End diff --

unused imports


> change some date time unit cases with specific timezone or Local
> 
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145714#comment-16145714
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/904#discussion_r135849992
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestCastFunctions.java
 ---
@@ -73,16 +74,23 @@ public void testCastByConstantFolding() throws 
Exception {
 
   @Test // DRILL-3769
   public void testToDateForTimeStamp() throws Exception {
-final String query = "select to_date(to_timestamp(-1)) as col \n" +
-"from (values(1))";
+String defaultTimezone = System.getProperty("user.timezone");
+try {
+  System.setProperty("user.timezone","Etc/GMT");
--- End diff --

Please add space after the comma.


> change some date time unit cases with specific timezone or Local
> 
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145711#comment-16145711
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/904#discussion_r135850258
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestCastFunctions.java
 ---
@@ -73,16 +74,23 @@ public void testCastByConstantFolding() throws 
Exception {
 
   @Test // DRILL-3769
   public void testToDateForTimeStamp() throws Exception {
-final String query = "select to_date(to_timestamp(-1)) as col \n" +
-"from (values(1))";
+String defaultTimezone = System.getProperty("user.timezone");
+try {
+  System.setProperty("user.timezone","Etc/GMT");
+  final String query = "select to_date(to_timestamp(-1)) as col \n"
++ "from (values(1))";
 
-testBuilder()
+  testBuilder()
 .sqlQuery(query)
 .ordered()
 .baselineColumns("col")
 .baselineValues(new DateTime(1969, 12, 31, 0, 0))
 .build()
 .run();
+}finally {
--- End diff --

please add space: `} finally {`


> change some date time unit cases with specific timezone or Local
> 
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145710#comment-16145710
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/904#discussion_r135848539
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestCastFunctions.java
 ---
@@ -22,14 +22,15 @@
 import org.apache.drill.BaseTestQuery;
 import org.apache.drill.common.util.FileUtils;
 import org.joda.time.DateTime;
+import org.junit.BeforeClass;
--- End diff --

Please remove unused imports.


> change some date time unit cases with specific timezone or Local
> 
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5749) Foreman and Netty threads occure deadlock

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145655#comment-16145655
 ] 

ASF GitHub Bot commented on DRILL-5749:
---

GitHub user weijietong opened a pull request:

https://github.com/apache/drill/pull/925

DRILL-5749: solve foreman and netty threads deadlock

break the nest invocation of channelClosed method to avoid nested lock 
holding

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/weijietong/drill DRILL-5749

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/925.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #925


commit b44f780a948c4a0898e7cee042c0590f0713f780
Author: weijietong 
Date:   2017-06-08T08:03:46Z

Merge pull request #1 from apache/master

sync

commit d045c757c80a759b435479cc89f33c749fc16ac2
Author: weijie.tong 
Date:   2017-08-11T08:01:36Z

Merge branch 'master' of github.com:weijietong/drill

commit 08b7006f4c70c45a17ebf7eae6beaa2bdb0d0454
Author: weijie.tong 
Date:   2017-08-20T12:05:51Z

update

commit 9e9ebb497a183e61a72665019e6e04070d912027
Author: weijie.tong 
Date:   2017-08-20T12:07:41Z

revert

commit 837d9fc58440fb584690f93b5f638ddcedf042a1
Author: weijie.tong 
Date:   2017-08-22T10:35:12Z

Merge branch 'master' of github.com:apache/drill

commit b1fc840ad9d0a9959b05a84bfd17f17067def32d
Author: weijie.tong 
Date:   2017-08-29T16:39:48Z

Merge branch 'master' of github.com:apache/drill

commit 03afe8650f76d182b86e2d8141780f002538f2b4
Author: weijie.tong 
Date:   2017-08-29T16:43:21Z

solve deadlock




> Foreman and Netty threads occure deadlock 
> --
>
> Key: DRILL-5749
> URL: https://issues.apache.org/jira/browse/DRILL-5749
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC
>Affects Versions: 1.10.0, 1.11.0
>Reporter: weijie.tong
>Priority: Critical
>
> when the cluster was in high concurrency query and the reused control 
> connection occured exceptoin, the foreman and netty threads both try to 
> acquire each other's lock then deadlock occured.  The netty thread hold the 
> map (RequestIdMap) lock then try to acquire the ReconnectingConnection lock 
> to send command, while the foreman thread hold the ReconnectingConnection 
> lock then try to acquire the RequestIdMap lock. So the deadlock happend.
> Below is the jstack dump:
> Found one Java-level deadlock:
> =
> "265aa5cb-e5e2-39ed-9c2f-7658b905372e:foreman":
>   waiting to lock monitor 0x7f935b721f48 (object 0x000656affc40, a 
> org.apache.drill.exec.rpc.control.ControlConnectionManager),
>   which is held by "265aa82f-d8c1-5df0-9946-003a4990db7e:foreman"
> "265aa82f-d8c1-5df0-9946-003a4990db7e:foreman":
>   waiting to lock monitor 0x7f90de3b9648 (object 0x0006b524d7e8, a 
> com.carrotsearch.hppc.IntObjectHashMap),
>   which is held by "BitServer-2"
> "BitServer-2":
>   waiting to lock monitor 0x7f935b721f48 (object 0x000656affc40, a 
> org.apache.drill.exec.rpc.control.ControlConnectionManager),
>   which is held by "265aa82f-d8c1-5df0-9946-003a4990db7e:foreman"
> Java stack information for the threads listed above:
> ===
> "265aa5cb-e5e2-39ed-9c2f-7658b905372e:foreman":
>   at 
> org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:72)
>   - waiting to lock <0x000656affc40> (a 
> org.apache.drill.exec.rpc.control.ControlConnectionManager)
>   at 
> org.apache.drill.exec.rpc.control.ControlTunnel.sendFragments(ControlTunnel.java:66)
>   at 
> org.apache.drill.exec.work.foreman.Foreman.sendRemoteFragments(Foreman.java:1210)
>   at 
> org.apache.drill.exec.work.foreman.Foreman.setupNonRootFragments(Foreman.java:1141)
>   at 
> org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:454)
>   at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1045)
>   at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
>   at java.lang.Thread.run(Thread.java:849)
> "265aa82f-d8c1-5df0-9946-003a4990db7e:foreman":
>   at 
> org.apache.drill.exec.rpc.RequestIdMap.createNewRpcListener(RequestIdMap.java:87)
>   - waiting to lock 

[jira] [Created] (DRILL-5749) Foreman and Netty threads occure deadlock

[weijie.tong (JIRA)]

weijie.tong created DRILL-5749:
--

 Summary: Foreman and Netty threads occure deadlock 
 Key: DRILL-5749
 URL: https://issues.apache.org/jira/browse/DRILL-5749
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - RPC
Affects Versions: 1.11.0, 1.10.0
Reporter: weijie.tong
Priority: Critical


when the cluster was in high concurrency query and the reused control 
connection occured exceptoin, the foreman and netty threads both try to acquire 
each other's lock then deadlock occured.  The netty thread hold the map 
(RequestIdMap) lock then try to acquire the ReconnectingConnection lock to send 
command, while the foreman thread hold the ReconnectingConnection lock then try 
to acquire the RequestIdMap lock. So the deadlock happend.

Below is the jstack dump:

Found one Java-level deadlock:
=
"265aa5cb-e5e2-39ed-9c2f-7658b905372e:foreman":
  waiting to lock monitor 0x7f935b721f48 (object 0x000656affc40, a 
org.apache.drill.exec.rpc.control.ControlConnectionManager),
  which is held by "265aa82f-d8c1-5df0-9946-003a4990db7e:foreman"
"265aa82f-d8c1-5df0-9946-003a4990db7e:foreman":
  waiting to lock monitor 0x7f90de3b9648 (object 0x0006b524d7e8, a 
com.carrotsearch.hppc.IntObjectHashMap),
  which is held by "BitServer-2"
"BitServer-2":
  waiting to lock monitor 0x7f935b721f48 (object 0x000656affc40, a 
org.apache.drill.exec.rpc.control.ControlConnectionManager),
  which is held by "265aa82f-d8c1-5df0-9946-003a4990db7e:foreman"

Java stack information for the threads listed above:
===
"265aa5cb-e5e2-39ed-9c2f-7658b905372e:foreman":
at 
org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:72)
- waiting to lock <0x000656affc40> (a 
org.apache.drill.exec.rpc.control.ControlConnectionManager)
at 
org.apache.drill.exec.rpc.control.ControlTunnel.sendFragments(ControlTunnel.java:66)
at 
org.apache.drill.exec.work.foreman.Foreman.sendRemoteFragments(Foreman.java:1210)
at 
org.apache.drill.exec.work.foreman.Foreman.setupNonRootFragments(Foreman.java:1141)
at 
org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:454)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1045)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:849)
"265aa82f-d8c1-5df0-9946-003a4990db7e:foreman":
at 
org.apache.drill.exec.rpc.RequestIdMap.createNewRpcListener(RequestIdMap.java:87)
- waiting to lock <0x0006b524d7e8> (a 
com.carrotsearch.hppc.IntObjectHashMap)
at 
org.apache.drill.exec.rpc.AbstractRemoteConnection.createNewRpcListener(AbstractRemoteConnection.java:153)
at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:115)
at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:89)
at 
org.apache.drill.exec.rpc.control.ControlConnection.send(ControlConnection.java:65)
at 
org.apache.drill.exec.rpc.control.ControlTunnel$SendFragment.doRpcCall(ControlTunnel.java:160)
at 
org.apache.drill.exec.rpc.control.ControlTunnel$SendFragment.doRpcCall(ControlTunnel.java:150)
at 
org.apache.drill.exec.rpc.ListeningCommand.connectionAvailable(ListeningCommand.java:38)
at 
org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:75)
- locked <0x000656affc40> (a 
org.apache.drill.exec.rpc.control.ControlConnectionManager)
at 
org.apache.drill.exec.rpc.control.ControlTunnel.sendFragments(ControlTunnel.java:66)
at 
org.apache.drill.exec.work.foreman.Foreman.sendRemoteFragments(Foreman.java:1210)
at 
org.apache.drill.exec.work.foreman.Foreman.setupNonRootFragments(Foreman.java:1141)
at 
org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:454)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1045)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:849)
"BitServer-2":
at 
org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:72)
- waiting to lock <0x000656affc40> (a 
org.apache.drill.exec.rpc.control.ControlConnectionManager)
at 
org.apache.drill.exec.rpc.control.ControlTunnel.cancelFragment(ControlTunnel.java:71)
at 

[jira] [Commented] (DRILL-5431) Support SSL

[Parth Chandra (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145577#comment-16145577
 ] 

Parth Chandra commented on DRILL-5431:
--

Initial implementation: 
https://github.com/parthchandra/drill/tree/DRILL-5431-q. Still testing this 
before I post a formal pull request.  
Comments are welcome.

> Support SSL
> ---
>
> Key: DRILL-5431
> URL: https://issues.apache.org/jira/browse/DRILL-5431
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Client - Java, Client - ODBC
>Reporter: Sudheesh Katkam
>Assignee: Sudheesh Katkam
>
> Support SSL between Drillbit and JDBC/ODBC drivers. Drill already supports 
> HTTPS for web traffic.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-3241) Query with window function runs out of direct memory and does not report back to client that it did

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145208#comment-16145208
 ] 

Khurram Faraaz commented on DRILL-3241:
---

[~kkhatua] Tried to verify the issue, ran the same query on SF1 TPC-DS JSON data
on latest apache master Drill 1.12.0-SNAPSHOT  commit id:  
a51c98b8bf210bbe9d3f4018361d937252d1226d 

we are seeing a different failure

{noformat}
[root@centos-01 bin]# ./sqlline -u 
"jdbc:drill:schema=dfs.tpcds_sf1_json_views;drillbit=10.10.100.201"
…
apache drill 1.12.0-SNAPSHOT 
"what ever the mind of man can conceive and believe, drill can query"
0: jdbc:drill:schema=dfs.tpcds_sf1_json_views> select 
. . . . . . . . . . . . . . . . . . . . . . .>   sum(ss_quantity) 
over(partition by ss_store_sk order by ss_sold_date_sk) 
. . . . . . . . . . . . . . . . . . . . . . .> from store_sales;
Error: DATA_READ ERROR: Error parsing JSON - Unexpected end-of-input 
within/between OBJECT entries

File  /drill/testdata/tpcds_sf1/json/store_sales/1_1_0.json
Record  266143
Column  2453
Fragment 3:1

[Error Id: f34cdf5a-9809-41a7-8aee-2938ab73bf04 on centos-04.qa.lab:31010] 
(state=,code=0)
0: jdbc:drill:schema=dfs.tpcds_sf1_json_views> 

{noformat}

Stack trace from drillbit.log

{noformat}
2017-08-29 12:17:55,254 [265aa48e-e317-1b8f-7d97-5c3b917da88f:frag:3:1] INFO  
o.a.d.e.s.easy.json.JSONRecordReader - User Error Occurred: Error parsing JSON 
- Unexpected end-of-input within/between OBJECT entries (Unexpected 
end-of-input within/between OBJECT entries
 at [Source: org.apache.drill.exec.store.dfs.DrillFSDataInputStream@3f8c6383; 
line: 6387411, column: 2453])
org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: Error 
parsing JSON - Unexpected end-of-input within/between OBJECT entries

File  /drill/testdata/tpcds_sf1/json/store_sales/1_1_0.json
Record  266143
Column  2453

[Error Id: f34cdf5a-9809-41a7-8aee-2938ab73bf04 ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
 ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.store.easy.json.JSONRecordReader.handleAndRaise(JSONRecordReader.java:188)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.store.easy.json.JSONRecordReader.next(JSONRecordReader.java:229)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:180) 
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:133)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:133)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105) 
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 

[jira] [Updated] (DRILL-5726) Support Impersonation without authentication for REST API

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5726:

Labels: ready-to-commit  (was: )

> Support Impersonation without authentication for REST API
> -
>
> Key: DRILL-5726
> URL: https://issues.apache.org/jira/browse/DRILL-5726
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
> Attachments: login_page.JPG, query_page_with_user_name.JPG
>
>
> Today if a user is not authenticated via REST API then there is no way to 
> provide a user name for executing queries. It will by default be executed as 
> "anonymous" user. This doesn't work when impersonation without authentication 
> is enabled on Drill server side, since anonymous user doesn't exist the query 
> will fail. We need a way to provide a user name when impersonation is enabled 
> on Drill side and query is executed from REST API.
> There are two approaches to achieve that:
> *1. Use form-based authentication*
> On Web UI user will be prompted to enter only login, then session for that 
> user will be created, user will be treated as admin. Form-based 
> authentication will cache user information, so user won't need to set user 
> name each time he / she wants to execute the query. Log in / out options will 
> be also available. Example screenshot of login page is attached 
> (login_page.JPG).
> From the programmatic perspective, user would need first to authenticate and 
> use cookie to get query result.
> *2. Use {{User-Name}} header in request*
> On Web UI on Query page additional input field will appear. User would need 
> to enter user name before issuing the query. Example screenshot of query page 
> is attached (query_page_with_user_name.JPG). Under the hood with user name 
> would be added to client request as request header. On server side this 
> header would be used to create user session principal. From the programmatic 
> perspective, user would need to add header when issuing the request.
> *_From the two above options second was chosen as it would ease REST API 
> usage from the programmatic perspective, plus using form-based authentication 
> may lead to false assumption that user is authenticated which is in reality 
> is not true._*
> *Implementation details of the second approach:*
> _Note: the below implementation will take affect only if authentication is 
> disabled and impersonation is enabled. By means of freemarker page won't 
> include js lib and script if condition is not met._
> On the client side additional input field was added to the query page. When 
> client is submitting the query, request would be changed using ajax to add 
> {{User-Name}} header which would be taken from the new input field. On the 
> server side, this header would be used to create session principal with 
> provided user name and admin rights. If user name header was not provided 
> (null or empty), the default anonymous principal will be used only in case 
> when it is not POST request to /query or /query.json. If it is POST request 
> to /query or /query.json {{User-Name}} header is required, so error will be 
> thrown. Also adding {{User Name}} input parameter is required from Web UI. 
> User won't be able to send request until field is filled in.
> *Adding user name header approaches:*
> _Web UI_ 
> enter user name in the User Name input field on Query page before submiiting 
> the query (query_page_with_user_name.JPG), this step is required.
> _sqlline_
> {code}./drill-localhost -n user1{code}
> _curl_ 
> {code} curl -v -H "Content-Type: application/json" -H "User-Name: user1" -d 
> '{"queryType":"SQL", "query": "select * from sys.version"}' 
> http://localhost:8047/query.json {code}
> _Java way_
> {code}
> String url = "http://localhost:8047/query.json;;
> URLConnection connection = new URL(url).openConnection();
> connection.setDoOutput(true); // Triggers POST.
> connection.addRequestProperty("User-Name", "user1");
> connection.setRequestProperty("Content-Type", "application/json");
> String data = "{\"queryType\":\"SQL\", \"query\": \"select * from 
> sys.version\"}";
> try (OutputStream output = connection.getOutputStream()) {
>   output.write(data.getBytes(StandardCharsets.UTF_8.name()));
> }
> try (InputStream response = connection.getInputStream()) {
>   String result = IOUtils.toString(response);
>   System.out.println(result);
> }
> {code}
> Note: {{Apache HttpClient}} can be used as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5726) Support Impersonation without authentication for REST API

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5726:

Description: 
Today if a user is not authenticated via REST API then there is no way to 
provide a user name for executing queries. It will by default be executed as 
"anonymous" user. This doesn't work when impersonation without authentication 
is enabled on Drill server side, since anonymous user doesn't exist the query 
will fail. We need a way to provide a user name when impersonation is enabled 
on Drill side and query is executed from REST API.

There are two approaches to achieve that:
*1. Use form-based authentication*
On Web UI user will be prompted to enter only login, then session for that user 
will be created, user will be treated as admin. Form-based authentication will 
cache user information, so user won't need to set user name each time he / she 
wants to execute the query. Log in / out options will be also available. 
Example screenshot of login page is attached (login_page.JPG).
>From the programmatic perspective, user would need first to authenticate and 
>use cookie to get query result.

*2. Use {{User-Name}} header in request*
On Web UI on Query page additional input field will appear. User would need to 
enter user name before issuing the query. Example screenshot of query page is 
attached (query_page_with_user_name.JPG). Under the hood with user name would 
be added to client request as request header. On server side this header would 
be used to create user session principal. From the programmatic perspective, 
user would need to add header when issuing the request.

*_From the two above options second was chosen as it would ease REST API usage 
from the programmatic perspective, plus using form-based authentication may 
lead to false assumption that user is authenticated which is in reality is not 
true._*

*Implementation details of the second approach:*
_Note: the below implementation will take affect only if authentication is 
disabled and impersonation is enabled. By means of freemarker page won't 
include js lib and script if condition is not met._
On the client side additional input field was added to the query page. When 
client is submitting the query, request would be changed using ajax to add 
{{User-Name}} header which would be taken from the new input field. On the 
server side, this header would be used to create session principal with 
provided user name and admin rights. If user name header was not provided (null 
or empty), the default anonymous principal will be used only in case when it is 
not POST request to /query or /query.json. If it is POST request to /query or 
/query.json {{User-Name}} header is required, so error will be thrown. Also 
adding {{User Name}} input parameter is required from Web UI. User won't be 
able to send request until field is filled in.

*Adding user name header approaches:*
_Web UI_ 
enter user name in the User Name input field on Query page before submiiting 
the query (query_page_with_user_name.JPG), this step is required.
_sqlline_
{code}./drill-localhost -n user1{code}
_curl_ 
{code} curl -v -H "Content-Type: application/json" -H "User-Name: user1" -d 
'{"queryType":"SQL", "query": "select * from sys.version"}' 
http://localhost:8047/query.json {code}
_Java way_
{code}
String url = "http://localhost:8047/query.json;;
URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true); // Triggers POST.
connection.addRequestProperty("User-Name", "user1");
connection.setRequestProperty("Content-Type", "application/json");

String data = "{\"queryType\":\"SQL\", \"query\": \"select * from 
sys.version\"}";
try (OutputStream output = connection.getOutputStream()) {
  output.write(data.getBytes(StandardCharsets.UTF_8.name()));
}

try (InputStream response = connection.getInputStream()) {
  String result = IOUtils.toString(response);
  System.out.println(result);
}
{code}
Note: {{Apache HttpClient}} can be used as well.

  was:
Today if a user is not authenticated via REST API then there is no way to 
provide a user name for executing queries. It will by default be executed as 
"anonymous" user. This doesn't work when impersonation without authentication 
is enabled on Drill server side, since anonymous user doesn't exist the query 
will fail. We need a way to provide a user name when impersonation is enabled 
on Drill side and query is executed from REST API.

There are two approaches to achieve that:
*1. Use form-based authentication*
On Web UI user will be prompted to enter only login, then session for that user 
will be created, user will be treated as admin. Form-based authentication will 
cache user information, so user won't need to set user name each time he / she 
wants to execute the query. Log in / out options will be also available. 
Example 

[jira] [Commented] (DRILL-4264) Allow field names to include dots

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145084#comment-16145084
 ] 

ASF GitHub Bot commented on DRILL-4264:
---

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/909#discussion_r135753493
  
--- Diff: 
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/binary/CompareFunctionsProcessor.java
 ---
@@ -82,466 +45,37 @@ public static CompareFunctionsProcessor 
process(FunctionCall call, boolean nullC
 LogicalExpression swapArg = valueArg;
 valueArg = nameArg;
 nameArg = swapArg;
-evaluator.functionName = 
COMPARE_FUNCTIONS_TRANSPOSE_MAP.get(functionName);
+
evaluator.setFunctionName(COMPARE_FUNCTIONS_TRANSPOSE_MAP.get(functionName));
   }
-  evaluator.success = nameArg.accept(evaluator, valueArg);
+  evaluator.setSuccess(nameArg.accept(evaluator, valueArg));
 } else if (nullComparatorSupported && call.args.get(0) instanceof 
SchemaPath) {
-  evaluator.success = true;
-  evaluator.path = (SchemaPath) nameArg;
+  evaluator.setSuccess(true);
+  evaluator.setPath((SchemaPath) nameArg);
 }
 
 return evaluator;
   }
 
-  public CompareFunctionsProcessor(String functionName) {
-this.success = false;
-this.functionName = functionName;
-this.isEqualityFn = 
COMPARE_FUNCTIONS_TRANSPOSE_MAP.containsKey(functionName)
-&& 
COMPARE_FUNCTIONS_TRANSPOSE_MAP.get(functionName).equals(functionName);
-this.isRowKeyPrefixComparison = false;
-this.sortOrderAscending = true;
-  }
-
-  public byte[] getValue() {
-return value;
-  }
-
-  public boolean isSuccess() {
-return success;
-  }
-
-  public SchemaPath getPath() {
-return path;
-  }
-
-  public String getFunctionName() {
-return functionName;
-  }
-
-  public boolean isRowKeyPrefixComparison() {
-   return isRowKeyPrefixComparison;
-  }
-
-  public byte[] getRowKeyPrefixStartRow() {
-return rowKeyPrefixStartRow;
-  }
-
-  public byte[] getRowKeyPrefixStopRow() {
-  return rowKeyPrefixStopRow;
-  }
-
-  public Filter getRowKeyPrefixFilter() {
-  return rowKeyPrefixFilter;
-  }
-
-  public boolean isSortOrderAscending() {
-return sortOrderAscending;
-  }
-
   @Override
-  public Boolean visitCastExpression(CastExpression e, LogicalExpression 
valueArg) throws RuntimeException {
-if (e.getInput() instanceof CastExpression || e.getInput() instanceof 
SchemaPath) {
-  return e.getInput().accept(this, valueArg);
-}
-return false;
-  }
-
-  @Override
-  public Boolean visitConvertExpression(ConvertExpression e, 
LogicalExpression valueArg) throws RuntimeException {
-if (e.getConvertFunction() == ConvertExpression.CONVERT_FROM) {
-
-  String encodingType = e.getEncodingType();
-  int prefixLength= 0;
-
-  // Handle scan pruning in the following scenario:
-  // The row-key is a composite key and the CONVERT_FROM() function 
has byte_substr() as input function which is
-  // querying for the first few bytes of the row-key(start-offset 1)
-  // Example WHERE clause:
-  // CONVERT_FROM(BYTE_SUBSTR(row_key, 1, 8), 'DATE_EPOCH_BE') < DATE 
'2015-06-17'
-  if (e.getInput() instanceof FunctionCall) {
-
-// We can prune scan range only for big-endian encoded data
-if (encodingType.endsWith("_BE") == false) {
-  return false;
-}
-
-FunctionCall call = (FunctionCall)e.getInput();
-String functionName = call.getName();
-if (!functionName.equalsIgnoreCase("byte_substr")) {
-  return false;
-}
-
-LogicalExpression nameArg = call.args.get(0);
-LogicalExpression valueArg1 = call.args.size() >= 2 ? 
call.args.get(1) : null;
-LogicalExpression valueArg2 = call.args.size() >= 3 ? 
call.args.get(2) : null;
-
-if (((nameArg instanceof SchemaPath) == false) ||
- (valueArg1 == null) || ((valueArg1 instanceof IntExpression) 
== false) ||
- (valueArg2 == null) || ((valueArg2 instanceof IntExpression) 
== false)) {
-  return false;
-}
-
-boolean isRowKey = 
((SchemaPath)nameArg).getAsUnescapedPath().equals(DrillHBaseConstants.ROW_KEY);
-int offset = ((IntExpression)valueArg1).getInt();
-
-if (!isRowKey || (offset != 1)) {
-  return false;
-}
-
-this.path= (SchemaPath)nameArg;
   

[jira] [Commented] (DRILL-5606) Some tests fail after creating a fresh clone

[Vitalii Diravka (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144966#comment-16144966
 ] 

Vitalii Diravka commented on DRILL-5606:


1. The above "TestCastFunctions.testToDateForTimeStamp()" test case is rely on 
timezone:
{code}
to_timestamp(0) for UTC timezone is Thu Jan 01 1970 00:00:00.
For Europe/Kiev is Thu Jan 01 1970 03:00:00.
{code}
Therefore test case can be rewritten (for example expected baseline value can 
be relied on timezone as well) or test case can be ignored (not good way).

2. The "TestNewDateFunctions.testIsDate()" can be edited since 
"Date.valueOf("1900-01-01").getTime()" returns not expected results for old 
dates for some timezones:
{code}.baselineValues(new DateTime(Date.valueOf("1900-01-01").getTime())){code}
can be replaced with {code}.baselineValues(new DateTime(1900, 1, 1, 0, 0)){code}



> Some tests fail after creating a fresh clone
> 
>
> Key: DRILL-5606
> URL: https://issues.apache.org/jira/browse/DRILL-5606
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
> Environment: {noformat}
> $ uname -a
> Linux mg-mate 4.4.0-81-generic #104-Ubuntu SMP Wed Jun 14 08:17:06 UTC 2017 
> x86_64 x86_64 x86_64 GNU/Linux
> $ lsb_release -a
> No LSB modules are available.
> Distributor ID:   Ubuntu
> Description:  Ubuntu 16.04.2 LTS
> Release:  16.04
> Codename: xenial
> $ java -version
> openjdk version "1.8.0_131"
> OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11)
> OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
> {noformat}
> Environment variables JAVA_HOME, JRE_HOME, JDK_HOME aren't configured. Java 
> executable is found as the PATH environment variables links to it. I can 
> provide more details if needed.
>Reporter: Muhammad Gelbana
> Attachments: failing_tests.tar.gz, full_log.txt.tar.gz, 
> surefire-reports.tar.gz
>
>
> I cloned Drill from Github using this url: 
> [https://github.com/apache/drill.git] and I didn't change the branch 
> afterwards, so I'm using *master*.
> Afterwards, I ran the following command
> {noformat}
> mvn clean install
> {noformat}
> I attached the full log but here is a snippet indicating the failing tests:
> {noformat}
> Failed tests: 
>   TestExtendedTypes.checkReadWriteExtended:60 expected:<...ateDay" : 
> "1997-07-1[6"
>   },
>   "drill_timestamp" : {
> "$date" : "2009-02-23T08:00:00.000Z"
>   },
>   "time" : {
> "$time" : "19:20:30.450Z"
>   },
>   "interval" : {
> "$interval" : "PT26.400S"
>   },
>   "integer" : {
> "$numberLong" : 4
>   },
>   "inner" : {
> "bin" : {
>   "$binary" : "ZHJpbGw="
> },
> "drill_date" : {
>   "$dateDay" : "1997-07-16]"
> },
> "drill_...> but was:<...ateDay" : "1997-07-1[5"
>   },
>   "drill_timestamp" : {
> "$date" : "2009-02-23T08:00:00.000Z"
>   },
>   "time" : {
> "$time" : "19:20:30.450Z"
>   },
>   "interval" : {
> "$interval" : "PT26.400S"
>   },
>   "integer" : {
> "$numberLong" : 4
>   },
>   "inner" : {
> "bin" : {
>   "$binary" : "ZHJpbGw="
> },
> "drill_date" : {
>   "$dateDay" : "1997-07-15]"
> },
> "drill_...>
> Tests in error: 
>   TestCastFunctions.testToDateForTimeStamp:79 »  at position 0 column '`col`' 
> mi...
>   TestNewDateFunctions.testIsDate:61 »  After matching 0 records, did not 
> find e...
> Tests run: 2128, Failures: 1, Errors: 2, Skipped: 139
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Apache Drill Root POM .. SUCCESS [ 19.805 
> s]
> [INFO] tools/Parent Pom ... SUCCESS [  0.605 
> s]
> [INFO] tools/freemarker codegen tooling ... SUCCESS [  7.077 
> s]
> [INFO] Drill Protocol . SUCCESS [  7.959 
> s]
> [INFO] Common (Logical Plan, Base expressions)  SUCCESS [  7.734 
> s]
> [INFO] Logical Plan, Base expressions . SUCCESS [  8.099 
> s]
> [INFO] exec/Parent Pom  SUCCESS [  0.575 
> s]
> [INFO] exec/memory/Parent Pom . SUCCESS [  0.513 
> s]
> [INFO] exec/memory/base ... SUCCESS [  4.666 
> s]
> [INFO] exec/rpc ... SUCCESS [  2.684 
> s]
> [INFO] exec/Vectors ... SUCCESS [01:11 
> min]
> [INFO] contrib/Parent Pom . SUCCESS [  0.547 
> s]
> [INFO] contrib/data/Parent Pom  SUCCESS [  0.496 
> s]
> [INFO] contrib/data/tpch-sample-data .. SUCCESS [  2.698 
> s]
> [INFO] exec/Java 

[jira] [Updated] (DRILL-5744) External sort fails with OOM error

[Robert Hou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Hou updated DRILL-5744:
--
Attachment: 265b163b-cf44-d2ff-2e70-4cd746b56611.sys.drill
q34.drillbit.log

> External sort fails with OOM error
> --
>
> Key: DRILL-5744
> URL: https://issues.apache.org/jira/browse/DRILL-5744
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.10.0
>Reporter: Robert Hou
>Assignee: Paul Rogers
> Fix For: 1.12.0
>
> Attachments: 265b163b-cf44-d2ff-2e70-4cd746b56611.sys.drill, 
> q34.drillbit.log
>
>
> Query is:
> {noformat}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.disable_exchanges` = true;
> alter session set `planner.width.max_per_query` = 1;
> alter session set `planner.memory.max_query_memory_per_node` = 152428800;
> select count(*) from (
>   select * from (
> select s1.type type, flatten(s1.rms.rptd) rptds, s1.rms, s1.uid 
> from (
>   select d.type type, d.uid uid, flatten(d.map.rm) rms from 
> dfs.`/drill/testdata/resource-manager/nested-large.json` d order by d.uid
> ) s1
>   ) s2
>   order by s2.rms.mapid
> );
> ALTER SESSION SET `exec.sort.disable_managed` = true;
> alter session set `planner.width.max_per_node` = 17;
> alter session set `planner.disable_exchanges` = false;
> alter session set `planner.width.max_per_query` = 1000;
> alter session set `planner.memory.max_query_memory_per_node` = 2147483648;
> {noformat}
> Stack trace is:
> {noformat}
> 2017-08-23 06:59:42,763 [266275e5-ebdb-14ae-d52d-00fa3a154f6d:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes
>  ran out of memory while executing the query. (Unable to allocate buffer of 
> size 4194304 (rounded from 3276750) due to memory limit. Current allocation: 7
> 9986944)
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
> nodes ran out of memory while executing the query.
> Unable to allocate buffer of size 4194304 (rounded from 3276750) due to 
> memory limit. Current allocation: 79986944
> [Error Id: 4f4959df-0921-4a50-b75e-56488469ab10 ]
>   at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
>  ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:244)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_51]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_51]
>   at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
> Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to 
> allocate buffer of size 4194304 (rounded from 3276750) due to memory limit. 
> Cur
> rent allocation: 79986944
>   at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:238) 
> ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:213) 
> ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:402)
>  ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:236)
>  ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.vector.AllocationHelper.allocatePrecomputedChildCount(AllocationHelper.java:33)
>  ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPS
> HOT]
>   at 
> org.apache.drill.exec.vector.AllocationHelper.allocate(AllocationHelper.java:46)
>  ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateVector(VectorInitializer.java:113)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT
> ]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateVector(VectorInitializer.java:95)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateMap(VectorInitializer.java:130)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateVector(VectorInitializer.java:93)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> 

[jira] [Updated] (DRILL-5744) External sort fails with OOM error

[Robert Hou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Hou updated DRILL-5744:
--
Attachment: (was: 266275e5-ebdb-14ae-d52d-00fa3a154f6d.sys.drill)

> External sort fails with OOM error
> --
>
> Key: DRILL-5744
> URL: https://issues.apache.org/jira/browse/DRILL-5744
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.10.0
>Reporter: Robert Hou
>Assignee: Paul Rogers
> Fix For: 1.12.0
>
>
> Query is:
> {noformat}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.disable_exchanges` = true;
> alter session set `planner.width.max_per_query` = 1;
> alter session set `planner.memory.max_query_memory_per_node` = 152428800;
> select count(*) from (
>   select * from (
> select s1.type type, flatten(s1.rms.rptd) rptds, s1.rms, s1.uid 
> from (
>   select d.type type, d.uid uid, flatten(d.map.rm) rms from 
> dfs.`/drill/testdata/resource-manager/nested-large.json` d order by d.uid
> ) s1
>   ) s2
>   order by s2.rms.mapid
> );
> ALTER SESSION SET `exec.sort.disable_managed` = true;
> alter session set `planner.width.max_per_node` = 17;
> alter session set `planner.disable_exchanges` = false;
> alter session set `planner.width.max_per_query` = 1000;
> alter session set `planner.memory.max_query_memory_per_node` = 2147483648;
> {noformat}
> Stack trace is:
> {noformat}
> 2017-08-23 06:59:42,763 [266275e5-ebdb-14ae-d52d-00fa3a154f6d:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes
>  ran out of memory while executing the query. (Unable to allocate buffer of 
> size 4194304 (rounded from 3276750) due to memory limit. Current allocation: 7
> 9986944)
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
> nodes ran out of memory while executing the query.
> Unable to allocate buffer of size 4194304 (rounded from 3276750) due to 
> memory limit. Current allocation: 79986944
> [Error Id: 4f4959df-0921-4a50-b75e-56488469ab10 ]
>   at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
>  ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:244)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_51]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_51]
>   at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
> Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to 
> allocate buffer of size 4194304 (rounded from 3276750) due to memory limit. 
> Cur
> rent allocation: 79986944
>   at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:238) 
> ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:213) 
> ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:402)
>  ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:236)
>  ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.vector.AllocationHelper.allocatePrecomputedChildCount(AllocationHelper.java:33)
>  ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPS
> HOT]
>   at 
> org.apache.drill.exec.vector.AllocationHelper.allocate(AllocationHelper.java:46)
>  ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateVector(VectorInitializer.java:113)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT
> ]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateVector(VectorInitializer.java:95)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateMap(VectorInitializer.java:130)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateVector(VectorInitializer.java:93)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateBatch(VectorInitializer.java:85)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> 

[jira] [Updated] (DRILL-5744) External sort fails with OOM error

[Robert Hou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Hou updated DRILL-5744:
--
Attachment: (was: drillbit.log)

> External sort fails with OOM error
> --
>
> Key: DRILL-5744
> URL: https://issues.apache.org/jira/browse/DRILL-5744
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.10.0
>Reporter: Robert Hou
>Assignee: Paul Rogers
> Fix For: 1.12.0
>
> Attachments: 266275e5-ebdb-14ae-d52d-00fa3a154f6d.sys.drill
>
>
> Query is:
> {noformat}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.disable_exchanges` = true;
> alter session set `planner.width.max_per_query` = 1;
> alter session set `planner.memory.max_query_memory_per_node` = 152428800;
> select count(*) from (
>   select * from (
> select s1.type type, flatten(s1.rms.rptd) rptds, s1.rms, s1.uid 
> from (
>   select d.type type, d.uid uid, flatten(d.map.rm) rms from 
> dfs.`/drill/testdata/resource-manager/nested-large.json` d order by d.uid
> ) s1
>   ) s2
>   order by s2.rms.mapid
> );
> ALTER SESSION SET `exec.sort.disable_managed` = true;
> alter session set `planner.width.max_per_node` = 17;
> alter session set `planner.disable_exchanges` = false;
> alter session set `planner.width.max_per_query` = 1000;
> alter session set `planner.memory.max_query_memory_per_node` = 2147483648;
> {noformat}
> Stack trace is:
> {noformat}
> 2017-08-23 06:59:42,763 [266275e5-ebdb-14ae-d52d-00fa3a154f6d:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes
>  ran out of memory while executing the query. (Unable to allocate buffer of 
> size 4194304 (rounded from 3276750) due to memory limit. Current allocation: 7
> 9986944)
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
> nodes ran out of memory while executing the query.
> Unable to allocate buffer of size 4194304 (rounded from 3276750) due to 
> memory limit. Current allocation: 79986944
> [Error Id: 4f4959df-0921-4a50-b75e-56488469ab10 ]
>   at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
>  ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:244)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_51]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_51]
>   at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
> Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to 
> allocate buffer of size 4194304 (rounded from 3276750) due to memory limit. 
> Cur
> rent allocation: 79986944
>   at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:238) 
> ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:213) 
> ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:402)
>  ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:236)
>  ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.vector.AllocationHelper.allocatePrecomputedChildCount(AllocationHelper.java:33)
>  ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPS
> HOT]
>   at 
> org.apache.drill.exec.vector.AllocationHelper.allocate(AllocationHelper.java:46)
>  ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateVector(VectorInitializer.java:113)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT
> ]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateVector(VectorInitializer.java:95)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateMap(VectorInitializer.java:130)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateVector(VectorInitializer.java:93)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.VectorInitializer.allocateBatch(VectorInitializer.java:85)
>  

[jira] [Updated] (DRILL-5744) External sort fails with OOM error

[Robert Hou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Hou updated DRILL-5744:
--
Description: 
Query is:
{noformat}
ALTER SESSION SET `exec.sort.disable_managed` = false;
alter session set `planner.width.max_per_node` = 1;
alter session set `planner.disable_exchanges` = true;
alter session set `planner.width.max_per_query` = 1;
alter session set `planner.memory.max_query_memory_per_node` = 152428800;
select count(*) from (
  select * from (
select s1.type type, flatten(s1.rms.rptd) rptds, s1.rms, s1.uid 
from (
  select d.type type, d.uid uid, flatten(d.map.rm) rms from 
dfs.`/drill/testdata/resource-manager/nested-large.json` d order by d.uid
) s1
  ) s2
  order by s2.rms.mapid
);
ALTER SESSION SET `exec.sort.disable_managed` = true;
alter session set `planner.width.max_per_node` = 17;
alter session set `planner.disable_exchanges` = false;
alter session set `planner.width.max_per_query` = 1000;
alter session set `planner.memory.max_query_memory_per_node` = 2147483648;
{noformat}

Stack trace is:
{noformat}
2017-08-23 06:59:42,763 [266275e5-ebdb-14ae-d52d-00fa3a154f6d:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes
 ran out of memory while executing the query. (Unable to allocate buffer of 
size 4194304 (rounded from 3276750) due to memory limit. Current allocation: 7
9986944)
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
nodes ran out of memory while executing the query.

Unable to allocate buffer of size 4194304 (rounded from 3276750) due to memory 
limit. Current allocation: 79986944

[Error Id: 4f4959df-0921-4a50-b75e-56488469ab10 ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
 ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:244)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_51]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to 
allocate buffer of size 4194304 (rounded from 3276750) due to memory limit. Cur
rent allocation: 79986944
at 
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:238) 
~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:213) 
~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:402) 
~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:236)
 ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.AllocationHelper.allocatePrecomputedChildCount(AllocationHelper.java:33)
 ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPS
HOT]
at 
org.apache.drill.exec.vector.AllocationHelper.allocate(AllocationHelper.java:46)
 ~[vector-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.VectorInitializer.allocateVector(VectorInitializer.java:113)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT
]
at 
org.apache.drill.exec.record.VectorInitializer.allocateVector(VectorInitializer.java:95)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.VectorInitializer.allocateMap(VectorInitializer.java:130)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.VectorInitializer.allocateVector(VectorInitializer.java:93)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.VectorInitializer.allocateBatch(VectorInitializer.java:85)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.xsort.managed.PriorityQueueCopierWrapper$BatchMerger.next(PriorityQueueCopierWrapper.java:262)
 ~[drill-java
-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load(ExternalSortBatch.java:374)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12
.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext(ExternalSortBatch.java:303)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar
:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
 

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144853#comment-16144853
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135710183
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java 
---
@@ -329,9 +326,11 @@ public TypedFieldId getValueVectorId(SchemaPath path) {
 
   @VisibleForTesting
   public static class Mutator implements OutputMutator {
-/** Whether schema has changed since last inquiry (via #isNewSchema}). 
 Is
- *  true before first inquiry. */
-private boolean schemaChanged = true;
+/** Flag keeping track whether top-level schema has changed since last 
inquiry (via #isNewSchema}).
+ * It's initialized to false, or reset to false after #isNewSchema or 
after #clear, until a new value vector
+ * or a value vector with different type is added to fieldVectorMap.
+ **/
+private boolean schemaChanged;
--- End diff --

No comments with respect to "flag vs counter". But the flag is in the 
existing code, and the patch I have simply 1) change the initial value of flag, 
2) add comments. 


> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144850#comment-16144850
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135709915
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java 
---
@@ -252,14 +235,28 @@ public IterOutcome next() {
 }
   }
 
+  private void releaseAssets() {
+container.zeroVectors();
+  }
+
+  private void clearFieldVectorMap() {
+for (final ValueVector v : mutator.fieldVectorMap().values()) {
+  v.clear();
+}
+  }
+
+  private void advanceNextReader() throws ExecutionSetupException {
+currentReader = readers.next();
+implicitValues = implicitColumns.hasNext() ? implicitColumns.next() : 
null;
+currentReader.setup(oContext, mutator);
--- End diff --

Nice catch. You are right we should not advance to next reader if we need 
pass a batch to downstream, since `reader.setup()` for new reader could modify 
both mutator and container.  This seems to be true for original ScanBatch code 
as well. 

In the revised batch,  I remove the code to get next reader in constructor, 
and put the code to get next reader in the beginning of loop when needed. In 
case of return a `OK_NEW_SCHEMA`, we will only get next reader in the next 
`next()` method call after `OK_NEW_SCHEMA` is returned.  
 


> Schema change problems caused by empty batch
> 
>
> Key: DRILL-5546
> URL: https://issues.apache.org/jira/browse/DRILL-5546
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> There have been a few JIRAs opened related to schema change failure caused by 
> empty batch. This JIRA is opened as an umbrella for all those related JIRAS ( 
> such as DRILL-4686, DRILL-4734, DRILL4476, DRILL-4255, etc).
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144842#comment-16144842
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/906#discussion_r135708467
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java 
---
@@ -152,97 +157,75 @@ public void kill(boolean sendUpstream) {
 }
   }
 
-  private void releaseAssets() {
-container.zeroVectors();
-  }
-
-  private void clearFieldVectorMap() {
-for (final ValueVector v : mutator.fieldVectorMap().values()) {
-  v.clear();
-}
-  }
-
   @Override
   public IterOutcome next() {
 if (done) {
   return IterOutcome.NONE;
 }
 oContext.getStats().startProcessing();
 try {
-  try {
-injector.injectChecked(context.getExecutionControls(), 
"next-allocate", OutOfMemoryException.class);
-
-currentReader.allocate(mutator.fieldVectorMap());
-  } catch (OutOfMemoryException e) {
-clearFieldVectorMap();
-throw UserException.memoryError(e).build(logger);
-  }
-  while ((recordCount = currentReader.next()) == 0) {
+  while (true) {
 try {
-  if (!readers.hasNext()) {
-// We're on the last reader, and it has no (more) rows.
-currentReader.close();
-releaseAssets();
-done = true;  // have any future call to next() return NONE
-
-if (mutator.isNewSchema()) {
-  // This last reader has a new schema (e.g., we have a 
zero-row
-  // file or other source).  (Note that some sources have a 
non-
-  // null/non-trivial schema even when there are no rows.)
+  injector.injectChecked(context.getExecutionControls(), 
"next-allocate", OutOfMemoryException.class);
+  currentReader.allocate(mutator.fieldVectorMap());
+} catch (OutOfMemoryException e) {
+  clearFieldVectorMap();
+  throw UserException.memoryError(e).build(logger);
+}
 
-  container.buildSchema(SelectionVectorMode.NONE);
-  schema = container.getSchema();
+recordCount = currentReader.next();
+Preconditions.checkArgument(recordCount >= 0,
+"recordCount from RecordReader.next() should not be negative");
 
-  return IterOutcome.OK_NEW_SCHEMA;
-}
-return IterOutcome.NONE;
-  }
-  // At this point, the reader that hit its end is not the last 
reader.
+boolean isNewRegularSchema = mutator.isNewSchema();
+// We should skip the reader, when recordCount = 0 && ! 
isNewRegularSchema.
+// Add/set implicit column vectors, only when reader gets > 0 row, 
or
+// when reader gets 0 row but with a schema with new field added
+if (recordCount > 0 || isNewRegularSchema) {
+  addImplicitVectors();
+  populateImplicitVectors();
+}
 
-  // If all the files we have read so far are just empty, the 
schema is not useful
-  if (! hasReadNonEmptyFile) {
-container.clear();
-clearFieldVectorMap();
-mutator.clear();
-  }
+boolean isNewImplicitSchema = mutator.isNewSchema();
+for (VectorWrapper w : container) {
+  w.getValueVector().getMutator().setValueCount(recordCount);
+}
+final boolean isNewSchema = isNewRegularSchema || 
isNewImplicitSchema;
+oContext.getStats().batchReceived(0, recordCount, isNewSchema);
 
+if (recordCount == 0) {
   currentReader.close();
-  currentReader = readers.next();
-  implicitValues = implicitColumns.hasNext() ? 
implicitColumns.next() : null;
-  currentReader.setup(oContext, mutator);
-  try {
-currentReader.allocate(mutator.fieldVectorMap());
-  } catch (OutOfMemoryException e) {
-clearFieldVectorMap();
-throw UserException.memoryError(e).build(logger);
+  if (isNewSchema) {
+// current reader presents a new schema in mutator even though 
it has 0 row.
+// This could happen when data sources have a non-trivial 
schema with 0 row.
+container.buildSchema(SelectionVectorMode.NONE);
+schema = container.getSchema();
+if (readers.hasNext()) {
+  advanceNextReader();
+} else {
+