[jira] [Commented] (DRILL-5394) Optimize query planning for MapR-DB tables by caching row counts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946530#comment-15946530
 ] 

ASF GitHub Bot commented on DRILL-5394:
---

Github user Agirish commented on the issue:

https://github.com/apache/drill/pull/802
  
Can confirm that with this fix, query times have improved (details will be 
updated in JIRA). Thanks for making this change.


> Optimize query planning for MapR-DB tables by caching row counts
> 
>
> Key: DRILL-5394
> URL: https://issues.apache.org/jira/browse/DRILL-5394
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning & Optimization, Storage - MapRDB
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Abhishek Girish
>Assignee: Padma Penumarthy
>  Labels: MapR-DB-Binary
> Fix For: 1.11.0
>
>
> On large MapR-DB tables, it was observed that the query planning time was 
> longer than expected. With DEBUG logs, it was understood that there were 
> multiple calls being made to get MapR-DB region locations and to fetch total 
> row count for tables.
> {code}
> 2017-02-23 13:59:55,246 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:05,006 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.planner.logical.DrillOptiq - Function
> ...
> 2017-02-23 14:00:05,031 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:16,438 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.planner.logical.DrillOptiq - Special
> ...
> 2017-02-23 14:00:16,439 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:28,479 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.planner.logical.DrillOptiq - Special
> ...
> 2017-02-23 14:00:28,480 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:42,396 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.planner.logical.DrillOptiq - Special
> ...
> 2017-02-23 14:00:42,397 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:54,609 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.p.s.h.DefaultSqlHandler - VOLCANO:Physical Planning (49588ms):
> {code}
> We should cache these stats and reuse them where all required during query 
> planning. This should help reduce query planning time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5395) Query on MapR-DB table fails with NPE due to an issue with assignment logic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946528#comment-15946528
 ] 

ASF GitHub Bot commented on DRILL-5395:
---

Github user Agirish commented on the issue:

https://github.com/apache/drill/pull/803
  
Can confirm that with this fix, the previously observed issue (NPE) is no 
longer reproducible. Thanks for making this change. 


> Query on MapR-DB table fails with NPE due to an issue with assignment logic
> ---
>
> Key: DRILL-5395
> URL: https://issues.apache.org/jira/browse/DRILL-5395
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization, Storage - MapRDB
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Abhishek Girish
>Assignee: Padma Penumarthy
>  Labels: MapR-DB-Binary
> Fix For: 1.11.0
>
> Attachments: drillbit.log.txt
>
>
> We uncovered this issue when working on DRILL-5394. 
> The MapR-DB table in question had 5 tablets with skewed data distribution (~6 
> million rows). A partial WIP fix for DRILL-5394 caused the number of rows to 
> be reported incorrectly (~300,000). 2 minor fragments were created (due to 
> filter selectivity) for scanning the 5 tablets. And this resulted in an NPE, 
> possibly related to an issue with assignment logic, that was now exposed. 
> Representative query:
> {code}
> SELECT Convert_from(avail.customer, 'UTF8') AS ABC, 
>Convert_from(prop.customer, 'UTF8')  AS PQR 
> FROM   (SELECT Convert_from(a.row_key, 'UTF8') 
>AS customer, 
>Cast(Convert_from(a.data .` l_discount ` , 'double_be') AS 
> FLOAT) 
>AS availability 
> FROM   db.tpch_maprdb.lineitem_1 a 
> WHERE  Convert_from(a.row_key, 'UTF8') = '%004%') AS avail 
>join 
>   (SELECT Convert_from(b.row_key, 'UTF8') 
>   AS customer, 
>Cast( 
>Convert_from(b.data .` l_discount ` , 'double_be') AS FLOAT) AS 
>availability 
> FROM   db.tpch_maprdb.lineitem_1 b 
> WHERE  Convert_from(b.row_key, 'UTF8') LIKE '%003%') AS prop 
>  ON avail.customer = prop.customer; 
> {code}
> Error:
> {code}
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> NullPointerException
> {code}
> Log attached. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5395) Query on MapR-DB table fails with NPE due to an issue with assignment logic

[Abhishek Girish (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946517#comment-15946517
 ] 

Abhishek Girish commented on DRILL-5395:


Thanks for such detailed explanation of the underlying issue, [~ppenumarthy]. 
Very useful! 

> Query on MapR-DB table fails with NPE due to an issue with assignment logic
> ---
>
> Key: DRILL-5395
> URL: https://issues.apache.org/jira/browse/DRILL-5395
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization, Storage - MapRDB
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Abhishek Girish
>Assignee: Padma Penumarthy
>  Labels: MapR-DB-Binary
> Fix For: 1.11.0
>
> Attachments: drillbit.log.txt
>
>
> We uncovered this issue when working on DRILL-5394. 
> The MapR-DB table in question had 5 tablets with skewed data distribution (~6 
> million rows). A partial WIP fix for DRILL-5394 caused the number of rows to 
> be reported incorrectly (~300,000). 2 minor fragments were created (due to 
> filter selectivity) for scanning the 5 tablets. And this resulted in an NPE, 
> possibly related to an issue with assignment logic, that was now exposed. 
> Representative query:
> {code}
> SELECT Convert_from(avail.customer, 'UTF8') AS ABC, 
>Convert_from(prop.customer, 'UTF8')  AS PQR 
> FROM   (SELECT Convert_from(a.row_key, 'UTF8') 
>AS customer, 
>Cast(Convert_from(a.data .` l_discount ` , 'double_be') AS 
> FLOAT) 
>AS availability 
> FROM   db.tpch_maprdb.lineitem_1 a 
> WHERE  Convert_from(a.row_key, 'UTF8') = '%004%') AS avail 
>join 
>   (SELECT Convert_from(b.row_key, 'UTF8') 
>   AS customer, 
>Cast( 
>Convert_from(b.data .` l_discount ` , 'double_be') AS FLOAT) AS 
>availability 
> FROM   db.tpch_maprdb.lineitem_1 b 
> WHERE  Convert_from(b.row_key, 'UTF8') LIKE '%003%') AS prop 
>  ON avail.customer = prop.customer; 
> {code}
> Error:
> {code}
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> NullPointerException
> {code}
> Log attached. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5395) Query on MapR-DB table fails with NPE due to an issue with assignment logic

[Abhishek Girish (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Girish updated DRILL-5395:
---
Attachment: drillbit.log.txt

> Query on MapR-DB table fails with NPE due to an issue with assignment logic
> ---
>
> Key: DRILL-5395
> URL: https://issues.apache.org/jira/browse/DRILL-5395
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization, Storage - MapRDB
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Abhishek Girish
>Assignee: Padma Penumarthy
>  Labels: MapR-DB-Binary
> Fix For: 1.11.0
>
> Attachments: drillbit.log.txt
>
>
> We uncovered this issue when working on DRILL-5394. 
> The MapR-DB table in question had 5 tablets with skewed data distribution (~6 
> million rows). A partial WIP fix for DRILL-5394 caused the number of rows to 
> be reported incorrectly (~300,000). 2 minor fragments were created (due to 
> filter selectivity) for scanning the 5 tablets. And this resulted in an NPE, 
> possibly related to an issue with assignment logic, that was now exposed. 
> Representative query:
> {code}
> SELECT Convert_from(avail.customer, 'UTF8') AS ABC, 
>Convert_from(prop.customer, 'UTF8')  AS PQR 
> FROM   (SELECT Convert_from(a.row_key, 'UTF8') 
>AS customer, 
>Cast(Convert_from(a.data .` l_discount ` , 'double_be') AS 
> FLOAT) 
>AS availability 
> FROM   db.tpch_maprdb.lineitem_1 a 
> WHERE  Convert_from(a.row_key, 'UTF8') = '%004%') AS avail 
>join 
>   (SELECT Convert_from(b.row_key, 'UTF8') 
>   AS customer, 
>Cast( 
>Convert_from(b.data .` l_discount ` , 'double_be') AS FLOAT) AS 
>availability 
> FROM   db.tpch_maprdb.lineitem_1 b 
> WHERE  Convert_from(b.row_key, 'UTF8') LIKE '%003%') AS prop 
>  ON avail.customer = prop.customer; 
> {code}
> Error:
> {code}
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> NullPointerException
> {code}
> Log attached. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5395) Query on MapR-DB table fails with NPE due to an issue with assignment logic

[Padma Penumarthy (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946511#comment-15946511
 ] 

Padma Penumarthy commented on DRILL-5395:
-

Based on number of fragments (decided by parallelization) and HBase regions to 
assign to these fragments, we calculate minimum and maximum regions to assign 
per fragment. 
Each region is assigned to a fragment on the node it is located on, fragments 
being picked in a round robin fashion. At the end, we make adjustments so each 
fragment is assigned at least the minimum and not more than maximum regions per 
fragment.
We can  have a case where we assigned up to minimum per fragment and still have 
regions to assign. In that case, minHeap will be null. We try to assign the 
unassigned regions to fragments with less than minimum by accessing minHeap, 
which is null and thus causing NPE.

In this case, 
we have 5 regions with ~300K rows. We apply selectivity of 0.5 and determine 
the cost as ~150,000. With slice target of 100K, we  create 2 fragments. On a 3 
node cluster, we have 2 fragments assigned to 2 nodes and none assigned to 3rd 
node. With 5 regions to assign, minimum regions to assign per fragment is 
calculated as 5/2 i.e. 2 and maximum regions to assign per fragment is 3.  In 
the first round,  each region is allocated to a fragment on the node on which 
it is located. 4 regions got assigned to 2 fragments i.e. both the fragments 
have 2 each. One of the regions is located on the node which does not have any 
fragments and it does not get assigned to a fragment in the first round. Once 
initial assignment based on region location is done, we try to adjust the 
assignments so that each fragment gets assigned at least the minimum and at 
most the maximum. For this adjustment phase, we build a minHeap which is 
supposed to contain fragments which have less than minimum assigned. In this 
case, since we assigned 2 regions each to 2 fragments, minHeap is null. For 
assigning the 5th region, we try to get fragment which has least number of 
regions assigned by accessing minHeap, which is NULL.  This causes Null Pointer 
Exception.

Fix is to include the fragments which have minimum assigned in minHeap (not 
just less than minimum).




> Query on MapR-DB table fails with NPE due to an issue with assignment logic
> ---
>
> Key: DRILL-5395
> URL: https://issues.apache.org/jira/browse/DRILL-5395
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization, Storage - MapRDB
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Abhishek Girish
>Assignee: Padma Penumarthy
>  Labels: MapR-DB-Binary
> Fix For: 1.11.0
>
>
> We uncovered this issue when working on DRILL-5394. 
> The MapR-DB table in question had 5 tablets with skewed data distribution (~6 
> million rows). A partial WIP fix for DRILL-5394 caused the number of rows to 
> be reported incorrectly (~300,000). 2 minor fragments were created (due to 
> filter selectivity) for scanning the 5 tablets. And this resulted in an NPE, 
> possibly related to an issue with assignment logic, that was now exposed. 
> Representative query:
> {code}
> SELECT Convert_from(avail.customer, 'UTF8') AS ABC, 
>Convert_from(prop.customer, 'UTF8')  AS PQR 
> FROM   (SELECT Convert_from(a.row_key, 'UTF8') 
>AS customer, 
>Cast(Convert_from(a.data .` l_discount ` , 'double_be') AS 
> FLOAT) 
>AS availability 
> FROM   db.tpch_maprdb.lineitem_1 a 
> WHERE  Convert_from(a.row_key, 'UTF8') = '%004%') AS avail 
>join 
>   (SELECT Convert_from(b.row_key, 'UTF8') 
>   AS customer, 
>Cast( 
>Convert_from(b.data .` l_discount ` , 'double_be') AS FLOAT) AS 
>availability 
> FROM   db.tpch_maprdb.lineitem_1 b 
> WHERE  Convert_from(b.row_key, 'UTF8') LIKE '%003%') AS prop 
>  ON avail.customer = prop.customer; 
> {code}
> Error:
> {code}
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> NullPointerException
> {code}
> Log attached. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5083) RecordIterator can sometimes restart a query on close

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5083:
---
Attachment: DrillOperatorErrorHandlingRedesign.pdf

Proposed design to address the issue at a fundamental level.

> RecordIterator can sometimes restart a query on close
> -
>
> Key: DRILL-5083
> URL: https://issues.apache.org/jira/browse/DRILL-5083
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
> Attachments: DrillOperatorErrorHandlingRedesign.pdf
>
>
> This one is very confusing...
> In a test with a MergeJoin and external sort, operators are stacked something 
> like this:
> {code}
> Screen
> - MergeJoin
> - - External Sort
> ...
> {code}
> Using the injector to force a OOM in spill, the external sort threw a 
> UserException up the stack. This was handed by:
> {code}
> IteratorValidatorBatchIterator.next( )
> RecordIterator.clearInflightBatches( )
> RecordIterator.close( )
> MergeJoinBatch.close( )
> {code}
> Which does the following:
> {code}
>   // Check whether next() should even have been called in current state.
>   if (null != exceptionState) {
> throw new IllegalStateException(
> {code}
> But, the exceptionState is set, so we end up throwing an 
> IllegalStateException during cleanup.
> Seems the code should agree: if {{next( )}} will be called during cleanup, 
> then {{next( )}} should gracefully handle that case.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5391) CTAS: folder and file permission should be configurable

[Chua Tianxiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946396#comment-15946396
 ] 

Chua Tianxiang commented on DRILL-5391:
---

I've checked our configuration and can confirm that we have kept HDFS 
fs.permissions.umask-mode as default to 022. 

I've also checked that our dfs.namenode.acls.enabled setting is set to false.

I'm not sure where else would this be configured.

> CTAS: folder and file permission should be configurable
> ---
>
> Key: DRILL-5391
> URL: https://issues.apache.org/jira/browse/DRILL-5391
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.9.0, 1.10.0
> Environment: CentOS 7, HDP 2.4
>Reporter: Chua Tianxiang
>Priority: Minor
> Attachments: Drill-1-10.PNG, Drill-1-9.PNG
>
>
> In Drill 1.9, CREATE TABLE AS creates a folder with permissions 777, while on 
> Drill 1.10, the same commands creates a folder with permission 775. Both 
> drills are started with root user, installed on the same servers and accesses 
> the same HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5395) Query on MapR-DB table fails with NPE due to an issue with assignment logic

[Abhishek Girish (JIRA)]

Abhishek Girish created DRILL-5395:
--

 Summary: Query on MapR-DB table fails with NPE due to an issue 
with assignment logic
 Key: DRILL-5395
 URL: https://issues.apache.org/jira/browse/DRILL-5395
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization, Storage - MapRDB
Affects Versions: 1.9.0, 1.10.0
Reporter: Abhishek Girish
Assignee: Padma Penumarthy
 Fix For: 1.11.0


We uncovered this issue when working on DRILL-5394. 

The MapR-DB table in question had 5 tablets with skewed data distribution (~6 
million rows). A partial WIP fix for DRILL-5394 caused the number of rows to be 
reported incorrectly (~300,000). 2 minor fragments were created (due to filter 
selectivity) for scanning the 5 tablets. And this resulted in an NPE, possibly 
related to an issue with assignment logic, that was now exposed. 

Representative query:
{code}
SELECT Convert_from(avail.customer, 'UTF8') AS ABC, 
   Convert_from(prop.customer, 'UTF8')  AS PQR 
FROM   (SELECT Convert_from(a.row_key, 'UTF8') 
   AS customer, 
   Cast(Convert_from(a.data .` l_discount ` , 'double_be') AS 
FLOAT) 
   AS availability 
FROM   db.tpch_maprdb.lineitem_1 a 
WHERE  Convert_from(a.row_key, 'UTF8') = '%004%') AS avail 
   join 
  (SELECT Convert_from(b.row_key, 'UTF8') 
  AS customer, 
   Cast( 
   Convert_from(b.data .` l_discount ` , 'double_be') AS FLOAT) AS 
   availability 
FROM   db.tpch_maprdb.lineitem_1 b 
WHERE  Convert_from(b.row_key, 'UTF8') LIKE '%003%') AS prop 
 ON avail.customer = prop.customer; 
{code}
Error:
{code}
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
NullPointerException
{code}

Log attached. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-4764) Parquet file with INT_16, etc. logical types not supported by simple SELECT

[Rahul Challapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Challapalli closed DRILL-4764.


Verified and automated tests

> Parquet file with INT_16, etc. logical types not supported by simple SELECT
> ---
>
> Key: DRILL-4764
> URL: https://issues.apache.org/jira/browse/DRILL-4764
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.6.0
>Reporter: Paul Rogers
>Assignee: Serhii Harnyk
> Fix For: 1.10.0
>
> Attachments: int_16.parquet, int_8.parquet, uint_16.parquet, 
> uint_32.parquet, uint_8.parquet
>
>
> Create a Parquet file with the following schema:
> message int16Data { required int32 index; required int32 value (INT_16); }
> Store it as int_16.parquet in the local file system. Query it with:
> SELECT * from `local`.`root`.`int_16.parquet`;
> The result, in the web UI, is this error:
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
> UnsupportedOperationException: unsupported type: INT32 INT_16 Fragment 0:0 
> [Error Id: c63f66b4-e5a9-4a35-9ceb-546b74645dd4 on 172.30.1.28:31010]
> The INT_16 logical (or "original") type simply tells consumers of the file 
> that the data is actually a 16-bit signed int. Presumably, this should tell 
> Drill to use the SmallIntVector (or NullableSmallIntVector) class for 
> storage. Without supporting this annotation, even 16-bit integers must be 
> stored as 32-bits within Drill.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5394) Optimize query planning for MapR-DB tables by caching row counts

[Abhishek Girish (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Girish updated DRILL-5394:
---
Labels: MapR-DB-Binary  (was: )

> Optimize query planning for MapR-DB tables by caching row counts
> 
>
> Key: DRILL-5394
> URL: https://issues.apache.org/jira/browse/DRILL-5394
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning & Optimization, Storage - MapRDB
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Abhishek Girish
>Assignee: Padma Penumarthy
>  Labels: MapR-DB-Binary
> Fix For: 1.11.0
>
>
> On large MapR-DB tables, it was observed that the query planning time was 
> longer than expected. With DEBUG logs, it was understood that there were 
> multiple calls being made to get MapR-DB region locations and to fetch total 
> row count for tables.
> {code}
> 2017-02-23 13:59:55,246 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:05,006 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.planner.logical.DrillOptiq - Function
> ...
> 2017-02-23 14:00:05,031 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:16,438 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.planner.logical.DrillOptiq - Special
> ...
> 2017-02-23 14:00:16,439 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:28,479 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.planner.logical.DrillOptiq - Special
> ...
> 2017-02-23 14:00:28,480 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:42,396 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.planner.logical.DrillOptiq - Special
> ...
> 2017-02-23 14:00:42,397 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:54,609 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
> o.a.d.e.p.s.h.DefaultSqlHandler - VOLCANO:Physical Planning (49588ms):
> {code}
> We should cache these stats and reuse them where all required during query 
> planning. This should help reduce query planning time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5394) Optimize query planning for MapR-DB tables by caching row counts

[Abhishek Girish (JIRA)]

Abhishek Girish created DRILL-5394:
--

 Summary: Optimize query planning for MapR-DB tables by caching row 
counts
 Key: DRILL-5394
 URL: https://issues.apache.org/jira/browse/DRILL-5394
 Project: Apache Drill
  Issue Type: Improvement
  Components: Query Planning & Optimization, Storage - MapRDB
Affects Versions: 1.9.0, 1.10.0
Reporter: Abhishek Girish
Assignee: Padma Penumarthy
 Fix For: 1.11.0


On large MapR-DB tables, it was observed that the query planning time was 
longer than expected. With DEBUG logs, it was understood that there were 
multiple calls being made to get MapR-DB region locations and to fetch total 
row count for tables.

{code}
2017-02-23 13:59:55,246 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
2017-02-23 14:00:05,006 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
o.a.d.e.planner.logical.DrillOptiq - Function
...
2017-02-23 14:00:05,031 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
2017-02-23 14:00:16,438 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
o.a.d.e.planner.logical.DrillOptiq - Special
...
2017-02-23 14:00:16,439 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
2017-02-23 14:00:28,479 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
o.a.d.e.planner.logical.DrillOptiq - Special
...
2017-02-23 14:00:28,480 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
2017-02-23 14:00:42,396 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
o.a.d.e.planner.logical.DrillOptiq - Special
...
2017-02-23 14:00:42,397 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
2017-02-23 14:00:54,609 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG 
o.a.d.e.p.s.h.DefaultSqlHandler - VOLCANO:Physical Planning (49588ms):
{code}

We should cache these stats and reuse them where all required during query 
planning. This should help reduce query planning time.





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5326) Unit tests failures related to the SERVER_METADTA

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5326.
---

> Unit tests failures related to the SERVER_METADTA
> -
>
> Key: DRILL-5326
> URL: https://issues.apache.org/jira/browse/DRILL-5326
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: 1.10.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Blocker
> Fix For: 1.10.0
>
>
> 1. In DRILL-5301 a new SERVER_META rpc call was introduced. The server will 
> support this method only from 1.10.0 drill version. For drill 1.10.0-SNAPHOT 
> it is disabled. 
> When I enabled this method (by way of upgrading drill version to 1.10.0 or 
> 1.11.0-SNAPSHOT) I found the following exception:
> {code}
> java.lang.AssertionError: Unexpected/unhandled MinorType value GENERIC_OBJECT
> {code}
> It appears in several tests (for example in 
> DatabaseMetadataTest#testNullsAreSortedMethodsSaySortedHigh).
> The reason of it is "GENERIC_OBJECT" RPC-/protobuf-level type is appeared in 
> the ServerMetadata#ConvertSupportList. (Supporting of GENERIC_OBJECT was 
> added in DRILL-1126).
> The proposed solution is to add the appropriate "JAVA_OBJECT" sql type name 
> for this "GENERIC_OBJECT" RPC-/protobuf-level data type.
> 2. After fixing the first one the mentioned above test still fails by reason 
> of the incorrect "NullCollation" value in the "ServerMetaProvider". According 
> to the [doc|https://drill.apache.org/docs/order-by-clause/#usage-notes] the 
> default val should be NC_HIGH (NULL is the highest value).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5326) Unit tests failures related to the SERVER_METADTA

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946326#comment-15946326
 ] 

Kunal Khatua commented on DRILL-5326:
-

QA Verification not required

> Unit tests failures related to the SERVER_METADTA
> -
>
> Key: DRILL-5326
> URL: https://issues.apache.org/jira/browse/DRILL-5326
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: 1.10.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Blocker
> Fix For: 1.10.0
>
>
> 1. In DRILL-5301 a new SERVER_META rpc call was introduced. The server will 
> support this method only from 1.10.0 drill version. For drill 1.10.0-SNAPHOT 
> it is disabled. 
> When I enabled this method (by way of upgrading drill version to 1.10.0 or 
> 1.11.0-SNAPSHOT) I found the following exception:
> {code}
> java.lang.AssertionError: Unexpected/unhandled MinorType value GENERIC_OBJECT
> {code}
> It appears in several tests (for example in 
> DatabaseMetadataTest#testNullsAreSortedMethodsSaySortedHigh).
> The reason of it is "GENERIC_OBJECT" RPC-/protobuf-level type is appeared in 
> the ServerMetadata#ConvertSupportList. (Supporting of GENERIC_OBJECT was 
> added in DRILL-1126).
> The proposed solution is to add the appropriate "JAVA_OBJECT" sql type name 
> for this "GENERIC_OBJECT" RPC-/protobuf-level data type.
> 2. After fixing the first one the mentioned above test still fails by reason 
> of the incorrect "NullCollation" value in the "ServerMetaProvider". According 
> to the [doc|https://drill.apache.org/docs/order-by-clause/#usage-notes] the 
> default val should be NC_HIGH (NULL is the highest value).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5159) ProjectMergeRule in Drill should operate on RelNodes with same convention trait.

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-5159:

Reviewer: Rahul Challapalli  (was: Aman Sinha)

> ProjectMergeRule in Drill should operate on RelNodes with same convention 
> trait.
> 
>
> Key: DRILL-5159
> URL: https://issues.apache.org/jira/browse/DRILL-5159
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill extended version of  Calcite's ProjectMergeRule is used in a 
> VolcanoPlanner where RelNodes with different convention could match with this 
> rule. 
> For instance, we could see this rule could be invoked when a DrillProject on 
> top of a LogicalProject. Also, since the output RelNode is built from the 
> default Project RelFactory, such rule execution could end up with a 
> LogicalProject.
> {code}
> DrillProjecttransform  
> \  ===>   LogicalProject
> LogicalProject
> {code}
>  
> This leads to un-necessary rule execution, or in certain case could lead to 
> an infinite loop.  
> The proposed fix is to check matched RelNodes to make sure that they do have 
> Calcite Logical convention. That way, both inputs and output of this rule 
> would have same convention trait.  This should reduce planning time, and 
> avoid the possiblity of loop.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5159) ProjectMergeRule in Drill should operate on RelNodes with same convention trait.

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946277#comment-15946277
 ] 

Kunal Khatua commented on DRILL-5159:
-

[~rkins] Please close the bug when you have tests automated .

> ProjectMergeRule in Drill should operate on RelNodes with same convention 
> trait.
> 
>
> Key: DRILL-5159
> URL: https://issues.apache.org/jira/browse/DRILL-5159
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill extended version of  Calcite's ProjectMergeRule is used in a 
> VolcanoPlanner where RelNodes with different convention could match with this 
> rule. 
> For instance, we could see this rule could be invoked when a DrillProject on 
> top of a LogicalProject. Also, since the output RelNode is built from the 
> default Project RelFactory, such rule execution could end up with a 
> LogicalProject.
> {code}
> DrillProjecttransform  
> \  ===>   LogicalProject
> LogicalProject
> {code}
>  
> This leads to un-necessary rule execution, or in certain case could lead to 
> an infinite loop.  
> The proposed fix is to check matched RelNodes to make sure that they do have 
> Calcite Logical convention. That way, both inputs and output of this rule 
> would have same convention trait.  This should reduce planning time, and 
> avoid the possiblity of loop.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4980) Upgrading of the approach of parquet date correctness status detection

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946260#comment-15946260
 ] 

Rahul Challapalli commented on DRILL-4980:
--

[~vitalii] Does this have any functional impact? Or can I come up with a 
query/data combination to reproduce the issue? I am asking as I want to 
automate the testcase in our functional regression runs

> Upgrading of the approach of parquet date correctness status detection
> --
>
> Key: DRILL-4980
> URL: https://issues.apache.org/jira/browse/DRILL-4980
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Parquet
>Affects Versions: 1.9.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
> Fix For: 1.10.0
>
>
> This jira is an addition for the 
> [DRILL-4203|https://issues.apache.org/jira/browse/DRILL-4203].
> The date correctness label for the new generated parquet files should be 
> upgraded. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5354) Create CTTAS Documentaion

[Bridget Bevens (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bridget Bevens closed DRILL-5354.
-

Closing ticket. 


> Create CTTAS Documentaion
> -
>
> Key: DRILL-5354
> URL: https://issues.apache.org/jira/browse/DRILL-5354
> Project: Apache Drill
>  Issue Type: Task
>  Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Padma Heid
>Assignee: Bridget Bevens
>Priority: Minor
> Fix For: 1.10.0
>
> Attachments: new_line.JPG, unnecessary_paragraph.JPG
>
>
> Work with Dev ,QA and PM to create user documentation for the CTTAS command.  
> When docs are posted, this link will be available: 
> https://drill.apache.org/docs/create-temporary-table-as-cttas/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5354) Create CTTAS Documentaion

[Bridget Bevens (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946239#comment-15946239
 ] 

Bridget Bevens commented on DRILL-5354:
---

Padma created the doc based on Arina's design doc in DRILL-4956:
https://docs.google.com/document/d/1gSRo_w6q2WR5fPx7SsQ5IaVmJXJ6xCOJfYGyqpVOC-g/edit
Padma and Arina worked together to make sure the content was accurate. 
I edited the content based on notes from Arina in this ticket. 
Closing this ticket. 


> Create CTTAS Documentaion
> -
>
> Key: DRILL-5354
> URL: https://issues.apache.org/jira/browse/DRILL-5354
> Project: Apache Drill
>  Issue Type: Task
>  Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Padma Heid
>Assignee: Bridget Bevens
>Priority: Minor
> Fix For: 1.10.0
>
> Attachments: new_line.JPG, unnecessary_paragraph.JPG
>
>
> Work with Dev ,QA and PM to create user documentation for the CTTAS command.  
> When docs are posted, this link will be available: 
> https://drill.apache.org/docs/create-temporary-table-as-cttas/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5088) Error when reading DBRef column

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5088.
---

> Error when reading DBRef column
> ---
>
> Key: DRILL-5088
> URL: https://issues.apache.org/jira/browse/DRILL-5088
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
> Environment: drill 1.9.0
> mongo 3.2
>Reporter: Guillaume Champion
>Assignee: Chunhui Shi
> Fix For: 1.10.0
>
>
> In a mongo database with DBRef, when a DBRef is inserted in the first line of 
> a mongo's collection drill query failed :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> {code}
> Simple example to reproduce:
> In mongo instance
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> [Error Id: 2944d766-e483-4453-a706-3d481397b186 on Analytics-Biznet:31010] 
> (state=,code=0)
> {code}
> If the first line doesn't contain de DBRef, drill will querying correctly :
> In a mongo instance :
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8939") });
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> +--+---+
> | _id  |account   
>  |
> +--+---+
> | {"$oid":"582081d96b69060001fd8939"}  | {"$id":{}}   
>  |
> | {"$oid":"582081d96b69060001fd8938"}  | 
> {"$ref":"contact","$id":{"$oid":"999cbf116b69060001fd8611"}}  |
> +--+---+
> 2 rows selected (0,563 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5088) Error when reading DBRef column

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946235#comment-15946235
 ] 

Kunal Khatua commented on DRILL-5088:
-

QA verification is not needed as added unit tests confirm the fix.

> Error when reading DBRef column
> ---
>
> Key: DRILL-5088
> URL: https://issues.apache.org/jira/browse/DRILL-5088
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
> Environment: drill 1.9.0
> mongo 3.2
>Reporter: Guillaume Champion
>Assignee: Chunhui Shi
> Fix For: 1.10.0
>
>
> In a mongo database with DBRef, when a DBRef is inserted in the first line of 
> a mongo's collection drill query failed :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> {code}
> Simple example to reproduce:
> In mongo instance
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> [Error Id: 2944d766-e483-4453-a706-3d481397b186 on Analytics-Biznet:31010] 
> (state=,code=0)
> {code}
> If the first line doesn't contain de DBRef, drill will querying correctly :
> In a mongo instance :
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8939") });
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> +--+---+
> | _id  |account   
>  |
> +--+---+
> | {"$oid":"582081d96b69060001fd8939"}  | {"$id":{}}   
>  |
> | {"$oid":"582081d96b69060001fd8938"}  | 
> {"$ref":"contact","$id":{"$oid":"999cbf116b69060001fd8611"}}  |
> +--+---+
> 2 rows selected (0,563 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5196) Could not run a single MongoDB unit test case through command line or IDE

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946236#comment-15946236
 ] 

Kunal Khatua commented on DRILL-5196:
-

QA verification is not needed as added unit tests confirm the fix.

> Could not run a single MongoDB unit test case through command line or IDE
> -
>
> Key: DRILL-5196
> URL: https://issues.apache.org/jira/browse/DRILL-5196
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Chunhui Shi
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Could not run a single MongoDB's unit test through IDE or command line. The 
> reason is when running a single test case, the MongoDB instance did not get 
> started thus a 'table not found' error for 'mongo.employee.empinfo' would be 
> raised.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5196) Could not run a single MongoDB unit test case through command line or IDE

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5196.
---

> Could not run a single MongoDB unit test case through command line or IDE
> -
>
> Key: DRILL-5196
> URL: https://issues.apache.org/jira/browse/DRILL-5196
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Chunhui Shi
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Could not run a single MongoDB's unit test through IDE or command line. The 
> reason is when running a single test case, the MongoDB instance did not get 
> started thus a 'table not found' error for 'mongo.employee.empinfo' would be 
> raised.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5318) Create a sub-operator test framework

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946215#comment-15946215
 ] 

ASF GitHub Bot commented on DRILL-5318:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/788#discussion_r108094926
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/test/ConfigBuilder.java ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ 
**/
+package org.apache.drill.test;
+
+import java.util.Properties;
+import java.util.Map.Entry;
+
+import org.apache.drill.common.config.DrillConfig;
+
+/**
+ * Builds a {@link DrillConfig} for use in tests. Use this when a config
+ * is needed by itself, separate from an embedded Drillbit.
+ */
+public class ConfigBuilder {
+
+  protected String configResource;
+  protected Properties configProps;
+
+  /**
+   * Use the given configuration properties as overrides.
+   * @param configProps a collection of config properties
+   * @return this builder
+   * @see {@link #configProperty(String, Object)}
+   */
+
+  public ConfigBuilder configProps(Properties configProps) {
+if (hasResource()) {
+  // Drill provides no constructor for this use case.
+  throw new IllegalArgumentException( "Cannot provide both a config 
resource and config properties.");
+}
+if (this.configProps == null) {
+  this.configProps = configProps;
+} else {
+  this.configProps.putAll(configProps);
+}
+return this;
+  }
+
+  /**
+   * Use the given configuration file, stored as a resource, to initialize
+   * the Drill config. Note that the resource file should have the two
+   * following settings to work as a config for an embedded Drillbit:
+   * 
+   * drill.exec.sys.store.provider.local.write : false,
+   * drill.exec.http.enabled : false
+   * 
+   * It may be more convenient to add your settings to the default
+   * config settings with {@link #configProperty(String, Object)}.
+   * @param configResource path to the file that contains the
+   * config file to be read
+   * @return this builder
+   * @see {@link #configProperty(String, Object)}
+   */
+
+  public ConfigBuilder resource(String configResource) {
+
+if (configProps != null) {
+  // Drill provides no constructor for this use case.
+  throw new IllegalArgumentException( "Cannot provide both a config 
resource and config properties.");
+}
+
+// TypeSafe gets unhappy about a leading slash, but other functions
+// require it. Silently discard the leading slash if given to
+// preserve the test writer's sanity.
+
+this.configResource = ClusterFixture.trimSlash(configResource);
+return this;
+  }
+
+  /**
+   * Add an additional boot-time property for the embedded Drillbit.
+   * @param key config property name
+   * @param value property value
+   * @return this builder
+   */
+
+  public ConfigBuilder put(String key, Object value) {
+if (hasResource()) {
+  // Drill provides no constructor for this use case.
+  throw new IllegalArgumentException( "Cannot provide both a config 
resource and config properties.");
+}
+if (configProps == null) {
+  configProps = new Properties();
+}
+configProps.put(key, value.toString());
+return this;
+  }
+
+  public DrillConfig build() {
+
+// Create a config
+// Because of the way DrillConfig works, we can set the ZK
+// connection string only if a property set is provided.
+
+if (hasResource()) {
+  return DrillConfig.create(configResource);
+} else if (configProps != null) {
+  return 

[jira] [Commented] (DRILL-5318) Create a sub-operator test framework

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946216#comment-15946216
 ] 

ASF GitHub Bot commented on DRILL-5318:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/788#discussion_r108095955
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/test/QueryBuilder.java ---
@@ -271,6 +276,91 @@ public QuerySummary run() throws Exception {
   }
 
   /**
+   * Run the query and return the first result set as a
+   * {@link DirectRowSet} object that can be inspected directly
+   * by the code using a {@link RowSetReader}.
+   * 
+   * An enhancement is to provide a way to read a series of result
+   * batches as row sets.
+   * @return a row set that represents the first batch returned from
+   * the query
+   * @throws RpcException if anything goes wrong
+   */
+
+  public DirectRowSet rowSet() throws RpcException {
+
+// Ignore all but the first non-empty batch.
+
+QueryDataBatch dataBatch = null;
+for (QueryDataBatch batch : results()) {
+  if (dataBatch == null  &&  batch.getHeader().getRowCount() != 0) {
+dataBatch = batch;
--- End diff --

break ? after first non-empty batch is found ?


> Create a sub-operator test framework
> 
>
> Key: DRILL-5318
> URL: https://issues.apache.org/jira/browse/DRILL-5318
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
> Attachments: Sub-OperatorTestFramework.pdf
>
>
> Drill provides two unit test frameworks for whole-server, SQL-based testing: 
> the original {{BaseTestQuery}} and the newer {{ClusterFixture}}. Both use the 
> {{TestBuilder}} mechanism to build system-level functional tests that run 
> queries and check results.
> Jason provided an operator-level test framework based, in part on mocks: 
> As Drill operators become more complex, we have a crying need for true 
> unit-level tests at a level below the whole system and below operators. That 
> is, we need to test the individual pieces that, together, form the operator.
> This umbrella ticket includes a number of tasks needed to create the 
> sub-operator framework. Our intention is that, over time, as we find the need 
> to revisit existing operators, or create new ones, we can employ the 
> sub-operator test framework to exercise code at a finer granularity than is 
> possible prior to this framework.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5318) Create a sub-operator test framework

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946217#comment-15946217
 ] 

ASF GitHub Bot commented on DRILL-5318:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/788#discussion_r108563545
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/test/QueryBuilder.java ---
@@ -271,6 +276,91 @@ public QuerySummary run() throws Exception {
   }
 
   /**
+   * Run the query and return the first result set as a
+   * {@link DirectRowSet} object that can be inspected directly
+   * by the code using a {@link RowSetReader}.
+   * 
+   * An enhancement is to provide a way to read a series of result
+   * batches as row sets.
+   * @return a row set that represents the first batch returned from
+   * the query
+   * @throws RpcException if anything goes wrong
+   */
+
+  public DirectRowSet rowSet() throws RpcException {
+
+// Ignore all but the first non-empty batch.
+
+QueryDataBatch dataBatch = null;
+for (QueryDataBatch batch : results()) {
+  if (dataBatch == null  &&  batch.getHeader().getRowCount() != 0) {
+dataBatch = batch;
+  } else {
+batch.release();
+  }
+}
+
+// No results?
+
+if (dataBatch == null) {
+  return null;
+}
+
+// Unload the batch and convert to a row set.
+
+final RecordBatchLoader loader = new 
RecordBatchLoader(client.allocator());
+try {
+  loader.load(dataBatch.getHeader().getDef(), dataBatch.getData());
+  dataBatch.release();
+  VectorContainer container = loader.getContainer();
+  container.setRecordCount(loader.getRecordCount());
+  return new DirectRowSet(client.allocator(), container);
+} catch (SchemaChangeException e) {
+  throw new IllegalStateException(e);
+}
+  }
+
+  /**
+   * Run the query that is expected to return (at least) one row
+   * with the only (or first) column returning a long value.
+   *
+   * @return the value of the first column of the first row
+   * @throws RpcException if anything goes wrong
+   */
+
+  public long singletonLong() throws RpcException {
+RowSet rowSet = rowSet();
+if (rowSet == null) {
+  throw new IllegalStateException("No rows returned");
+}
+RowSetReader reader = rowSet.reader();
+reader.next();
+long value = reader.column(0).getLong();
--- End diff --

Can we please refactor the common portion in singletonLong and singletonInt 
? Since I guess in future if we support retrieving more types then again it 
will be repeated?


> Create a sub-operator test framework
> 
>
> Key: DRILL-5318
> URL: https://issues.apache.org/jira/browse/DRILL-5318
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
> Attachments: Sub-OperatorTestFramework.pdf
>
>
> Drill provides two unit test frameworks for whole-server, SQL-based testing: 
> the original {{BaseTestQuery}} and the newer {{ClusterFixture}}. Both use the 
> {{TestBuilder}} mechanism to build system-level functional tests that run 
> queries and check results.
> Jason provided an operator-level test framework based, in part on mocks: 
> As Drill operators become more complex, we have a crying need for true 
> unit-level tests at a level below the whole system and below operators. That 
> is, we need to test the individual pieces that, together, form the operator.
> This umbrella ticket includes a number of tasks needed to create the 
> sub-operator framework. Our intention is that, over time, as we find the need 
> to revisit existing operators, or create new ones, we can employ the 
> sub-operator test framework to exercise code at a finer granularity than is 
> possible prior to this framework.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5275) Sort spill serialization is slow due to repeated buffer allocations

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5275.
---

> Sort spill serialization is slow due to repeated buffer allocations
> ---
>
> Key: DRILL-5275
> URL: https://issues.apache.org/jira/browse/DRILL-5275
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill provides a sort operator that spills to disk. The spill and read 
> operations use the serialization code in the 
> {{VectorAccessibleSerializable}}. This code, in turn, uses the 
> {{DrillBuf.getBytes()}} method to write to an output stream. (Yes, the "get" 
> method writes, and the "write" method reads...)
> The DrillBuf method turns around and calls the UDLE method that does:
> {code}
> byte[] tmp = new byte[length];
> PlatformDependent.copyMemory(addr(index), tmp, 0, length);
> out.write(tmp);
> {code}
> That is, for each write the code allocates a heap buffer. Since Drill buffers 
> can be quite large (4, 8, 16 MB or larger), the above rapidly fills the heap 
> and causes GC.
> The result is slow performance. On a Mac, with an SSD that can do 700 MB/s of 
> I/O, we get only about 40 MB/s. Very likely because of excessive CPU cost and 
> GC.
> The solution is to allocate a single read or write buffer, then use that same 
> buffer over and over when reading or writing. This must be done in 
> {{VectorAccessibleSerializable}} as it is a per-thread class that has 
> visibility to all the buffers to be written.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5275) Sort spill serialization is slow due to repeated buffer allocations

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946196#comment-15946196
 ] 

Kunal Khatua commented on DRILL-5275:
-

This involves primarily testing the performance of the Sort Spill feature. 
Since there is work in progress in this area and tests being written for that, 
those tests should be sufficient to quantify the benefits. 
Closing this JIRA as no specific QA verification is required for it.

> Sort spill serialization is slow due to repeated buffer allocations
> ---
>
> Key: DRILL-5275
> URL: https://issues.apache.org/jira/browse/DRILL-5275
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill provides a sort operator that spills to disk. The spill and read 
> operations use the serialization code in the 
> {{VectorAccessibleSerializable}}. This code, in turn, uses the 
> {{DrillBuf.getBytes()}} method to write to an output stream. (Yes, the "get" 
> method writes, and the "write" method reads...)
> The DrillBuf method turns around and calls the UDLE method that does:
> {code}
> byte[] tmp = new byte[length];
> PlatformDependent.copyMemory(addr(index), tmp, 0, length);
> out.write(tmp);
> {code}
> That is, for each write the code allocates a heap buffer. Since Drill buffers 
> can be quite large (4, 8, 16 MB or larger), the above rapidly fills the heap 
> and causes GC.
> The result is slow performance. On a Mac, with an SSD that can do 700 MB/s of 
> I/O, we get only about 40 MB/s. Very likely because of excessive CPU cost and 
> GC.
> The solution is to allocate a single read or write buffer, then use that same 
> buffer over and over when reading or writing. This must be done in 
> {{VectorAccessibleSerializable}} as it is a per-thread class that has 
> visibility to all the buffers to be written.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-4980) Upgrading of the approach of parquet date correctness status detection

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-4980:

Reviewer: Rahul Challapalli

> Upgrading of the approach of parquet date correctness status detection
> --
>
> Key: DRILL-4980
> URL: https://issues.apache.org/jira/browse/DRILL-4980
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Parquet
>Affects Versions: 1.9.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
> Fix For: 1.10.0
>
>
> This jira is an addition for the 
> [DRILL-4203|https://issues.apache.org/jira/browse/DRILL-4203].
> The date correctness label for the new generated parquet files should be 
> upgraded. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4980) Upgrading of the approach of parquet date correctness status detection

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946187#comment-15946187
 ] 

Kunal Khatua commented on DRILL-4980:
-

[~rkins] Assigning this to you as you have reviewed DRILL-4203 as well.

> Upgrading of the approach of parquet date correctness status detection
> --
>
> Key: DRILL-4980
> URL: https://issues.apache.org/jira/browse/DRILL-4980
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Parquet
>Affects Versions: 1.9.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
> Fix For: 1.10.0
>
>
> This jira is an addition for the 
> [DRILL-4203|https://issues.apache.org/jira/browse/DRILL-4203].
> The date correctness label for the new generated parquet files should be 
> upgraded. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4272) When sort runs out of memory and query fails, resources are seemingly not freed

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946183#comment-15946183
 ] 

Paul Rogers commented on DRILL-4272:


Bug was filed in Jan. of 2016 against 1.5. Did you try that version? Else, we 
can assume that the bug has since been fixed.

The important point is that the problem does not occur in the current version.

We can change the resolution to "not a bug" or "can'r reproduce" if that is 
more descriptive...

> When sort runs out of memory and query fails, resources are seemingly not 
> freed
> ---
>
> Key: DRILL-4272
> URL: https://issues.apache.org/jira/browse/DRILL-4272
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Execution - Relational Operators
>Affects Versions: 1.5.0
>Reporter: Victoria Markman
>Assignee: Paul Rogers
>Priority: Critical
> Fix For: 1.10.0
>
>
> Executed query11.sql from resources/Advanced/tpcds/tpcds_sf1/original/parquet
> Query runs out of memory:
> {code}
> Error: RESOURCE ERROR: One or more nodes ran out of memory while executing 
> the query.
> Unable to allocate sv2 for 32768 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 19961472
> allocator limit 2000
> Fragment 19:0
> [Error Id: 87aa32b8-17eb-488e-90cb-5f5b9aec on atsqa4-133.qa.lab:31010] 
> (state=,code=0)
> {code}
> And leaves fragments running, holding resources:
> {code}
> 2016-01-14 22:46:32,435 [Drillbit-ShutdownHook#0] INFO  
> o.apache.drill.exec.server.Drillbit - Received shutdown request.
> 2016-01-14 22:46:32,546 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - Foreman atsqa4-136.qa.lab no longer 
> active.  Cancelling fragment 2967db08-cd38-925a-4960-9e881f537af8:19:0.
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:19:0: State change requested 
> CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:19:0: Ignoring unexpected state 
> transition CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - Foreman atsqa4-136.qa.lab no longer 
> active.  Cancelling fragment 2967db08-cd38-925a-4960-9e881f537af8:17:0.
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:17:0: State change requested 
> CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:17:0: Ignoring unexpected state 
> transition CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:33,563 [BitServer-1] INFO  
> o.a.d.exec.rpc.control.ControlClient - Channel closed /10.10.88.134:59069 
> <--> atsqa4-136.qa.lab/10.10.88.136:31011.
> 2016-01-14 22:46:33,563 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:34802 <--> 
> atsqa4-136.qa.lab/10.10.88.136:31012.
> 2016-01-14 22:46:33,590 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:36937 <--> 
> atsqa4-135.qa.lab/10.10.88.135:31012.
> 2016-01-14 22:46:33,595 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:53860 <--> 
> atsqa4-133.qa.lab/10.10.88.133:31012.
> 2016-01-14 22:46:38,467 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:48276 <--> 
> atsqa4-134.qa.lab/10.10.88.134:31012.
> 2016-01-14 22:46:39,470 [pool-6-thread-1] INFO  
> o.a.drill.exec.rpc.user.UserServer - closed eventLoopGroup 
> io.netty.channel.nio.NioEventLoopGroup@6fb32dfb in 1003 ms
> 2016-01-14 22:46:39,470 [pool-6-thread-2] INFO  
> o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup 
> io.netty.channel.nio.NioEventLoopGroup@5c93dd80 in 1003 ms
> 2016-01-14 22:46:39,470 [pool-6-thread-1] INFO  
> o.a.drill.exec.service.ServiceEngine - closed userServer in 1004 ms
> 2016-01-14 22:46:39,470 [pool-6-thread-2] INFO  
> o.a.drill.exec.service.ServiceEngine - closed dataPool in 1005 ms
> 2016-01-14 22:46:39,483 [Drillbit-ShutdownHook#0] WARN  
> o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 2 
> running fragments.
> 2016-01-14 22:46:41,489 [Drillbit-ShutdownHook#0] ERROR 
> o.a.d.exec.server.BootStrapContext - Pool did not terminate
> 2016-01-14 22:46:41,498 [Drillbit-ShutdownHook#0] WARN  
> o.apache.drill.exec.server.Drillbit - Failure on close()
> java.lang.RuntimeException: Exception 

[jira] [Closed] (DRILL-4272) When sort runs out of memory and query fails, resources are seemingly not freed

[Rahul Challapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Challapalli closed DRILL-4272.


I am able to reproduce the error on the existing sort but there is no evidence 
suggesting that fragments were not cleaned up properly. I checked the logs and 
the jstack output.

> When sort runs out of memory and query fails, resources are seemingly not 
> freed
> ---
>
> Key: DRILL-4272
> URL: https://issues.apache.org/jira/browse/DRILL-4272
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Execution - Relational Operators
>Affects Versions: 1.5.0
>Reporter: Victoria Markman
>Assignee: Paul Rogers
>Priority: Critical
> Fix For: 1.10.0
>
>
> Executed query11.sql from resources/Advanced/tpcds/tpcds_sf1/original/parquet
> Query runs out of memory:
> {code}
> Error: RESOURCE ERROR: One or more nodes ran out of memory while executing 
> the query.
> Unable to allocate sv2 for 32768 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 19961472
> allocator limit 2000
> Fragment 19:0
> [Error Id: 87aa32b8-17eb-488e-90cb-5f5b9aec on atsqa4-133.qa.lab:31010] 
> (state=,code=0)
> {code}
> And leaves fragments running, holding resources:
> {code}
> 2016-01-14 22:46:32,435 [Drillbit-ShutdownHook#0] INFO  
> o.apache.drill.exec.server.Drillbit - Received shutdown request.
> 2016-01-14 22:46:32,546 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - Foreman atsqa4-136.qa.lab no longer 
> active.  Cancelling fragment 2967db08-cd38-925a-4960-9e881f537af8:19:0.
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:19:0: State change requested 
> CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:19:0: Ignoring unexpected state 
> transition CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - Foreman atsqa4-136.qa.lab no longer 
> active.  Cancelling fragment 2967db08-cd38-925a-4960-9e881f537af8:17:0.
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:17:0: State change requested 
> CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:17:0: Ignoring unexpected state 
> transition CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:33,563 [BitServer-1] INFO  
> o.a.d.exec.rpc.control.ControlClient - Channel closed /10.10.88.134:59069 
> <--> atsqa4-136.qa.lab/10.10.88.136:31011.
> 2016-01-14 22:46:33,563 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:34802 <--> 
> atsqa4-136.qa.lab/10.10.88.136:31012.
> 2016-01-14 22:46:33,590 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:36937 <--> 
> atsqa4-135.qa.lab/10.10.88.135:31012.
> 2016-01-14 22:46:33,595 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:53860 <--> 
> atsqa4-133.qa.lab/10.10.88.133:31012.
> 2016-01-14 22:46:38,467 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:48276 <--> 
> atsqa4-134.qa.lab/10.10.88.134:31012.
> 2016-01-14 22:46:39,470 [pool-6-thread-1] INFO  
> o.a.drill.exec.rpc.user.UserServer - closed eventLoopGroup 
> io.netty.channel.nio.NioEventLoopGroup@6fb32dfb in 1003 ms
> 2016-01-14 22:46:39,470 [pool-6-thread-2] INFO  
> o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup 
> io.netty.channel.nio.NioEventLoopGroup@5c93dd80 in 1003 ms
> 2016-01-14 22:46:39,470 [pool-6-thread-1] INFO  
> o.a.drill.exec.service.ServiceEngine - closed userServer in 1004 ms
> 2016-01-14 22:46:39,470 [pool-6-thread-2] INFO  
> o.a.drill.exec.service.ServiceEngine - closed dataPool in 1005 ms
> 2016-01-14 22:46:39,483 [Drillbit-ShutdownHook#0] WARN  
> o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 2 
> running fragments.
> 2016-01-14 22:46:41,489 [Drillbit-ShutdownHook#0] ERROR 
> o.a.d.exec.server.BootStrapContext - Pool did not terminate
> 2016-01-14 22:46:41,498 [Drillbit-ShutdownHook#0] WARN  
> o.apache.drill.exec.server.Drillbit - Failure on close()
> java.lang.RuntimeException: Exception while closing
> at 
> org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46)
>  

[jira] [Commented] (DRILL-5378) Put more information into SchemaChangeException when HashJoin hit SchemaChangeException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946158#comment-15946158
 ] 

ASF GitHub Bot commented on DRILL-5378:
---

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/801
  
Addressed review comments. Pls let me know if you have any further 
comments. thx. 


> Put more information into SchemaChangeException when HashJoin hit 
> SchemaChangeException
> ---
>
> Key: DRILL-5378
> URL: https://issues.apache.org/jira/browse/DRILL-5378
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>Priority: Minor
>
> HashJoin currently does not allow schema change in either build side or probe 
> side. When HashJoin hit SchemaChangeException in the middle of execution, 
> Drill reports a brief error message about SchemaChangeException, without 
> providing any information what schemas are in the incoming batches. That 
> makes hard to analyze the error, and understand what's going on. 
> It probably makes sense to put the two differing schemas in the error 
> message, so that user could get better idea about the schema change. 
> Before Drill can provide support for schema change in HashJoin, the detailed 
> error message would help user debug error. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4301) OOM : Unable to allocate sv2 for 1000 records, and not enough batchGroups to spill.

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946153#comment-15946153
 ] 

Paul Rogers commented on DRILL-4301:


Much work was done in the managed version of the external sort to tightly 
control memory and when to spill. Changed resolution version to 1.11 along with 
other managed sort fixes.

Please, if the example in the bug is wrong, find a correct example, test with 
the "old" sort to reproduce, then test with the managed version to verify a fix.

It may be possible to set up the "not enough batch groups to spill" issue under 
very peculiar circumstances; but the new code attempts to handle all of them. 
If all else fails, the new code puts a warning into the log file that an OOM is 
likely to occur if the sort is asked to sort data when there is insufficient 
memory to hold two incoming batches (which is the scenario defined here.)

Anything else you want to see?

> OOM : Unable to allocate sv2 for 1000 records, and not enough batchGroups to 
> spill.
> ---
>
> Key: DRILL-4301
> URL: https://issues.apache.org/jira/browse/DRILL-4301
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Execution - Flow
>Affects Versions: 1.5.0
> Environment: 4 node cluster
>Reporter: Khurram Faraaz
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> Query below in Functional tests, fails due to OOM 
> {code}
> select * from dfs.`/drill/testdata/metadata_caching/fewtypes_boolpartition` 
> where bool_col = true;
> {code}
> Drill version : drill-1.5.0
> JAVA_VERSION=1.8.0
> {noformat}
> version   commit_id   commit_message  commit_time build_email 
> build_time
> 1.5.0-SNAPSHOT2f0e3f27e630d5ac15cdaef808564e01708c3c55
> DRILL-4190 Don't hold on to batches from left side of merge join.   
> 20.01.2016 @ 22:30:26 UTC   Unknown 20.01.2016 @ 23:48:33 UTC
> framework/framework/resources/Functional/metadata_caching/data/bool_partition1.q
>  (connection: 808078113)
> [#1378] Query failed: 
> oadd.org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: 
> One or more nodes ran out of memory while executing the query.
> Unable to allocate sv2 for 1000 records, and not enough batchGroups to spill.
> batchGroups.size 0
> spilledBatchGroups.size 0
> allocated memory 48326272
> allocator limit 46684427
> Fragment 0:0
> [Error Id: 97d58ea3-8aff-48cf-a25e-32363b8e0ecd on drill-demod2:31010]
>   at 
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:119)
>   at 
> oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:113)
>   at 
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
>   at 
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
>   at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:67)
>   at 
> oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:374)
>   at 
> oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
>   at 
> oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:252)
>   at 
> oadd.org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
>   at 
> oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:285)
>   at 
> oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:257)
>   at 
> oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at 
> oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at 
> oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at 
> oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
>   at 
> 

[jira] [Updated] (DRILL-4301) OOM : Unable to allocate sv2 for 1000 records, and not enough batchGroups to spill.

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-4301:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> OOM : Unable to allocate sv2 for 1000 records, and not enough batchGroups to 
> spill.
> ---
>
> Key: DRILL-4301
> URL: https://issues.apache.org/jira/browse/DRILL-4301
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Execution - Flow
>Affects Versions: 1.5.0
> Environment: 4 node cluster
>Reporter: Khurram Faraaz
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> Query below in Functional tests, fails due to OOM 
> {code}
> select * from dfs.`/drill/testdata/metadata_caching/fewtypes_boolpartition` 
> where bool_col = true;
> {code}
> Drill version : drill-1.5.0
> JAVA_VERSION=1.8.0
> {noformat}
> version   commit_id   commit_message  commit_time build_email 
> build_time
> 1.5.0-SNAPSHOT2f0e3f27e630d5ac15cdaef808564e01708c3c55
> DRILL-4190 Don't hold on to batches from left side of merge join.   
> 20.01.2016 @ 22:30:26 UTC   Unknown 20.01.2016 @ 23:48:33 UTC
> framework/framework/resources/Functional/metadata_caching/data/bool_partition1.q
>  (connection: 808078113)
> [#1378] Query failed: 
> oadd.org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: 
> One or more nodes ran out of memory while executing the query.
> Unable to allocate sv2 for 1000 records, and not enough batchGroups to spill.
> batchGroups.size 0
> spilledBatchGroups.size 0
> allocated memory 48326272
> allocator limit 46684427
> Fragment 0:0
> [Error Id: 97d58ea3-8aff-48cf-a25e-32363b8e0ecd on drill-demod2:31010]
>   at 
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:119)
>   at 
> oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:113)
>   at 
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
>   at 
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
>   at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:67)
>   at 
> oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:374)
>   at 
> oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
>   at 
> oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:252)
>   at 
> oadd.org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
>   at 
> oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:285)
>   at 
> oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:257)
>   at 
> oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at 
> oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at 
> oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at 
> oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at 
> oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at 
> oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
>

[jira] [Commented] (DRILL-5378) Put more information into SchemaChangeException when HashJoin hit SchemaChangeException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946148#comment-15946148
 ] 

ASF GitHub Bot commented on DRILL-5378:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/801#discussion_r108559083
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 ---
@@ -278,7 +281,7 @@ public void setup(HashAggregate hashAggrConfig, 
HashTableConfig htConfig, Fragme
   }
 
   @Override
-  public AggOutcome doWork() {
+  public AggOutcome doWork() throws SchemaChangeException {
--- End diff --

You are right that Streaming aggregator can not handle schema change 
exception currently. 

In the current code since Drill's planner put sort operator before 
streaming aggregator, if the input has schema change, it would hit schema 
change exception in sort operator first, before come to streaming aggregator.

Anyway, I modified the code in StreamingAggregator, in case Drill planner 
generate a plan without sort for streaming aggregator.

I also simplified the change, since I realized we do not have to change in 
AggTemplate; the detailed error message could be produced in AggBatch.
  


> Put more information into SchemaChangeException when HashJoin hit 
> SchemaChangeException
> ---
>
> Key: DRILL-5378
> URL: https://issues.apache.org/jira/browse/DRILL-5378
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>Priority: Minor
>
> HashJoin currently does not allow schema change in either build side or probe 
> side. When HashJoin hit SchemaChangeException in the middle of execution, 
> Drill reports a brief error message about SchemaChangeException, without 
> providing any information what schemas are in the incoming batches. That 
> makes hard to analyze the error, and understand what's going on. 
> It probably makes sense to put the two differing schemas in the error 
> message, so that user could get better idea about the schema change. 
> Before Drill can provide support for schema change in HashJoin, the detailed 
> error message would help user debug error. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5388) Correct Parquet reader option name in documentation

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5388.
---

> Correct Parquet reader option name in documentation
> ---
>
> Key: DRILL-5388
> URL: https://issues.apache.org/jira/browse/DRILL-5388
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Bridget Bevens
>Assignee: Bridget Bevens
> Fix For: 1.10.0
>
>
> You need to update this link also.
> https://drill.apache.org/docs/parquet-format/#about-int96-support
> Parquet Format - Apache Drill
> drill.apache.org
> Configuring the Parquet Storage Format. To read or write Parquet data, you 
> need to include the Parquet format in the storage plugin format definitions.
> Yes, you are right. We need to update the documentation with 
> the correct option name.  Thanks for bringing it up.
>|
> Today, 1:57 AM
> > According to this page
> > , Drill can
> > implicitly interprets the INT96 timestamp data type in Parquet files after
> > setting the *store.parquet.int96_as_timestamp* option to *true*.
> > 
> > I believe the option name should be
> > *store.parquet.reader.int96_as_timestamp*
> >\



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (DRILL-5354) Create CTTAS Documentaion

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua reassigned DRILL-5354:
---

Assignee: Bridget Bevens
Reviewer: Khurram Faraaz

Marking [~bbevens] as Assignee as she created the documentation. (This is for 
historical tracking purposes).

[~khfaraaz] Please go through the documentation and close the JIRA is 
everything looks fine. 

> Create CTTAS Documentaion
> -
>
> Key: DRILL-5354
> URL: https://issues.apache.org/jira/browse/DRILL-5354
> Project: Apache Drill
>  Issue Type: Task
>  Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Padma Heid
>Assignee: Bridget Bevens
>Priority: Minor
> Fix For: 1.10.0
>
> Attachments: new_line.JPG, unnecessary_paragraph.JPG
>
>
> Work with Dev ,QA and PM to create user documentation for the CTTAS command.  
> When docs are posted, this link will be available: 
> https://drill.apache.org/docs/create-temporary-table-as-cttas/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5378) Put more information into SchemaChangeException when HashJoin hit SchemaChangeException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946140#comment-15946140
 ] 

ASF GitHub Bot commented on DRILL-5378:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/801#discussion_r108558271
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/exception/SchemaChangeException.java
 ---
@@ -49,4 +50,16 @@ public SchemaChangeException(String message, 
Object...objects){
   public SchemaChangeException(String message, Throwable cause, 
Object...objects){
 super(String.format(message, objects), cause);
   }
+
+  public static SchemaChangeException schemChanged(String message, 
BatchSchema priorSchema, BatchSchema newSchema) {
--- End diff --

Nice catch.  Modified it. 


> Put more information into SchemaChangeException when HashJoin hit 
> SchemaChangeException
> ---
>
> Key: DRILL-5378
> URL: https://issues.apache.org/jira/browse/DRILL-5378
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>Priority: Minor
>
> HashJoin currently does not allow schema change in either build side or probe 
> side. When HashJoin hit SchemaChangeException in the middle of execution, 
> Drill reports a brief error message about SchemaChangeException, without 
> providing any information what schemas are in the incoming batches. That 
> makes hard to analyze the error, and understand what's going on. 
> It probably makes sense to put the two differing schemas in the error 
> message, so that user could get better idea about the schema change. 
> Before Drill can provide support for schema change in HashJoin, the detailed 
> error message would help user debug error. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4272) When sort runs out of memory and query fails, resources are seemingly not freed

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946137#comment-15946137
 ] 

Paul Rogers commented on DRILL-4272:


The gist of the "managed sort" was to fix this issue. The code fixes were part 
of the many commits for this issue. Sorry, it is not always easy to map PRs 
exactly to Jira tickets; sometimes a single fix resolves multiple tickets. 
That's why this one was changed to a sub-task.

> When sort runs out of memory and query fails, resources are seemingly not 
> freed
> ---
>
> Key: DRILL-4272
> URL: https://issues.apache.org/jira/browse/DRILL-4272
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Execution - Relational Operators
>Affects Versions: 1.5.0
>Reporter: Victoria Markman
>Assignee: Paul Rogers
>Priority: Critical
> Fix For: 1.10.0
>
>
> Executed query11.sql from resources/Advanced/tpcds/tpcds_sf1/original/parquet
> Query runs out of memory:
> {code}
> Error: RESOURCE ERROR: One or more nodes ran out of memory while executing 
> the query.
> Unable to allocate sv2 for 32768 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 19961472
> allocator limit 2000
> Fragment 19:0
> [Error Id: 87aa32b8-17eb-488e-90cb-5f5b9aec on atsqa4-133.qa.lab:31010] 
> (state=,code=0)
> {code}
> And leaves fragments running, holding resources:
> {code}
> 2016-01-14 22:46:32,435 [Drillbit-ShutdownHook#0] INFO  
> o.apache.drill.exec.server.Drillbit - Received shutdown request.
> 2016-01-14 22:46:32,546 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - Foreman atsqa4-136.qa.lab no longer 
> active.  Cancelling fragment 2967db08-cd38-925a-4960-9e881f537af8:19:0.
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:19:0: State change requested 
> CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:19:0: Ignoring unexpected state 
> transition CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - Foreman atsqa4-136.qa.lab no longer 
> active.  Cancelling fragment 2967db08-cd38-925a-4960-9e881f537af8:17:0.
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:17:0: State change requested 
> CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:17:0: Ignoring unexpected state 
> transition CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:33,563 [BitServer-1] INFO  
> o.a.d.exec.rpc.control.ControlClient - Channel closed /10.10.88.134:59069 
> <--> atsqa4-136.qa.lab/10.10.88.136:31011.
> 2016-01-14 22:46:33,563 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:34802 <--> 
> atsqa4-136.qa.lab/10.10.88.136:31012.
> 2016-01-14 22:46:33,590 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:36937 <--> 
> atsqa4-135.qa.lab/10.10.88.135:31012.
> 2016-01-14 22:46:33,595 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:53860 <--> 
> atsqa4-133.qa.lab/10.10.88.133:31012.
> 2016-01-14 22:46:38,467 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:48276 <--> 
> atsqa4-134.qa.lab/10.10.88.134:31012.
> 2016-01-14 22:46:39,470 [pool-6-thread-1] INFO  
> o.a.drill.exec.rpc.user.UserServer - closed eventLoopGroup 
> io.netty.channel.nio.NioEventLoopGroup@6fb32dfb in 1003 ms
> 2016-01-14 22:46:39,470 [pool-6-thread-2] INFO  
> o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup 
> io.netty.channel.nio.NioEventLoopGroup@5c93dd80 in 1003 ms
> 2016-01-14 22:46:39,470 [pool-6-thread-1] INFO  
> o.a.drill.exec.service.ServiceEngine - closed userServer in 1004 ms
> 2016-01-14 22:46:39,470 [pool-6-thread-2] INFO  
> o.a.drill.exec.service.ServiceEngine - closed dataPool in 1005 ms
> 2016-01-14 22:46:39,483 [Drillbit-ShutdownHook#0] WARN  
> o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 2 
> running fragments.
> 2016-01-14 22:46:41,489 [Drillbit-ShutdownHook#0] ERROR 
> o.a.d.exec.server.BootStrapContext - Pool did not terminate
> 2016-01-14 22:46:41,498 [Drillbit-ShutdownHook#0] WARN  
> o.apache.drill.exec.server.Drillbit - Failure on close()
> java.lang.RuntimeException: Exception while closing
> at 

[jira] [Commented] (DRILL-5388) Correct Parquet reader option name in documentation

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946133#comment-15946133
 ] 

Kunal Khatua commented on DRILL-5388:
-

Verified changes on the website

> Correct Parquet reader option name in documentation
> ---
>
> Key: DRILL-5388
> URL: https://issues.apache.org/jira/browse/DRILL-5388
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Bridget Bevens
>Assignee: Bridget Bevens
> Fix For: 1.10.0
>
>
> You need to update this link also.
> https://drill.apache.org/docs/parquet-format/#about-int96-support
> Parquet Format - Apache Drill
> drill.apache.org
> Configuring the Parquet Storage Format. To read or write Parquet data, you 
> need to include the Parquet format in the storage plugin format definitions.
> Yes, you are right. We need to update the documentation with 
> the correct option name.  Thanks for bringing it up.
>|
> Today, 1:57 AM
> > According to this page
> > , Drill can
> > implicitly interprets the INT96 timestamp data type in Parquet files after
> > setting the *store.parquet.int96_as_timestamp* option to *true*.
> > 
> > I believe the option name should be
> > *store.parquet.reader.int96_as_timestamp*
> >\



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5254) Enhance default reduction factors in optimizer

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-5254:

   Labels: performance tuning  (was: )
Fix Version/s: (was: 1.10.0)
   Future

Moving this to Future target although it has been marked as _Resolved_ with a 
_Won't_ _Fix_ resolution.

Applying _performance_ and _tuning_ labels for tracking purposes.

> Enhance default reduction factors in optimizer
> --
>
> Key: DRILL-5254
> URL: https://issues.apache.org/jira/browse/DRILL-5254
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: performance, tuning
> Fix For: Future
>
>
> Drill uses Calcite for query parsing and optimization. Drill uses Calcite's 
> default selectivity (reduction factor) rules to compute the number of rows 
> removed by a filter.
> The default rules appear to be overly aggressive in estimating reductions. In 
> a production use case, an input with 4 billion rows was estimated to return 
> just 40K rows from a filter. That is, the filter estimated a 1/1,000,000 
> reduction in rows. As it turns out, the actual reduction was closer to 1/2.
> The result was that the planner compared the expected 40K rows against 
> another input of 2.5 million rows, and decided the 40K rows would be best on 
> the build side of a hash join. When confronted with the actual 3 billion 
> rows, the hash join ran out of memory.
> The moral of the story is that, in Drill, it is worth being conservative when 
> planning for memory-intensive operations.
> The (sanitized) filter is the following, annotated with (a guess at) the 
> default reduction factors in each term:
> {code}
> col1_s20 in ('Value1','Value2','Value3','Value4',
>  'Value5','Value6','Value7','Value8','Value9') -- 25%
> AND col2_i <=3 -- 25%
> AND col3_s1 = 'Y' -- 15%
> AND col4_s1 = 'Y' -- 15%
> AND col5_s6 not like '%str1%' -- 25%
> AND col5_s6 not like '%str2%' -- 25%
> AND col5_s6 not like '%str3%' -- 25%
> AND col5_s6 not like '%str4%' -- 25%
> {code}
> Total reduction is something like:
> {code}
> .25 * .25 * .15 ^ 2 * .25 ^ 4 = 0.05
> {code}
> Filter estimation is a known hard problem. In general, one needs statistics 
> and other data, and even then the estimates are just guesses.
> Still it is possible to ensure that the defaults are at least unbiased. That 
> is if we assume that the probability of A LIKE B being 25%, then the 
> probability of A NOT LIKE B should be 75%, not also 25%.
> This JIRA suggests creating an experimental set of defaults based on the 
> "core" Calcite defaults, but with other reduction factors derived using the 
> laws of probability. In particular:
> || Operator || Revised || Explanation || Calcite Default
> | = | 0.15 | Default in Calcite | 0.15
> | <> | 0.85 | 1 - p(=) | 0.5
> | < | 0.425 | p(<>) / 2 | 0.5
> | > | 0.425 | p(<>) / 2 | 0.5
> | <= | 0.575 | p(<) + p(=) | 0.5
> | >= | 0.575 | p(>) + p(=) | 0.5
> | LIKE | 0.25 | Default in Calcite | 0.25
> | NOT LIKE | 0.75 | 1 - p(LIKE) | 0.25
> | NOT NULL | 0.90 | Default in Calcite | 0.90
> | IS NULL | 0.10 | 1 - p(NOT NULL) | 0.25
> | val IS TRUE | 0.5 | 1 / 2 | 0.25
> | val IS FALSE | 0.5 | 1 / 2 | 0.25
> | val IS NOT TRUE | 0.55 | (1 + p(NULL)) / 2 | 0.25
> | val IS NOT FALSE | 0.55 | (1 + p(NULL)) / 2 | 0.25
> | expr IS TRUE | Varies | p(expr) | 0.25
> | expr IS FALSE | Varies | 1 - p(expr) | 0.25
> | expr IS NOT TRUE | Varies | p(expr IS FALSE) | 0.25
> | expr IS NOT FALSE | Varies | p(expr IS TRUE) | 0.25
> | A OR B | Varies | min(p(A) + p(B), 0.5) | 0.5
> | A AND B | Varies | p(A ^ B) = p(A) * p(B) | Same
> | IN (a) | 0.15 | p(=) | 0.25
> | x IN (a, b, c, ...) | Varies | p(x = a v x = b v x = c v ...) | 0.25
> | NOT A | Varies | 1 - p(A) | 0.25
> | BETWEEN a AND b | 0.33 | p(<= ^ >=) | 0.25
> | NOT BETWEEN a AND b | 0.67 | 1 - p(BETWEEN) | 0.25
> The Calcite defaults were identified by inspection and verified by tests. The 
> Calcite rules make sense if one considers conditional probability: that the 
> user applied a particular expression to the data with the expectation that 
> given that data set, the expression matches 25% of the rows.
> The probability of the IS NOT TRUE statement assumes the presence of nulls, 
> while IS TRUE does not. This is an example of conditional probability. If the 
> user asks if something is TRUE, we can assume that they are nor worried about 
> nulls. If they ask if it is NOT FALSE, the only justification for such syntax 
> is to include nulls. Since we assume nulls (when present) make up 10% of 
> data, in such a case, half the non-null data is true, half is false.
> Note that, in the proposed implementation, each expression in an IS TRUE or 
> similar 

[jira] [Commented] (DRILL-4272) When sort runs out of memory and query fails, resources are seemingly not freed

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946128#comment-15946128
 ] 

Rahul Challapalli commented on DRILL-4272:
--

[~Paul.Rogers] I do not see any PR associated with this jira. I am wondering 
why it is marked as closed. Were you unable to reproduce this issue with the 
existing sort based on the criteria mentioned by Vicky?

> When sort runs out of memory and query fails, resources are seemingly not 
> freed
> ---
>
> Key: DRILL-4272
> URL: https://issues.apache.org/jira/browse/DRILL-4272
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Execution - Relational Operators
>Affects Versions: 1.5.0
>Reporter: Victoria Markman
>Assignee: Paul Rogers
>Priority: Critical
> Fix For: 1.10.0
>
>
> Executed query11.sql from resources/Advanced/tpcds/tpcds_sf1/original/parquet
> Query runs out of memory:
> {code}
> Error: RESOURCE ERROR: One or more nodes ran out of memory while executing 
> the query.
> Unable to allocate sv2 for 32768 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 19961472
> allocator limit 2000
> Fragment 19:0
> [Error Id: 87aa32b8-17eb-488e-90cb-5f5b9aec on atsqa4-133.qa.lab:31010] 
> (state=,code=0)
> {code}
> And leaves fragments running, holding resources:
> {code}
> 2016-01-14 22:46:32,435 [Drillbit-ShutdownHook#0] INFO  
> o.apache.drill.exec.server.Drillbit - Received shutdown request.
> 2016-01-14 22:46:32,546 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - Foreman atsqa4-136.qa.lab no longer 
> active.  Cancelling fragment 2967db08-cd38-925a-4960-9e881f537af8:19:0.
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:19:0: State change requested 
> CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:19:0: Ignoring unexpected state 
> transition CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - Foreman atsqa4-136.qa.lab no longer 
> active.  Cancelling fragment 2967db08-cd38-925a-4960-9e881f537af8:17:0.
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:17:0: State change requested 
> CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:32,547 [Curator-ServiceCache-0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2967db08-cd38-925a-4960-9e881f537af8:17:0: Ignoring unexpected state 
> transition CANCELLATION_REQUESTED --> CANCELLATION_REQUESTED
> 2016-01-14 22:46:33,563 [BitServer-1] INFO  
> o.a.d.exec.rpc.control.ControlClient - Channel closed /10.10.88.134:59069 
> <--> atsqa4-136.qa.lab/10.10.88.136:31011.
> 2016-01-14 22:46:33,563 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:34802 <--> 
> atsqa4-136.qa.lab/10.10.88.136:31012.
> 2016-01-14 22:46:33,590 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:36937 <--> 
> atsqa4-135.qa.lab/10.10.88.135:31012.
> 2016-01-14 22:46:33,595 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:53860 <--> 
> atsqa4-133.qa.lab/10.10.88.133:31012.
> 2016-01-14 22:46:38,467 [BitClient-1] INFO  
> o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.88.134:48276 <--> 
> atsqa4-134.qa.lab/10.10.88.134:31012.
> 2016-01-14 22:46:39,470 [pool-6-thread-1] INFO  
> o.a.drill.exec.rpc.user.UserServer - closed eventLoopGroup 
> io.netty.channel.nio.NioEventLoopGroup@6fb32dfb in 1003 ms
> 2016-01-14 22:46:39,470 [pool-6-thread-2] INFO  
> o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup 
> io.netty.channel.nio.NioEventLoopGroup@5c93dd80 in 1003 ms
> 2016-01-14 22:46:39,470 [pool-6-thread-1] INFO  
> o.a.drill.exec.service.ServiceEngine - closed userServer in 1004 ms
> 2016-01-14 22:46:39,470 [pool-6-thread-2] INFO  
> o.a.drill.exec.service.ServiceEngine - closed dataPool in 1005 ms
> 2016-01-14 22:46:39,483 [Drillbit-ShutdownHook#0] WARN  
> o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 2 
> running fragments.
> 2016-01-14 22:46:41,489 [Drillbit-ShutdownHook#0] ERROR 
> o.a.d.exec.server.BootStrapContext - Pool did not terminate
> 2016-01-14 22:46:41,498 [Drillbit-ShutdownHook#0] WARN  
> o.apache.drill.exec.server.Drillbit - Failure on close()
> java.lang.RuntimeException: Exception while closing
> at 
> 

[jira] [Commented] (DRILL-4301) OOM : Unable to allocate sv2 for 1000 records, and not enough batchGroups to spill.

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946117#comment-15946117
 ] 

Rahul Challapalli commented on DRILL-4301:
--

Like hakim noted the query that [~khfaraaz] reported in the description could 
not possibly have caused this failure. I don't know what query [~Paul.Rogers] 
used for testing this. 

Irrespective of any of the above we should not close this jira as managed sort 
is not enabled by default in 1.10.0, unless Paul made changes which affect both 
the existing sort and the managed sort.

> OOM : Unable to allocate sv2 for 1000 records, and not enough batchGroups to 
> spill.
> ---
>
> Key: DRILL-4301
> URL: https://issues.apache.org/jira/browse/DRILL-4301
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Execution - Flow
>Affects Versions: 1.5.0
> Environment: 4 node cluster
>Reporter: Khurram Faraaz
>Assignee: Paul Rogers
> Fix For: 1.10.0
>
>
> Query below in Functional tests, fails due to OOM 
> {code}
> select * from dfs.`/drill/testdata/metadata_caching/fewtypes_boolpartition` 
> where bool_col = true;
> {code}
> Drill version : drill-1.5.0
> JAVA_VERSION=1.8.0
> {noformat}
> version   commit_id   commit_message  commit_time build_email 
> build_time
> 1.5.0-SNAPSHOT2f0e3f27e630d5ac15cdaef808564e01708c3c55
> DRILL-4190 Don't hold on to batches from left side of merge join.   
> 20.01.2016 @ 22:30:26 UTC   Unknown 20.01.2016 @ 23:48:33 UTC
> framework/framework/resources/Functional/metadata_caching/data/bool_partition1.q
>  (connection: 808078113)
> [#1378] Query failed: 
> oadd.org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: 
> One or more nodes ran out of memory while executing the query.
> Unable to allocate sv2 for 1000 records, and not enough batchGroups to spill.
> batchGroups.size 0
> spilledBatchGroups.size 0
> allocated memory 48326272
> allocator limit 46684427
> Fragment 0:0
> [Error Id: 97d58ea3-8aff-48cf-a25e-32363b8e0ecd on drill-demod2:31010]
>   at 
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:119)
>   at 
> oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:113)
>   at 
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
>   at 
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
>   at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:67)
>   at 
> oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:374)
>   at 
> oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
>   at 
> oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:252)
>   at 
> oadd.org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
>   at 
> oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:285)
>   at 
> oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:257)
>   at 
> oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at 
> oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at 
> oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at 
> oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>   at 
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>   at 
> oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>   at 
> 

[jira] [Updated] (DRILL-5080) Create a memory-managed version of the External Sort operator

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5080:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> Create a memory-managed version of the External Sort operator
> -
>
> Key: DRILL-5080
> URL: https://issues.apache.org/jira/browse/DRILL-5080
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.11.0
>
> Attachments: ManagedExternalSortDesign.pdf
>
>
> We propose to create a "managed" version of the external sort operator that 
> works to a clearly-defined memory limit. Attached is a design specification 
> for the work.
> The project will include fixing a number of bugs related to the external 
> sort, include as sub-tasks of this umbrella task.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5112) Unit tests derived from PopUnitTestBase fail in IDE due to config errors

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946108#comment-15946108
 ] 

Kunal Khatua commented on DRILL-5112:
-

Closing as no QA verification is required

> Unit tests derived from PopUnitTestBase fail in IDE due to config errors
> 
>
> Key: DRILL-5112
> URL: https://issues.apache.org/jira/browse/DRILL-5112
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill provides a wide variety of unit tests. Many derive from 
> {{PopUnitTestBase}} to test the Physical OPerators.
> The tests use a default configuration:
> {code}
> protected static DrillConfig CONFIG;
>   @BeforeClass
>   public static void setup() {
> CONFIG = DrillConfig.create();
>   }
> {code}
> The tests rely on config settings specified in the {{pom.xml}} file (see note 
> below.) When run in Eclipse, no such config exists, so the tests use only the 
> default config. The defaults allow a web server to be started.
> Many tests start multiple Drillbits using the above config. When this occurs, 
> each tries to start a web server. The second one fails because the HTTP port 
> is already in use.
> The solution is to initialize the config using the same settings as used in 
> the {{BaseTestQuery}} test case: the unit tests then work fine in Eclipse.
> As an aside, having multiple ways to set up the Drill config (and other 
> items) leads to much wasted time as each engineer must learn the quirks of 
> each test hierarchy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5070) Code gen: create methods in fixed order to allow test verification

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5070.
---

> Code gen: create methods in fixed order to allow test verification
> --
>
> Key: DRILL-5070
> URL: https://issues.apache.org/jira/browse/DRILL-5070
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.10.0
>
>
> A handy technique in testing is to compare generated code against a "golden" 
> copy that defines the expected results. However, at present, Drill generates 
> code using the method order returned by {{Class.getDeclaredMethods}}, but 
> this method makes no guarantee about the order of the methods. The order 
> varies from one run to the next. There is some evidence [this 
> link|http://stackoverflow.com/questions/28585843/java-reflection-getdeclaredmethods-in-declared-order-strange-behaviour]
>  that order can vary even within a single run, though a quick test was unable 
> to reproduce this case.
> If method order does indeed vary within a single run, then the order can 
> impact the Drill code cache since it compares the sources from two different 
> generation events to detect duplicate code.
> This issue appeared when attempting to modify tests to capture generated code 
> for comparison to future results. Even a simple generated case from 
> {{ExpressionTest.testBasicExpression()}} that generates {{if(true) then 1 
> else 0 end}} (all constants) produced methods in different orders on each 
> test run.
> The fix is simple, in the {{SignatureHolder}} constructor, sort methods by 
> name after retrieving them from the class. The sort ensures that method order 
> is deterministic. Fortunately, the number of methods is small, so the sort 
> step adds little cost.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5108) Reduce output from Maven git-commit-id-plugin

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946107#comment-15946107
 ] 

Kunal Khatua commented on DRILL-5108:
-

Closing as no QA verification is required

> Reduce output from Maven git-commit-id-plugin
> -
>
> Key: DRILL-5108
> URL: https://issues.apache.org/jira/browse/DRILL-5108
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> The git-commit-id-plugin grabs information from Git to display during a 
> build. It prints many e-mail addresses and other generic project information. 
> As part of the effort to trim down unit test output, we propose to turn off 
> the verbose output from this plugin.
> Specific change:
> {code}
>   
> pl.project13.maven
> git-commit-id-plugin
> ...
> 
>  false
> {code}
> That is, change the verbose setting from true to false.
> In the unlikely event that some build process depends on the verbose output, 
> we can make the setting a configurable parameter, defaulting to false.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5113) Upgrade Maven RAT plugin to avoid annoying XML errors

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5113.
---

> Upgrade Maven RAT plugin to avoid annoying XML errors
> -
>
> Key: DRILL-5113
> URL: https://issues.apache.org/jira/browse/DRILL-5113
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Build Drill with most Maven logging turned off. On every sub-project you will 
> see the following:
> {code}
> Compiler warnings:
>   WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> [INFO] Starting audit...
> Audit done.
> {code}
> The warning is a known issue with Java: 
> http://bugs.java.com/view_bug.do?bug_id=8016153
> The RAT folks seem to have done a patch: version 0.12 of the plugin no longer 
> has the warning. Upgrade Drill's {{pom.xml}} file to use this version instead 
> of the anonymous version currently used.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5113) Upgrade Maven RAT plugin to avoid annoying XML errors

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946109#comment-15946109
 ] 

Kunal Khatua commented on DRILL-5113:
-

Closing as no QA verification is required

> Upgrade Maven RAT plugin to avoid annoying XML errors
> -
>
> Key: DRILL-5113
> URL: https://issues.apache.org/jira/browse/DRILL-5113
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Build Drill with most Maven logging turned off. On every sub-project you will 
> see the following:
> {code}
> Compiler warnings:
>   WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> [INFO] Starting audit...
> Audit done.
> {code}
> The warning is a known issue with Java: 
> http://bugs.java.com/view_bug.do?bug_id=8016153
> The RAT folks seem to have done a patch: version 0.12 of the plugin no longer 
> has the warning. Upgrade Drill's {{pom.xml}} file to use this version instead 
> of the anonymous version currently used.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5112) Unit tests derived from PopUnitTestBase fail in IDE due to config errors

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5112.
---

> Unit tests derived from PopUnitTestBase fail in IDE due to config errors
> 
>
> Key: DRILL-5112
> URL: https://issues.apache.org/jira/browse/DRILL-5112
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill provides a wide variety of unit tests. Many derive from 
> {{PopUnitTestBase}} to test the Physical OPerators.
> The tests use a default configuration:
> {code}
> protected static DrillConfig CONFIG;
>   @BeforeClass
>   public static void setup() {
> CONFIG = DrillConfig.create();
>   }
> {code}
> The tests rely on config settings specified in the {{pom.xml}} file (see note 
> below.) When run in Eclipse, no such config exists, so the tests use only the 
> default config. The defaults allow a web server to be started.
> Many tests start multiple Drillbits using the above config. When this occurs, 
> each tries to start a web server. The second one fails because the HTTP port 
> is already in use.
> The solution is to initialize the config using the same settings as used in 
> the {{BaseTestQuery}} test case: the unit tests then work fine in Eclipse.
> As an aside, having multiple ways to set up the Drill config (and other 
> items) leads to much wasted time as each engineer must learn the quirks of 
> each test hierarchy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5108) Reduce output from Maven git-commit-id-plugin

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5108.
---

> Reduce output from Maven git-commit-id-plugin
> -
>
> Key: DRILL-5108
> URL: https://issues.apache.org/jira/browse/DRILL-5108
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> The git-commit-id-plugin grabs information from Git to display during a 
> build. It prints many e-mail addresses and other generic project information. 
> As part of the effort to trim down unit test output, we propose to turn off 
> the verbose output from this plugin.
> Specific change:
> {code}
>   
> pl.project13.maven
> git-commit-id-plugin
> ...
> 
>  false
> {code}
> That is, change the verbose setting from true to false.
> In the unlikely event that some build process depends on the verbose output, 
> we can make the setting a configurable parameter, defaulting to false.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5070) Code gen: create methods in fixed order to allow test verification

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946110#comment-15946110
 ] 

Kunal Khatua commented on DRILL-5070:
-

Closing as no QA verification is required

> Code gen: create methods in fixed order to allow test verification
> --
>
> Key: DRILL-5070
> URL: https://issues.apache.org/jira/browse/DRILL-5070
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.10.0
>
>
> A handy technique in testing is to compare generated code against a "golden" 
> copy that defines the expected results. However, at present, Drill generates 
> code using the method order returned by {{Class.getDeclaredMethods}}, but 
> this method makes no guarantee about the order of the methods. The order 
> varies from one run to the next. There is some evidence [this 
> link|http://stackoverflow.com/questions/28585843/java-reflection-getdeclaredmethods-in-declared-order-strange-behaviour]
>  that order can vary even within a single run, though a quick test was unable 
> to reproduce this case.
> If method order does indeed vary within a single run, then the order can 
> impact the Drill code cache since it compares the sources from two different 
> generation events to detect duplicate code.
> This issue appeared when attempting to modify tests to capture generated code 
> for comparison to future results. Even a simple generated case from 
> {{ExpressionTest.testBasicExpression()}} that generates {{if(true) then 1 
> else 0 end}} (all constants) produced methods in different orders on each 
> test run.
> The fix is simple, in the {{SignatureHolder}} constructor, sort methods by 
> name after retrieving them from the class. The sort ensures that method order 
> is deterministic. Fortunately, the number of methods is small, so the sort 
> step adds little cost.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5116) Enable generated code debugging in each Drill operator

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946103#comment-15946103
 ] 

Kunal Khatua commented on DRILL-5116:
-

Closing as no QA verification is required

> Enable generated code debugging in each Drill operator
> --
>
> Key: DRILL-5116
> URL: https://issues.apache.org/jira/browse/DRILL-5116
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> DRILL-5052 adds the ability to debug generated code. Some of the code 
> generated by Drill's operators has minor problems when compiled directly 
> using the new technique. These issues are ignore by the byte-code-merge 
> technique uses in production. This ticket asks to try the DRILL-5052 feature 
> in each operator, clean up any minor problems, and ensure each operator 
> generates code suitable for debugging. Use the new 
> {{CodeGenerator.plainOldJavaCapable()}} method to mark each generated class 
> as ready for "plain-old Java" code gen.
> The advantages of this feature are two:
> 1. Ability to step through the generated code to increase understanding of 
> existing operators and to ease development of improvements to existing 
> operators and of any new operators we choose to create.
> 2. Open the door to experimenting with how to improve performance of the 
> generated code.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5126) Provide simplified, unified "cluster fixture" for tests

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5126.
---

> Provide simplified, unified "cluster fixture" for tests
> ---
>
> Key: DRILL-5126
> URL: https://issues.apache.org/jira/browse/DRILL-5126
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill provides a robust selection of test frameworks that have evolved to 
> satisfy the needs of a variety of test cases. For newbies, however, the 
> result is a bewildering array of ways to do basically the same thing: set up 
> an embedded Drill cluster, run queries and check results.
> Further, some key test settings are distributed: some are in the pom.xml 
> file, some in config files stored as resources, some in hard-coded settings 
> in base test classes.
> Also, some test base classes helpfully set up a test cluster, but then 
> individual tests need a different config, so they immediately tear down the 
> default cluster and create a new one.
> This ticket proposes a new test framework, available for new tests, that 
> combines the best of the existing test frameworks into a single, easy-to-use 
> package.
> * Builder for the cluster
> * Accept config-time options
> * Accept run-time session and system options
> * Specify number of Drillbits
> * Simplified API for the most common options
> * AutoCloseable for use in try-with-resources statements
> * Integration with existing test builder classes
> And so on.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5017) Config param drill.exec.sort.external.batch.size is not used

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5017.
---

> Config param drill.exec.sort.external.batch.size is not used
> 
>
> Key: DRILL-5017
> URL: https://issues.apache.org/jira/browse/DRILL-5017
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.10.0
>
>
> The Drill config file defines the {{drill.exec.sort.external.batch.size}} 
> parameter, as does {{ExecConstants}}:
> {code}
>   String EXTERNAL_SORT_TARGET_BATCH_SIZE = 
> "drill.exec.sort.external.batch.size";
> {code}
> However, this parameter is never used. It seems to be a duplicate of:
> {code}
>   String EXTERNAL_SORT_TARGET_SPILL_BATCH_SIZE = 
> "drill.exec.sort.external.spill.batch.size";
> {code}
> Which, itself, is never used.
> Remove these parameters from {{ExecConstants}}, {{drill-module.conf}}, 
> {{drill-override-example.conf}} (if they appear in those files) and from the 
> documentation (if they appear in the docs.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5258) Allow "extended" mock tables access from SQL queries

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5258.
---

> Allow "extended" mock tables access from SQL queries
> 
>
> Key: DRILL-5258
> URL: https://issues.apache.org/jira/browse/DRILL-5258
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> DRILL-5152 provided a simple way to generate sample data in SQL using a new, 
> simplified version of the mock data generator. This approach is very 
> convenient, but is inherently limited. For example, the limited syntax 
> available in SQL does not encoding much information about columns such as 
> repeat count, data generator or so on. The simple SQL approach does not allow 
> generating multiple groups of data.
> However, all these features are present in the original mock data source via 
> a special JSON configuration file. Previously, only physical plans could 
> access that extended syntax.
> This ticket requests a SQL interface to the extended mock data source:
> {code}
> SELECT * FROM `mock`.`example/mock-options.json`
> {code}
> Mock data source options are always stored as a JSON file. Since the existing 
> mock data generator for SQL never uses JSON files, a simple rule is that if 
> the table name ends in ".json" then it is a specification, else the 
> information is encoded in table and column names.
> The format of the data generation syntax is documented in the mock data 
> source classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5126) Provide simplified, unified "cluster fixture" for tests

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946102#comment-15946102
 ] 

Kunal Khatua commented on DRILL-5126:
-

Closing as no QA verification is required

> Provide simplified, unified "cluster fixture" for tests
> ---
>
> Key: DRILL-5126
> URL: https://issues.apache.org/jira/browse/DRILL-5126
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill provides a robust selection of test frameworks that have evolved to 
> satisfy the needs of a variety of test cases. For newbies, however, the 
> result is a bewildering array of ways to do basically the same thing: set up 
> an embedded Drill cluster, run queries and check results.
> Further, some key test settings are distributed: some are in the pom.xml 
> file, some in config files stored as resources, some in hard-coded settings 
> in base test classes.
> Also, some test base classes helpfully set up a test cluster, but then 
> individual tests need a different config, so they immediately tear down the 
> default cluster and create a new one.
> This ticket proposes a new test framework, available for new tests, that 
> combines the best of the existing test frameworks into a single, easy-to-use 
> package.
> * Builder for the cluster
> * Accept config-time options
> * Accept run-time session and system options
> * Specify number of Drillbits
> * Simplified API for the most common options
> * AutoCloseable for use in try-with-resources statements
> * Integration with existing test builder classes
> And so on.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5017) Config param drill.exec.sort.external.batch.size is not used

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946105#comment-15946105
 ] 

Kunal Khatua commented on DRILL-5017:
-

Closing as no QA verification is required

> Config param drill.exec.sort.external.batch.size is not used
> 
>
> Key: DRILL-5017
> URL: https://issues.apache.org/jira/browse/DRILL-5017
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.10.0
>
>
> The Drill config file defines the {{drill.exec.sort.external.batch.size}} 
> parameter, as does {{ExecConstants}}:
> {code}
>   String EXTERNAL_SORT_TARGET_BATCH_SIZE = 
> "drill.exec.sort.external.batch.size";
> {code}
> However, this parameter is never used. It seems to be a duplicate of:
> {code}
>   String EXTERNAL_SORT_TARGET_SPILL_BATCH_SIZE = 
> "drill.exec.sort.external.spill.batch.size";
> {code}
> Which, itself, is never used.
> Remove these parameters from {{ExecConstants}}, {{drill-module.conf}}, 
> {{drill-override-example.conf}} (if they appear in those files) and from the 
> documentation (if they appear in the docs.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5116) Enable generated code debugging in each Drill operator

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5116.
---

> Enable generated code debugging in each Drill operator
> --
>
> Key: DRILL-5116
> URL: https://issues.apache.org/jira/browse/DRILL-5116
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> DRILL-5052 adds the ability to debug generated code. Some of the code 
> generated by Drill's operators has minor problems when compiled directly 
> using the new technique. These issues are ignore by the byte-code-merge 
> technique uses in production. This ticket asks to try the DRILL-5052 feature 
> in each operator, clean up any minor problems, and ensure each operator 
> generates code suitable for debugging. Use the new 
> {{CodeGenerator.plainOldJavaCapable()}} method to mark each generated class 
> as ready for "plain-old Java" code gen.
> The advantages of this feature are two:
> 1. Ability to step through the generated code to increase understanding of 
> existing operators and to ease development of improvements to existing 
> operators and of any new operators we choose to create.
> 2. Open the door to experimenting with how to improve performance of the 
> generated code.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5260) Refinements to new "Cluster Fixture" test framework

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946104#comment-15946104
 ] 

Kunal Khatua commented on DRILL-5260:
-

Closing as no QA verification is required

> Refinements to new "Cluster Fixture" test framework
> ---
>
> Key: DRILL-5260
> URL: https://issues.apache.org/jira/browse/DRILL-5260
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Roll-up of a number of enhancements to the cluster fixture framework.
> * Config option to suppress printing of CSV and other output. (Allows 
> printing for single tests, not printing when running from Maven.)
> * Parsing of query profiles to extract plan and run time information.
> * Fix bug in log fixture when enabling logging for a package.
> * Improved ZK support.
> * Set up the new CTTAS default temporary workspace for tests.
> * Revise TestDrillbitResiliance to use the new framework.
> * Revise TestWindowFrame to to use the new framework.
> * Revise TestMergeJoinWithSchemaChanges to use the new framework.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5258) Allow "extended" mock tables access from SQL queries

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946106#comment-15946106
 ] 

Kunal Khatua commented on DRILL-5258:
-

Closing as no QA verification is required

> Allow "extended" mock tables access from SQL queries
> 
>
> Key: DRILL-5258
> URL: https://issues.apache.org/jira/browse/DRILL-5258
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> DRILL-5152 provided a simple way to generate sample data in SQL using a new, 
> simplified version of the mock data generator. This approach is very 
> convenient, but is inherently limited. For example, the limited syntax 
> available in SQL does not encoding much information about columns such as 
> repeat count, data generator or so on. The simple SQL approach does not allow 
> generating multiple groups of data.
> However, all these features are present in the original mock data source via 
> a special JSON configuration file. Previously, only physical plans could 
> access that extended syntax.
> This ticket requests a SQL interface to the extended mock data source:
> {code}
> SELECT * FROM `mock`.`example/mock-options.json`
> {code}
> Mock data source options are always stored as a JSON file. Since the existing 
> mock data generator for SQL never uses JSON files, a simple rule is that if 
> the table name ends in ".json" then it is a specification, else the 
> information is encoded in table and column names.
> The format of the data generation syntax is documented in the mock data 
> source classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5260) Refinements to new "Cluster Fixture" test framework

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5260.
---

> Refinements to new "Cluster Fixture" test framework
> ---
>
> Key: DRILL-5260
> URL: https://issues.apache.org/jira/browse/DRILL-5260
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Roll-up of a number of enhancements to the cluster fixture framework.
> * Config option to suppress printing of CSV and other output. (Allows 
> printing for single tests, not printing when running from Maven.)
> * Parsing of query profiles to extract plan and run time information.
> * Fix bug in log fixture when enabling logging for a package.
> * Improved ZK support.
> * Set up the new CTTAS default temporary workspace for tests.
> * Revise TestDrillbitResiliance to use the new framework.
> * Revise TestWindowFrame to to use the new framework.
> * Revise TestMergeJoinWithSchemaChanges to use the new framework.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946101#comment-15946101
 ] 

Kunal Khatua commented on DRILL-5052:
-

Closing as no QA verification is required

> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5052) Option to debug generated Java code using an IDE

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5052.
---

> Option to debug generated Java code using an IDE
> 
>
> Key: DRILL-5052
> URL: https://issues.apache.org/jira/browse/DRILL-5052
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill makes extensive use of Java code generation to implement its operators. 
> Drill uses sophisticated techniques to blend generated code with pre-compiled 
> template code. An unfortunate side-effect of this behavior is that it is very 
> difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself 
> version of subclassing. The Drill "template" is the parent class, the 
> generated code is the subclass. But, rather than using plain-old subclassing, 
> Drill combines the code from the two classes into a single "artificial" 
> packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" 
> compilation: the generated code is a subclass of the template. Compile the 
> generated code as a plain-old Java class with no byte-code fix-up. Write the 
> code to a known location that the IDE can search when looking for source 
> files.
> With this change, developers can turn on the above feature, set a breakpoint 
> in a template, then step directly into the generated Java code called from 
> the template.
> This feature should be an option, enabled by developers when needed. The 
> existing byte-code technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5157) Multiple Snappy versions on class path; causes unit test failures

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946099#comment-15946099
 ] 

Kunal Khatua commented on DRILL-5157:
-

No QA Verification required

> Multiple Snappy versions on class path; causes unit test failures
> -
>
> Key: DRILL-5157
> URL: https://issues.apache.org/jira/browse/DRILL-5157
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Run the {{TestDrillbitResilience.doMemoryLeaksWhenCancelled}} unit test. It 
> fails with the following stack trace and the memory leak trace shown second.
> Strangely, this error appears only if the test is run as part of the overall 
> suite. The error does not appear if the test is run individually in the 
> debugger. This suggests that the problem described here is a side-effect of a 
> problem created by an earlier test.
> Investigation (see notes below) reveal that this is a Maven dependency 
> management problem.
> Stack trace that seems to show that the code was trying to find a Snappy 
> native library:
> {code}
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
>   at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
>   at org.xerial.snappy.Snappy.(Snappy.java:44)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$DecompressionHelper.decompress(AsyncPageReader.java:402)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.decompress(AsyncPageReader.java:169)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.getDecompressedPageData(AsyncPageReader.java:96)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.nextInternal(AsyncPageReader.java:219)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.PageReader.next(PageReader.java:280)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:250)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:178)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:130)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:485)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:479)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:562)
>   at 
> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:178)
>   at 
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> …
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-5157) Multiple Snappy versions on class path; causes unit test failures

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua closed DRILL-5157.
---

> Multiple Snappy versions on class path; causes unit test failures
> -
>
> Key: DRILL-5157
> URL: https://issues.apache.org/jira/browse/DRILL-5157
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Run the {{TestDrillbitResilience.doMemoryLeaksWhenCancelled}} unit test. It 
> fails with the following stack trace and the memory leak trace shown second.
> Strangely, this error appears only if the test is run as part of the overall 
> suite. The error does not appear if the test is run individually in the 
> debugger. This suggests that the problem described here is a side-effect of a 
> problem created by an earlier test.
> Investigation (see notes below) reveal that this is a Maven dependency 
> management problem.
> Stack trace that seems to show that the code was trying to find a Snappy 
> native library:
> {code}
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
>   at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
>   at org.xerial.snappy.Snappy.(Snappy.java:44)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$DecompressionHelper.decompress(AsyncPageReader.java:402)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.decompress(AsyncPageReader.java:169)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.getDecompressedPageData(AsyncPageReader.java:96)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.nextInternal(AsyncPageReader.java:219)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.PageReader.next(PageReader.java:280)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:250)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:178)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:130)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:485)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:479)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:562)
>   at 
> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:178)
>   at 
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> …
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5091) JDBC unit test fail on Java 8

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-5091:

Reviewer: Krystal  (was: Padma Penumarthy)

> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946092#comment-15946092
 ] 

Kunal Khatua commented on DRILL-5091:
-

Krystal, Please verify and close this bug

> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5080) Create a memory-managed version of the External Sort operator

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946084#comment-15946084
 ] 

Rahul Challapalli commented on DRILL-5080:
--

[~Paul.Rogers] Since the sub-tasks are marked as resolved for 1.11.0, would it 
make sense to also change the fix version for this umbrella jira? On the other 
hand we did create a memory managed external sort. Its just not enabled by 
default in 1.10.0

> Create a memory-managed version of the External Sort operator
> -
>
> Key: DRILL-5080
> URL: https://issues.apache.org/jira/browse/DRILL-5080
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
> Attachments: ManagedExternalSortDesign.pdf
>
>
> We propose to create a "managed" version of the external sort operator that 
> works to a clearly-defined memory limit. Attached is a design specification 
> for the work.
> The project will include fixing a number of bugs related to the external 
> sort, include as sub-tasks of this umbrella task.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5252) A condition returns always true

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946077#comment-15946077
 ] 

Rahul Challapalli commented on DRILL-5252:
--

I am wondering about a possible test case which could expose this issue. Also 
if a filter has an equality condition ("=") and the right side is a null 
constant, shouldn't drill always return false?

> A condition returns always true
> ---
>
> Key: DRILL-5252
> URL: https://issues.apache.org/jira/browse/DRILL-5252
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: JC
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> I've found the following code smell in recent github snapshot.
> Path: 
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/EqualityVisitor.java
> {code:java}
> 287 
> 288   @Override
> 289   public Boolean visitNullConstant(TypedNullConstant e, LogicalExpression 
> value) throws RuntimeException {
> 290 if (!(value instanceof TypedNullConstant)) {
> 291   return false;
> 292 }
> 293 return e.getMajorType().equals(e.getMajorType());
> 294   }
> 295
> {code}
> Should it be like this?
> {code:java}
> 292 }
> 293 return value.getMajorType().equals(e.getMajorType());
> 294   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5324) Provide simplified column reader/writer for use in tests

[Zelaine Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong updated DRILL-5324:

Reviewer: Karthikeyan Manivannan  (was: Gautam Kumar Parai)

Reassigned Reviewer to [~karthikm]

> Provide simplified column reader/writer for use in tests
> 
>
> Key: DRILL-5324
> URL: https://issues.apache.org/jira/browse/DRILL-5324
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> In support of DRILL-5323, we wish to provide a very easy way to work with row 
> sets. See the comment section for examples of the target API.
> Drill provides over 100 different value vectors, any of which may be required 
> to perform a specific unit test. Creating these vectors, populating them, and 
> retrieving values, is very tedious. The work is so complex that it acts to 
> discourage developers from writing such tests.
> To simplify the task, we wish to provide a simplified row set reader and 
> writer. To do that, we need to generate the corresponding column reader and 
> writer for each value vector. This ticket focuses on the column-level readers 
> and writers, and the required code generation.
> Drill already provides vector readers and writers derived from 
> {{FieldReader}}. However, these readers do not provide a uniform get/set 
> interface that is type independent on the application side. Instead, 
> application code must be aware of the type of the vector, something we seek 
> to avoid for test code.
> The reader and writer classes are designed to be used in many contexts, not 
> just for testing. As a result, their implementation makes no assumptions 
> about the broader row reader and writer, other than that a row index and the 
> required value vector are both available. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4847) Window function query results in OOM Exception.

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945938#comment-15945938
 ] 

Paul Rogers commented on DRILL-4847:


[~khfaraaz], turns out the above still uses the "old" "unmanaged" sort, not the 
"new" "managed" sort introduced in Drill 1.10. The managed sort is disabled by 
default. To enable the managed sort, you can either change the boot 
configuration:

{code}
drill.exec.sort.external.disable_managed = false
{code}

Or, you can make the setting at run time:

{code}
ALTER SESSION SET `exec.sort.disable_managed` = false
{code}

You'll know you've got it turned on if you look in the log file and see log 
entries for 
{{org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch}}.

> Window function query results in OOM Exception.
> ---
>
> Key: DRILL-4847
> URL: https://issues.apache.org/jira/browse/DRILL-4847
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.8.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Paul Rogers
>Priority: Critical
>  Labels: window_function
> Attachments: drillbit.log
>
>
> Window function query results in OOM Exception.
> Drill version 1.8.0-SNAPSHOT git commit ID: 38ce31ca
> MapRBuildVersion 5.1.0.37549.GA
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT clientname, audiencekey, spendprofileid, 
> postalcd, provincecd, provincename, postalcode_json, country_json, 
> province_json, town_json, dma_json, msa_json, ROW_NUMBER() OVER (PARTITION BY 
> spendprofileid  ORDER BY (CASE WHEN postalcd IS NULL THEN 9 ELSE 0 END) ASC, 
> provincecd ASC) as rn FROM `MD593.parquet` limit 3;
> Error: RESOURCE ERROR: One or more nodes ran out of memory while executing 
> the query.
> Failure while allocating buffer.
> Fragment 0:0
> [Error Id: 2287fe71-f0cb-469a-a563-11580fceb1c5 on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2016-08-16 07:25:44,590 [284d4006-9f9d-b893-9352-4f54f9b1d52a:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 284d4006-9f9d-b893-9352-4f54f9b1d52a: SELECT clientname, audiencekey, 
> spendprofileid, postalcd, provincecd, provincename, postalcode_json, 
> country_json, province_json, town_json, dma_json, msa_json, ROW_NUMBER() OVER 
> (PARTITION BY spendprofileid  ORDER BY (CASE WHEN postalcd IS NULL THEN 9 
> ELSE 0 END) ASC, provincecd ASC) as rn FROM `MD593.parquet` limit 3
> ...
> 2016-08-16 07:25:46,273 [284d4006-9f9d-b893-9352-4f54f9b1d52a:frag:0:0] INFO  
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to 
> /tmp/drill/spill/284d4006-9f9d-b893-9352-4f54f9b1d52a_majorfragment0_minorfragment0_operator8/2
> 2016-08-16 07:25:46,283 [284d4006-9f9d-b893-9352-4f54f9b1d52a:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
> nodes ran out of memory while executing the query.
> Failure while allocating buffer.
> [Error Id: 2287fe71-f0cb-469a-a563-11580fceb1c5 ]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:242)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
> Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Failure 
> while allocating buffer.
> at 
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:187)
>  ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapTransferPair.(RepeatedMapVector.java:331)
>  ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapTransferPair.(RepeatedMapVector.java:307)
>  ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.vector.complex.RepeatedMapVector.getTransferPair(RepeatedMapVector.java:161)
>  ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.SimpleVectorWrapper.cloneAndTransfer(SimpleVectorWrapper.java:66)
>  

[jira] [Closed] (DRILL-4704) select statement behavior is inconsistent for decimal values in parquet

[Rahul Challapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Challapalli closed DRILL-4704.


Verified and added tests

> select statement behavior is inconsistent for decimal values in parquet
> ---
>
> Key: DRILL-4704
> URL: https://issues.apache.org/jira/browse/DRILL-4704
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.6.0
> Environment: Windows 7 Pro, Java 1.8.0_91
>Reporter: Dave Oshinsky
> Fix For: 1.8.0
>
>
> A select statement that searches a parquet file for a decimal value matching 
> a specific value behaves inconsistently.  The query expressed most simply 
> finds nothing:
> 0: jdbc:drill:zk=local> select * from dfs.`c:/archiveHR/HR.EMPLOYEES` where 
> employee_id = 100;
> +--+-+++---+---+
> | EMPLOYEE_ID  | FIRST_NAME  | LAST_NAME  | EMAIL  | PHONE_NUMBER  | 
> HIRE_DATE |
> +--+-+++---+---+
> +--+-+++---+---+
> No rows selected (0.348 seconds)
> The query can be modified to find the matching row in a few ways, such as the 
> following (using between instead of '=', changing 100 to 100.0, or casting as 
> decimal:
> 0: jdbc:drill:zk=local> select * from dfs.`c:/archiveHR/HR.EMPLOYEES` where 
> employee_id between 100 and 100;
> +--+-+++---+---+
> | EMPLOYEE_ID  | FIRST_NAME  | LAST_NAME  | EMAIL  | PHONE_NUMBER  |   
> HIR |
> +--+-+++---+---+
> | 100  | Steven  | King   | SKING  | 515.123.4567  | 
> 2003-06-1 |
> +--+-+++---+---+
> 1 row selected (0.226 seconds)
> 0: jdbc:drill:zk=local> select * from dfs.`c:/archiveHR/HR.EMPLOYEES` where 
> employee_id = 100.0;
> +--+-+++---+---+
> | EMPLOYEE_ID  | FIRST_NAME  | LAST_NAME  | EMAIL  | PHONE_NUMBER  |   
> HIR |
> +--+-+++---+---+
> | 100  | Steven  | King   | SKING  | 515.123.4567  | 
> 2003-06-1 |
> +--+-+++---+---+
> 1 row selected (0.259 seconds)
> 0: jdbc:drill:zk=local> select * from dfs.`c:/archiveHR/HR.EMPLOYEES` where 
> cast(employee_id AS DECIMAL) = 100;
> +--+-+++---+---+
> | EMPLOYEE_ID  | FIRST_NAME  | LAST_NAME  | EMAIL  | PHONE_NUMBER  |   
> HIR |
> +--+-+++---+---+
> | 100  | Steven  | King   | SKING  | 515.123.4567  | 
> 2003-06-1 |
> +--+-+++---+---+
> 1 row selected (0.232 seconds)
> 0: jdbc:drill:zk=local>
> The schema of the parquet data that is being searched is as follows:
> $ java -jar parquet-tools*1.jar meta c:/archiveHR/HR.EMPLOYEES/1.parquet
> file:   file:/c:/archiveHR/HR.EMPLOYEES/1.parquet
> creator:parquet-mr version 1.8.1 (build 
> 4aba4dae7bb0d4edbcf7923ae1339f28fd3f7fcf)
> .
> file schema:HR.EMPLOYEES
> 
> EMPLOYEE_ID:REQUIRED FIXED_LEN_BYTE_ARRAY O:DECIMAL R:0 D:0
> FIRST_NAME: OPTIONAL BINARY O:UTF8 R:0 D:1
> LAST_NAME:  REQUIRED BINARY O:UTF8 R:0 D:0
> EMAIL:  REQUIRED BINARY O:UTF8 R:0 D:0
> PHONE_NUMBER:   OPTIONAL BINARY O:UTF8 R:0 D:1
> HIRE_DATE:  REQUIRED BINARY O:UTF8 R:0 D:0
> JOB_ID: REQUIRED BINARY O:UTF8 R:0 D:0
> SALARY: OPTIONAL FIXED_LEN_BYTE_ARRAY O:DECIMAL R:0 D:1
> COMMISSION_PCT: OPTIONAL FIXED_LEN_BYTE_ARRAY O:DECIMAL R:0 D:1
> MANAGER_ID: OPTIONAL FIXED_LEN_BYTE_ARRAY O:DECIMAL R:0 D:1
> DEPARTMENT_ID:  OPTIONAL FIXED_LEN_BYTE_ARRAY O:DECIMAL R:0 D:1
> row group 1:RC:107 TS:9943 OFFSET:4
> 
> EMPLOYEE_ID: FIXED_LEN_BYTE_ARRAY SNAPPY DO:0 FPO:4 SZ:360/355/0.99 
> VC:107 ENC:PLAIN,BIT_PACKED
> FIRST_NAME:  BINARY SNAPPY DO:0 FPO:364 SZ:902/1058/1.17 VC:107 
> ENC:PLAIN_DICTIONARY,RLE,BIT_PACKED
> LAST_NAME:   BINARY SNAPPY DO:0 FPO:1266 SZ:913//1.22 VC:107 
> ENC:PLAIN,BIT_PACKED
> EMAIL:   BINARY SNAPPY DO:0 FPO:2179 SZ:977/1184/1.21 VC:107 
> ENC:PLAIN,BIT_PACKED
> PHONE_NUMBER:BINARY SNAPPY DO:0 FPO:3156 SZ:750/1987/2.65 VC:107 
> ENC:PLAIN,RLE,BIT_PACKED
> HIRE_DATE:   BINARY SNAPPY DO:0 

[jira] [Updated] (DRILL-5005) Potential issues with external sort info in query profile

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5005:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> Potential issues with external sort info in query profile
> -
>
> Key: DRILL-5005
> URL: https://issues.apache.org/jira/browse/DRILL-5005
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> Run a query that will include an external sort.
> Look at the visualized plan. The external sort is shown in the tree as just 
> "Sort". Suggestion: say "External Sort".
> In the operator profiles section, memory use for EXTERNAL_SORT is listed as 
> 26MB. Yet the file being sorted is 388 MB. Even allowing for projects of a 
> subset of column, 26 MB seems awfully small to hold the data set. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5284) Roll-up of final fixes for managed sort

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5284:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

Responses to specific QA findings. Rerun those tests to verify fixes. This is a 
roll-up to allow multiple fixes to appear in a single PR; see the sub-tasks for 
details.

> Roll-up of final fixes for managed sort
> ---
>
> Key: DRILL-5284
> URL: https://issues.apache.org/jira/browse/DRILL-5284
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.11.0
>
>
> The managed external sort was introduced in DRILL-5080. Since that time, 
> extensive testing has identified a number of minor fixes and improvements. 
> Given the long PR cycles, it is not practical to spend a week or two to do a 
> PR for each fix individually. This ticket represents a roll-up of a 
> combination of a number of fixes. Small fixes are listed here, larger items 
> appear as sub-tasks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5275) Sort spill serialization is slow due to repeated buffer allocations

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945873#comment-15945873
 ] 

Paul Rogers commented on DRILL-5275:


Primarily a development issue; hard to test at the QA level.

> Sort spill serialization is slow due to repeated buffer allocations
> ---
>
> Key: DRILL-5275
> URL: https://issues.apache.org/jira/browse/DRILL-5275
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Drill provides a sort operator that spills to disk. The spill and read 
> operations use the serialization code in the 
> {{VectorAccessibleSerializable}}. This code, in turn, uses the 
> {{DrillBuf.getBytes()}} method to write to an output stream. (Yes, the "get" 
> method writes, and the "write" method reads...)
> The DrillBuf method turns around and calls the UDLE method that does:
> {code}
> byte[] tmp = new byte[length];
> PlatformDependent.copyMemory(addr(index), tmp, 0, length);
> out.write(tmp);
> {code}
> That is, for each write the code allocates a heap buffer. Since Drill buffers 
> can be quite large (4, 8, 16 MB or larger), the above rapidly fills the heap 
> and causes GC.
> The result is slow performance. On a Mac, with an SSD that can do 700 MB/s of 
> I/O, we get only about 40 MB/s. Very likely because of excessive CPU cost and 
> GC.
> The solution is to allocate a single read or write buffer, then use that same 
> buffer over and over when reading or writing. This must be done in 
> {{VectorAccessibleSerializable}} as it is a per-thread class that has 
> visibility to all the buffers to be written.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5157) Multiple Snappy versions on class path; causes unit test failures

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945871#comment-15945871
 ] 

Paul Rogers commented on DRILL-5157:


Development-only issue, no QA verification needed.


> Multiple Snappy versions on class path; causes unit test failures
> -
>
> Key: DRILL-5157
> URL: https://issues.apache.org/jira/browse/DRILL-5157
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Run the {{TestDrillbitResilience.doMemoryLeaksWhenCancelled}} unit test. It 
> fails with the following stack trace and the memory leak trace shown second.
> Strangely, this error appears only if the test is run as part of the overall 
> suite. The error does not appear if the test is run individually in the 
> debugger. This suggests that the problem described here is a side-effect of a 
> problem created by an earlier test.
> Investigation (see notes below) reveal that this is a Maven dependency 
> management problem.
> Stack trace that seems to show that the code was trying to find a Snappy 
> native library:
> {code}
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
>   at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
>   at org.xerial.snappy.Snappy.(Snappy.java:44)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$DecompressionHelper.decompress(AsyncPageReader.java:402)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.decompress(AsyncPageReader.java:169)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.getDecompressedPageData(AsyncPageReader.java:96)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.nextInternal(AsyncPageReader.java:219)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.PageReader.next(PageReader.java:280)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:250)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:178)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:130)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:485)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:479)
>   at 
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:562)
>   at 
> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:178)
>   at 
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> …
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5066) External sort attempts to retry sv2 memory alloc, even if can never succeed

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5066:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

Primarily a development issue; hard to test at the QA level.

> External sort attempts to retry sv2 memory alloc, even if can never succeed
> ---
>
> Key: DRILL-5066
> URL: https://issues.apache.org/jira/browse/DRILL-5066
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The external sort contains rather complex code to allocate an sv2 in the 
> method {{newSV2()}}. The code tries to allocate an sv2. If the allocation 
> fails, the code attempts to spill (which is fine) and try again. If things 
> still fail, the code waits 1 sec. and tries again. It will continue to wait 
> up to a minute, doubling the wait time each cycle.
> Presumably, this is so that some other part of Drill will release memory. 
> But, because of the way the allocator currently works, the allocation is 
> limited by the limit set on the external sort's own allocator. This limit 
> won't change by waiting.
> The loop only makes sense if the memory allocation failed because the 
> external sort's allocator is not above its limit, but the parent can't 
> provide memory.
> In practice, this scenario should not occur once the external sort is 
> resource managed, so the retry code can simply be removed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5020) ExternalSortBatch has inconsistent notions of the memory limit

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945867#comment-15945867
 ] 

Paul Rogers commented on DRILL-5020:


Primarily a development issue; hard to test at the QA level.

> ExternalSortBatch has inconsistent notions of the memory limit
> --
>
> Key: DRILL-5020
> URL: https://issues.apache.org/jira/browse/DRILL-5020
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The memory limit cases for the spill-needed test seem inconsistent:
> For the test for in-memory sort:
> {code}
> long currentlyAvailable =  popConfig.getMaxAllocation() - 
> oAllocator.getAllocatedMemory();
> {code}
> For reaching the memory limit:
> {code}
> oAllocator.getAllocatedMemory() > .95 * oAllocator.getLimit()
> {code}
> That is, one uses {{oAllocator.getLimit}} ("the current maximum limit this 
> allocator imposes"), the other uses {{popConfig.getMaxAllocation}} ("The 
> maximum memory this operator can allocate".)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5019) ExternalSortBatch spills all batches to disk even if even one spills

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945868#comment-15945868
 ] 

Paul Rogers commented on DRILL-5019:


Primarily a development issue; hard to test at the QA level.

> ExternalSortBatch spills all batches to disk even if even one spills
> 
>
> Key: DRILL-5019
> URL: https://issues.apache.org/jira/browse/DRILL-5019
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The ExternalSortBatch (ESB) operator sorts batches while spilling to disk to 
> stay within a defined memory budget.
> Assume the memory budget is 10 GB. Assume that the actual volume of data to 
> be sorted is 10.1 GB. The ESB spills the extra 0.1 GB to disk. (Actually 
> spills more than that, say 5 GB.)
> At the completion of the run, ESB has read all incoming batches. It must now 
> merge those batches. It does so by spilling **all** batches to disk, then 
> doing a disk-based merge.
> This means that exceeding the memory limit by even a small amount is the same 
> as having a very low memory limit: all batches must spill.
> This solution is simple, it works, and has some amount of logic.
> But, it would be better to have a slightly more advanced solution that spills 
> only the smallest possible set of batches to disk, then does a hybrid 
> in-memory, on-disk merge, saving the unnecessary write/read cycle.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5014) ExternalSortBatch cache size, spill count differs from config setting

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945869#comment-15945869
 ] 

Paul Rogers commented on DRILL-5014:


Primarily a development issue; hard to test at the QA level.

> ExternalSortBatch cache size, spill count differs from config setting
> -
>
> Key: DRILL-5014
> URL: https://issues.apache.org/jira/browse/DRILL-5014
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The ExternalSortBatch (ESB) operator sorts data while spilling to disk to 
> remain within a defined memory footprint. Spilling occurs based on a number 
> of factors. Among those are two config parameters:
> * {{drill.exec.sort.external.spill.group.size}}: The number of batches to 
> spill per spill event.
> * {{drill.exec.sort.external.spill.threshold}}: The number of batches to 
> accumulate in memory before starting a spill event.
> The expected behavior would be:
> * After the accumulated batches exceeds the threshold, and
> * More than "batch size" new batches have arrived since the last spill,
> * Spill half the accumulated records.
> That is if the threshold is 200, and the size is 150, we should accumulate 
> 200 batches, then spill 150 of them (leaving 50) and repeat.
> The actual behavior is:
> * When the accumulated records exceeds the threshold and,
> * More than "batch size" new batches have arrived since the last spill,
> * Spill half the accumulated records.
> The above can leave more batches in memory than desired, and spill more than 
> desired.
> The actual behavior for the (threshold=200, size=150) case is:
> {code}
> Before spilling, buffered batch count: 201
> After spilling, buffered batch count: 101
> Before spilling, buffered batch count: 251
> After spilling, buffered batch count: 126
> Before spilling, buffered batch count: 276
> After spilling, buffered batch count: 138
> Before spilling, buffered batch count: 288
> After spilling, buffered batch count: 144
> Before spilling, buffered batch count: 294
> After spilling, buffered batch count: 147
> Before spilling, buffered batch count: 297
> After spilling, buffered batch count: 149
> Before spilling, buffered batch count: 299
> After spilling, buffered batch count: 150
> Before spilling, buffered batch count: 300
> After spilling, buffered batch count: 150
> Before spilling, buffered batch count: 300
> {code}
> In short, the actual number of batches retained in memory is twice the spill 
> size, **NOT** the number set in the threshold. As a result, the ESB operator 
> will use more memory than expected.
> The work-around is to set a batch size that is half the threshold so that the 
> batch size (used in spill decisions) matches the actual spill count (as 
> implemented by the code.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5023) ExternalSortBatch does not spill fully, throws off spill calculations

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945864#comment-15945864
 ] 

Paul Rogers commented on DRILL-5023:


Primarily a development issue; hard to test at the QA level.

> ExternalSortBatch does not spill fully, throws off spill calculations
> -
>
> Key: DRILL-5023
> URL: https://issues.apache.org/jira/browse/DRILL-5023
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The {{ExternalSortBatch}} (ESB) operator sorts records, spilling to disk as 
> needed to operate within a defined memory budget.
> When needed, ESB spills accumulated record batches to disk. However, when 
> doing so, the ESB carves off the first spillable batch and holds it in memory:
> {code}
> // 1 output container is kept in memory, so we want to hold on to it and 
> transferClone
> // allows keeping ownership
> VectorContainer c1 = VectorContainer.getTransferClone(outputContainer, 
> oContext);
> c1.buildSchema(BatchSchema.SelectionVectorMode.NONE);
> c1.setRecordCount(count);
> ...
> BatchGroup newGroup = new BatchGroup(c1, fs, outputFile, oContext);
> {code}
> When the spill batch size gets larger (to fix DRILL-5022), the result is that 
> nothing is spilled as the first spillable batch is simply stored back into 
> memory on the (supposedly) spilled batches list.
> The desired behavior is for all spillable batches to be written to disk. If 
> the first batch is held back to work around some issue (to keep a schema, 
> say?), then fine a different solution that allows the actual data to spill.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5022) ExternalSortBatch sets two different limits for "copier" memory

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945865#comment-15945865
 ] 

Paul Rogers commented on DRILL-5022:


Primarily a development issue; hard to test at the QA level.

> ExternalSortBatch sets two different limits for "copier" memory
> ---
>
> Key: DRILL-5022
> URL: https://issues.apache.org/jira/browse/DRILL-5022
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The {{ExternalSortBatch}} (ESB) operator sorts rows and supports spilling to 
> disk to operate within a set memory budget.
> A key step in disk-based sorting is to merge "runs" of previously-sorted 
> records. ESB does this with a class created from the 
> {{PriorityQueueCopierTemplate}}, called the "copier" in the code.
> The sort runs are represented by record batches, each with an indirection 
> vector (AKA {{SelectionVector}}) that point to the records in sort order.
> The copier restructures the incoming runs: copying from the original batches 
> (from positions given by the indirection vector) into new output vectors in 
> sorted order. To do this work, the copier must allocate new vectors to hold 
> the merged data. These vectors consume memory, and must fit into the overall 
> memory budget assigned to the ESB.
> As it turns out, the ESB code has two conflicting ways of setting the limit. 
> One is hard-coded:
> {code}
>   private static final int COPIER_BATCH_MEM_LIMIT = 256 * 1024;
> {code}
> The other comes from config parameters:
> {code}
>   public static final long INITIAL_ALLOCATION = 10_000_000;
>   public static final long MAX_ALLOCATION = 20_000_000;
> copierAllocator = oAllocator.newChildAllocator(oAllocator.getName() + 
> ":copier",
> PriorityQueueCopier.INITIAL_ALLOCATION, 
> PriorityQueueCopier.MAX_ALLOCATION);
> {code}
> Strangely, the config parameters are used to set aside memory for the copier 
> to use. But, the {{COPIER_BATCH_MEM_LIMIT}} is used to determine how large of 
> a merged batch to actually create.
> The result is that we set aside 10 MB of memory, but use only 256K of it, 
> wasting 9 MB.
> This ticket asks to:
> * Determine the proper merged batch size.
> * Use that limit to set the memory allocation for the copier.
> Elsewhere in Drill batch sizes tend to be on the order of 32K records. In the 
> ESB, the low {{COPIER_BATCH_MEM_LIMIT}} tends to favor smaller batches: A 
> test case has a row width of 114 bytes, and produces batches of just 2299 
> records. So, likely the proper choice is the larger 10 MB memory allocator 
> limit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5026) ExternalSortBatch uses two memory allocators; one will do

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945860#comment-15945860
 ] 

Paul Rogers commented on DRILL-5026:


Primarily a development issue; hard to test at the QA level.

> ExternalSortBatch uses two memory allocators; one will do
> -
>
> Key: DRILL-5026
> URL: https://issues.apache.org/jira/browse/DRILL-5026
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The {{ExternalSortBatch}} (ESB) operator performs a sort while spilling to 
> disk to stay within a defined memory budget.
> ESB uses two memory allocators:
> * The operator allocator for most operations,
> * The copier allocator (child of the operator allocator) for merge operations.
> The copier allocator is used only when merging prior to spill.
> However, since the ESB must manage within a single memory budget, the code is 
> made more complex by using two allocators, and keeping track of the memory 
> used in both.
> Better to use a single allocator. When checking for the need to spill, the 
> code already checks if enough memory is available for an in-memory sort, if 
> needed. Similarly, the code should also check if sufficient memory is 
> available for the copy step when needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5025) ExternalSortBatch provides weak control over spill file size

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945862#comment-15945862
 ] 

Paul Rogers commented on DRILL-5025:


Primarily a development issue; hard to test at the QA level.

> ExternalSortBatch provides weak control over spill file size
> 
>
> Key: DRILL-5025
> URL: https://issues.apache.org/jira/browse/DRILL-5025
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The ExternalSortBatch (ESB) operator sorts records while spilling to disk to 
> control memory use. The size of the spill file is not easy to control. It is 
> a function of the accumulated batches size (half of the accumulated total), 
> which is determined by either the memory budget or the 
> {{drill.exec.sort.external.group.size}} parameter. (But, even with the 
> parameter, the actual file size is still half the accumulated batches.)
> The proposed solution is to provide an explicit parameter that sets the 
> maximum spill file size: {{drill.exec.sort.external.spill.size}}. If the ESB 
> needs to spill more than this amount of data, ESB should split the spill into 
> multiple files.
> The spill.size should be in bytes (or MB). (A size in records makes the file 
> size data-dependent, which would not be helpful.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5027) ExternalSortBatch is inefficient: rewrites data unnecessarily

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945859#comment-15945859
 ] 

Paul Rogers commented on DRILL-5027:


Primarily a development issue; hard to test at the QA level.

> ExternalSortBatch is inefficient: rewrites data unnecessarily
> -
>
> Key: DRILL-5027
> URL: https://issues.apache.org/jira/browse/DRILL-5027
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
> needed to operate within a memory budget.
> The sort happens in two phases:
> 1. Gather the incoming batches from the upstream operator, sort them, and 
> spill to disk as needed.
> 2. Merge the "runs" spilled in step 1.
> In most cases, the second step should run within the memory available for the 
> first step (which is why severity is only Minor). Large queries need multiple 
> sort "phases" in which previously spilled runs are read back into memory, 
> merged, and again spilled. It is here that ESB has an issue. This process 
> correctly limit the amount of memory used, but at the cost or rewriting the 
> same data over and over.
> Consider current Drill behavior:
> {code}
> a b c d (re-spill)
> abcd e f g h (re-spill)
> abcefgh i j k
> {code}
> That is batches, a, b, c and d are re-spilled to create the combined abcd, 
> and so on. The same data is rewritten over and over.
> Note that spilled batches take no (direct) memory in Drill, and require only 
> a small on-heap memento. So, maintaining data on disk s "free". So, better 
> would be to re-spill only newer data:
> {code}
> a b c d (re-spill)
> abcd | e f g h (re-spill)
> abcd efgh | i j k
> {code}
> Where the bar indicates a moving point at which we've already merged and do 
> not need to do so again. If each letter is one unit of disk I/O, the original 
> method uses 35 units while the revised method uses 27 units.
> At some point the process may have to repeat by merging the second-generation 
> spill files and so on.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5055) External Sort does not delete spill file if error occurs during close

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945858#comment-15945858
 ] 

Paul Rogers commented on DRILL-5055:


Hard to verify for QA. Verified with a unit test.

> External Sort does not delete spill file if error occurs during close
> -
>
> Key: DRILL-5055
> URL: https://issues.apache.org/jira/browse/DRILL-5055
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> DRILL-3898 recently fixed a case in which disk space was exhausted during a 
> spill event for the external sort. In this case, the call to close failed 
> because close attempted to flush remaining buffered data, but that also 
> failed due to out of space.
> While the fix works, the fix causes the partially-completed spill file to be 
> left on disk. Consider this code in {{BatchGroup.close( )}}
> {code}
> if (outputStream != null) {
>   outputStream.close();
> }
> ...
> if (fs != null && fs.exists(path)) {
>   fs.delete(path, false);
> }
> {code}
> Notice that, if the output stream close fails, the spill file is not deleted. 
> The fix is to put the delete in a finally block so that it is always deleted.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5055) External Sort does not delete spill file if error occurs during close

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5055:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> External Sort does not delete spill file if error occurs during close
> -
>
> Key: DRILL-5055
> URL: https://issues.apache.org/jira/browse/DRILL-5055
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> DRILL-3898 recently fixed a case in which disk space was exhausted during a 
> spill event for the external sort. In this case, the call to close failed 
> because close attempted to flush remaining buffered data, but that also 
> failed due to out of space.
> While the fix works, the fix causes the partially-completed spill file to be 
> left on disk. Consider this code in {{BatchGroup.close( )}}
> {code}
> if (outputStream != null) {
>   outputStream.close();
> }
> ...
> if (fs != null && fs.exists(path)) {
>   fs.delete(path, false);
> }
> {code}
> Notice that, if the output stream close fails, the spill file is not deleted. 
> The fix is to put the delete in a finally block so that it is always deleted.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5027) ExternalSortBatch is inefficient: rewrites data unnecessarily

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5027:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> ExternalSortBatch is inefficient: rewrites data unnecessarily
> -
>
> Key: DRILL-5027
> URL: https://issues.apache.org/jira/browse/DRILL-5027
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
> needed to operate within a memory budget.
> The sort happens in two phases:
> 1. Gather the incoming batches from the upstream operator, sort them, and 
> spill to disk as needed.
> 2. Merge the "runs" spilled in step 1.
> In most cases, the second step should run within the memory available for the 
> first step (which is why severity is only Minor). Large queries need multiple 
> sort "phases" in which previously spilled runs are read back into memory, 
> merged, and again spilled. It is here that ESB has an issue. This process 
> correctly limit the amount of memory used, but at the cost or rewriting the 
> same data over and over.
> Consider current Drill behavior:
> {code}
> a b c d (re-spill)
> abcd e f g h (re-spill)
> abcefgh i j k
> {code}
> That is batches, a, b, c and d are re-spilled to create the combined abcd, 
> and so on. The same data is rewritten over and over.
> Note that spilled batches take no (direct) memory in Drill, and require only 
> a small on-heap memento. So, maintaining data on disk s "free". So, better 
> would be to re-spill only newer data:
> {code}
> a b c d (re-spill)
> abcd | e f g h (re-spill)
> abcd efgh | i j k
> {code}
> Where the bar indicates a moving point at which we've already merged and do 
> not need to do so again. If each letter is one unit of disk I/O, the original 
> method uses 35 units while the revised method uses 27 units.
> At some point the process may have to repeat by merging the second-generation 
> spill files and so on.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5023) ExternalSortBatch does not spill fully, throws off spill calculations

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5023:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> ExternalSortBatch does not spill fully, throws off spill calculations
> -
>
> Key: DRILL-5023
> URL: https://issues.apache.org/jira/browse/DRILL-5023
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The {{ExternalSortBatch}} (ESB) operator sorts records, spilling to disk as 
> needed to operate within a defined memory budget.
> When needed, ESB spills accumulated record batches to disk. However, when 
> doing so, the ESB carves off the first spillable batch and holds it in memory:
> {code}
> // 1 output container is kept in memory, so we want to hold on to it and 
> transferClone
> // allows keeping ownership
> VectorContainer c1 = VectorContainer.getTransferClone(outputContainer, 
> oContext);
> c1.buildSchema(BatchSchema.SelectionVectorMode.NONE);
> c1.setRecordCount(count);
> ...
> BatchGroup newGroup = new BatchGroup(c1, fs, outputFile, oContext);
> {code}
> When the spill batch size gets larger (to fix DRILL-5022), the result is that 
> nothing is spilled as the first spillable batch is simply stored back into 
> memory on the (supposedly) spilled batches list.
> The desired behavior is for all spillable batches to be written to disk. If 
> the first batch is held back to work around some issue (to keep a schema, 
> say?), then fine a different solution that allows the actual data to spill.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5025) ExternalSortBatch provides weak control over spill file size

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5025:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> ExternalSortBatch provides weak control over spill file size
> 
>
> Key: DRILL-5025
> URL: https://issues.apache.org/jira/browse/DRILL-5025
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The ExternalSortBatch (ESB) operator sorts records while spilling to disk to 
> control memory use. The size of the spill file is not easy to control. It is 
> a function of the accumulated batches size (half of the accumulated total), 
> which is determined by either the memory budget or the 
> {{drill.exec.sort.external.group.size}} parameter. (But, even with the 
> parameter, the actual file size is still half the accumulated batches.)
> The proposed solution is to provide an explicit parameter that sets the 
> maximum spill file size: {{drill.exec.sort.external.spill.size}}. If the ESB 
> needs to spill more than this amount of data, ESB should split the spill into 
> multiple files.
> The spill.size should be in bytes (or MB). (A size in records makes the file 
> size data-dependent, which would not be helpful.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5026) ExternalSortBatch uses two memory allocators; one will do

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5026:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> ExternalSortBatch uses two memory allocators; one will do
> -
>
> Key: DRILL-5026
> URL: https://issues.apache.org/jira/browse/DRILL-5026
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The {{ExternalSortBatch}} (ESB) operator performs a sort while spilling to 
> disk to stay within a defined memory budget.
> ESB uses two memory allocators:
> * The operator allocator for most operations,
> * The copier allocator (child of the operator allocator) for merge operations.
> The copier allocator is used only when merging prior to spill.
> However, since the ESB must manage within a single memory budget, the code is 
> made more complex by using two allocators, and keeping track of the memory 
> used in both.
> Better to use a single allocator. When checking for the need to spill, the 
> code already checks if enough memory is available for an in-memory sort, if 
> needed. Similarly, the code should also check if sufficient memory is 
> available for the copy step when needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5022) ExternalSortBatch sets two different limits for "copier" memory

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5022:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> ExternalSortBatch sets two different limits for "copier" memory
> ---
>
> Key: DRILL-5022
> URL: https://issues.apache.org/jira/browse/DRILL-5022
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The {{ExternalSortBatch}} (ESB) operator sorts rows and supports spilling to 
> disk to operate within a set memory budget.
> A key step in disk-based sorting is to merge "runs" of previously-sorted 
> records. ESB does this with a class created from the 
> {{PriorityQueueCopierTemplate}}, called the "copier" in the code.
> The sort runs are represented by record batches, each with an indirection 
> vector (AKA {{SelectionVector}}) that point to the records in sort order.
> The copier restructures the incoming runs: copying from the original batches 
> (from positions given by the indirection vector) into new output vectors in 
> sorted order. To do this work, the copier must allocate new vectors to hold 
> the merged data. These vectors consume memory, and must fit into the overall 
> memory budget assigned to the ESB.
> As it turns out, the ESB code has two conflicting ways of setting the limit. 
> One is hard-coded:
> {code}
>   private static final int COPIER_BATCH_MEM_LIMIT = 256 * 1024;
> {code}
> The other comes from config parameters:
> {code}
>   public static final long INITIAL_ALLOCATION = 10_000_000;
>   public static final long MAX_ALLOCATION = 20_000_000;
> copierAllocator = oAllocator.newChildAllocator(oAllocator.getName() + 
> ":copier",
> PriorityQueueCopier.INITIAL_ALLOCATION, 
> PriorityQueueCopier.MAX_ALLOCATION);
> {code}
> Strangely, the config parameters are used to set aside memory for the copier 
> to use. But, the {{COPIER_BATCH_MEM_LIMIT}} is used to determine how large of 
> a merged batch to actually create.
> The result is that we set aside 10 MB of memory, but use only 256K of it, 
> wasting 9 MB.
> This ticket asks to:
> * Determine the proper merged batch size.
> * Use that limit to set the memory allocation for the copier.
> Elsewhere in Drill batch sizes tend to be on the order of 32K records. In the 
> ESB, the low {{COPIER_BATCH_MEM_LIMIT}} tends to favor smaller batches: A 
> test case has a row width of 114 bytes, and produces batches of just 2299 
> records. So, likely the proper choice is the larger 10 MB memory allocator 
> limit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5019) ExternalSortBatch spills all batches to disk even if even one spills

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5019:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> ExternalSortBatch spills all batches to disk even if even one spills
> 
>
> Key: DRILL-5019
> URL: https://issues.apache.org/jira/browse/DRILL-5019
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The ExternalSortBatch (ESB) operator sorts batches while spilling to disk to 
> stay within a defined memory budget.
> Assume the memory budget is 10 GB. Assume that the actual volume of data to 
> be sorted is 10.1 GB. The ESB spills the extra 0.1 GB to disk. (Actually 
> spills more than that, say 5 GB.)
> At the completion of the run, ESB has read all incoming batches. It must now 
> merge those batches. It does so by spilling **all** batches to disk, then 
> doing a disk-based merge.
> This means that exceeding the memory limit by even a small amount is the same 
> as having a very low memory limit: all batches must spill.
> This solution is simple, it works, and has some amount of logic.
> But, it would be better to have a slightly more advanced solution that spills 
> only the smallest possible set of batches to disk, then does a hybrid 
> in-memory, on-disk merge, saving the unnecessary write/read cycle.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5020) ExternalSortBatch has inconsistent notions of the memory limit

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5020:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> ExternalSortBatch has inconsistent notions of the memory limit
> --
>
> Key: DRILL-5020
> URL: https://issues.apache.org/jira/browse/DRILL-5020
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The memory limit cases for the spill-needed test seem inconsistent:
> For the test for in-memory sort:
> {code}
> long currentlyAvailable =  popConfig.getMaxAllocation() - 
> oAllocator.getAllocatedMemory();
> {code}
> For reaching the memory limit:
> {code}
> oAllocator.getAllocatedMemory() > .95 * oAllocator.getLimit()
> {code}
> That is, one uses {{oAllocator.getLimit}} ("the current maximum limit this 
> allocator imposes"), the other uses {{popConfig.getMaxAllocation}} ("The 
> maximum memory this operator can allocate".)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5014) ExternalSortBatch cache size, spill count differs from config setting

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5014:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> ExternalSortBatch cache size, spill count differs from config setting
> -
>
> Key: DRILL-5014
> URL: https://issues.apache.org/jira/browse/DRILL-5014
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> The ExternalSortBatch (ESB) operator sorts data while spilling to disk to 
> remain within a defined memory footprint. Spilling occurs based on a number 
> of factors. Among those are two config parameters:
> * {{drill.exec.sort.external.spill.group.size}}: The number of batches to 
> spill per spill event.
> * {{drill.exec.sort.external.spill.threshold}}: The number of batches to 
> accumulate in memory before starting a spill event.
> The expected behavior would be:
> * After the accumulated batches exceeds the threshold, and
> * More than "batch size" new batches have arrived since the last spill,
> * Spill half the accumulated records.
> That is if the threshold is 200, and the size is 150, we should accumulate 
> 200 batches, then spill 150 of them (leaving 50) and repeat.
> The actual behavior is:
> * When the accumulated records exceeds the threshold and,
> * More than "batch size" new batches have arrived since the last spill,
> * Spill half the accumulated records.
> The above can leave more batches in memory than desired, and spill more than 
> desired.
> The actual behavior for the (threshold=200, size=150) case is:
> {code}
> Before spilling, buffered batch count: 201
> After spilling, buffered batch count: 101
> Before spilling, buffered batch count: 251
> After spilling, buffered batch count: 126
> Before spilling, buffered batch count: 276
> After spilling, buffered batch count: 138
> Before spilling, buffered batch count: 288
> After spilling, buffered batch count: 144
> Before spilling, buffered batch count: 294
> After spilling, buffered batch count: 147
> Before spilling, buffered batch count: 297
> After spilling, buffered batch count: 149
> Before spilling, buffered batch count: 299
> After spilling, buffered batch count: 150
> Before spilling, buffered batch count: 300
> After spilling, buffered batch count: 150
> Before spilling, buffered batch count: 300
> {code}
> In short, the actual number of batches retained in memory is twice the spill 
> size, **NOT** the number set in the threshold. As a result, the ESB operator 
> will use more memory than expected.
> The work-around is to set a batch size that is half the threshold so that the 
> batch size (used in spill decisions) matches the actual spill count (as 
> implemented by the code.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5210) External Sort BatchGroup leaks memory if an OOM occurs during read

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5210:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> External Sort BatchGroup leaks memory if an OOM occurs during read
> --
>
> Key: DRILL-5210
> URL: https://issues.apache.org/jira/browse/DRILL-5210
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> The External Sort operator (batch) can spill to disk when it runs out of 
> memory. To do so, it uses a class called {{BatchGroup}}. Later, when the sort 
> merges spilled runs, {{BatchGroup}} reads the run back into memory one batch 
> at a time.
> If an OOM error occurs during the read operation, the partially-read batches 
> leak: they are not released. The fragment executor then issues a memory leak 
> error while shutting down the query.
> This error has probably not been caught until now because the {{BatchGroup}} 
> code does not make use of the fault injector. Elsewhere in the external sort, 
> we use the fault injector to insert a (simulated) OOM exception so we can 
> determine if clean-up occurs properly. No such fault injection code exists in 
> {{BatchGroup}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5285) Provide detailed, accurate estimate of size consumed by a record batch

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5285:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> Provide detailed, accurate estimate of size consumed by a record batch
> --
>
> Key: DRILL-5285
> URL: https://issues.apache.org/jira/browse/DRILL-5285
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> DRILL-5080 introduced a {{RecordBatchSizer}} that estimates the space taken 
> by a record batch and determines batch "density."
> Drill provides a large variety of vectors, each with their own internal 
> structure and collections of vectors. For example, fixed vectors use just a 
> data vector. Nullable vectors add an "is set" vector. Variable length vectors 
> add an offset vector. Repeated vectors add a second offset vector.
> The original {{RecordBatchSizer}} attempted to compute sizes for all these 
> vector types. But, the complexity got to be out of hand. This ticket requests 
> to simply bite the bullet and move the calculations into each vector type so 
> that the {{RecordBatchSizer}} can simply use the results of the calculations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (DRILL-2288) ScanBatch violates IterOutcome protocol for zero-row sources [was: missing JDBC metadata (schema) for 0-row results...]

[Rahul Challapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Challapalli closed DRILL-2288.


This fix is best verified with a unit test and the patch already has unit tests

> ScanBatch violates IterOutcome protocol for zero-row sources [was: missing 
> JDBC metadata (schema) for 0-row results...]
> ---
>
> Key: DRILL-2288
> URL: https://issues.apache.org/jira/browse/DRILL-2288
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Reporter: Daniel Barclay
>Assignee: Daniel Barclay
> Fix For: 1.4.0
>
> Attachments: Drill2288NoResultSetMetadataWhenZeroRowsTest.java
>
>
> The ResultSetMetaData object from getMetadata() of a ResultSet is not set up 
> (getColumnCount() returns zero, and trying to access any other metadata 
> throws IndexOutOfBoundsException) for a result set with zero rows, at least 
> for one from DatabaseMetaData.getColumns(...).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5008) Refactor, document and simplify ExternalSortBatch

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5008:
---
Fix Version/s: (was: 1.10.0)
   1.11.0

> Refactor, document and simplify ExternalSortBatch
> -
>
> Key: DRILL-5008
> URL: https://issues.apache.org/jira/browse/DRILL-5008
> Project: Apache Drill
>  Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> ExternalSortBatch provides a spillable sort operator for Drill. The code 
> works fine, but can be a hard to follow and understand. Make the following 
> changes to improve ease-of-use for developers:
> 1. Refactor the large methods into bite-sized chunks to aid understanding.
> 2. Provide additional explanation of the theory and operation of this 
> operator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)