[jira] [Created] (DRILL-2090) Update HBase storage plugin to support HBase 0.98

[Aditya Kishore (JIRA)]

Aditya Kishore created DRILL-2090:
-

 Summary: Update HBase storage plugin to support HBase 0.98
 Key: DRILL-2090
 URL: https://issues.apache.org/jira/browse/DRILL-2090
 Project: Apache Drill
  Issue Type: Sub-task
  Components: Storage - HBase
Affects Versions: 0.8.0
Reporter: Aditya Kishore
Assignee: Aditya Kishore


As the first step, we need to update the code and maven dependencies to use 
HBase 0.98 APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 30362: DRILL-2090: Update HBase storage plugin to support HBase 0.98

[Aditya Kishore]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30362/
---

Review request for drill, Jacques Nadeau and Steven Phillips.


Bugs: DRILL-2090
https://issues.apache.org/jira/browse/DRILL-2090


Repository: drill-git


Description
---

Updated the storage plugin to use HBase 0.98 Classes and APIs.

HBase maven dependency version updated to 0.98.7-hadoop2.


Diffs
-

  contrib/storage-hbase/pom.xml 7dcf350 
  
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseFilterBuilder.java
 5c49f18 
  
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java
 16ccc15 
  
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseUtils.java
 3c8aee2 
  
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/TableStatsCalculator.java
 9c8fbad 
  
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePStoreProvider.java
 b3947b4 
  
contrib/storage-hbase/src/test/java/org/apache/drill/hbase/HBaseTestsSuite.java 
96a887e 
  
contrib/storage-hbase/src/test/java/org/apache/drill/hbase/TestHBaseRegionScanAssignments.java
 bfc0558 
  distribution/pom.xml 2aefe08 
  exec/java-exec/pom.xml 90734a5 
  pom.xml 2087a0b 

Diff: https://reviews.apache.org/r/30362/diff/


Testing
---

All HBase unit tests (and other Drill unit tests) passed. 


Thanks,

Aditya Kishore

[jira] [Created] (DRILL-2091) NPE in AbstractSqlAccessor

[Oleg Danilov (JIRA)]

Oleg Danilov created DRILL-2091:
---

 Summary: NPE in AbstractSqlAccessor
 Key: DRILL-2091
 URL: https://issues.apache.org/jira/browse/DRILL-2091
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - JDBC
Affects Versions: 0.8.0
Reporter: Oleg Danilov
Assignee: Daniel Barclay (Drill/MapR)


I got the following NPE trying to use Drill with JDBC:

java.lang.NullPointerException
at 
org.apache.drill.exec.vector.accessor.AbstractSqlAccessor.getString(AbstractSqlAccessor.java:102)
at 
org.apache.drill.exec.vector.accessor.NullableFloat8Accessor.getString(NullableFloat8Accessor.java:87)
at 
org.apache.drill.jdbc.AvaticaDrillSqlAccessor.getString(AvaticaDrillSqlAccessor.java:66)
at 
net.hydromatic.avatica.AvaticaResultSet.getString(AvaticaResultSet.java:205)

It was caused by NULL value stored as DOUBLE.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2092) Incorrect result with count distinct and sum aggregates

[Victoria Markman (JIRA)]

Victoria Markman created DRILL-2092:
---

 Summary: Incorrect result with count distinct and sum aggregates
 Key: DRILL-2092
 URL: https://issues.apache.org/jira/browse/DRILL-2092
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization
Affects Versions: 0.8.0
Reporter: Victoria Markman
Assignee: Jinfeng Ni


test.json
{code}
{ "a1" : 10 , "b1" : 10 }
{ "a1" : 20 , "b1" : 20 }
{ "a1" : 20 , "b1" : 20}
{ "a1" : 30 , "b1" : 30 }
{ "a1" : null , "b1": null}
{code}

{code}
0: jdbc:drill:schema=dfs> select a1, count(distinct a1) from `test.json` group 
by a1;
+++
| a1 |   EXPR$1   |
+++
| 10 | 1  |
| 20 | 1  |
| 30 | 1  |
| null   | 0  |
+++
4 rows selected (0.096 seconds)
{code}

If  I add sum on the same column, I  get wrong result (null group is gone):

{code}
0: jdbc:drill:schema=dfs> select a1, count(distinct a1), sum(a1) from 
`test.json` group by a1;
++++
| a1 |   EXPR$1   |   EXPR$2   |
++++
| 10 | 1  | 10 |
| 20 | 1  | 40 |
| 30 | 1  | 30 |
++++
3 rows selected (0.137 seconds)
{code}

Non-distinct count works correctly:

{code}
0: jdbc:drill:schema=dfs> select a1, count(a1), sum(a1) from `test.json` group 
by a1;
++++
| a1 |   EXPR$1   |   EXPR$2   |
++++
| 10 | 1  | 10 |
| 20 | 2  | 40 |
| 30 | 1  | 30 |
| null   | 0  | null   |
++++
4 rows selected (0.187 seconds)
{code}

Plan for the query with the wrong result:
{code}
00-01  Project(a1=[$0], EXPR$1=[$1], EXPR$2=[$2])
00-02Project(a1=[$0], EXPR$1=[$3], EXPR$2=[$1])
00-03  HashJoin(condition=[IS NOT DISTINCT FROM($0, $2)], 
joinType=[inner])
00-05HashAgg(group=[{0}], EXPR$2=[SUM($0)])
00-07  Scan(groupscan=[EasyGroupScan [selectionRoot=/test.json, 
numFiles=1, columns=[`a1`], files=[maprfs:/test.json]]])
00-04Project(a10=[$0], EXPR$1=[$1])
00-06  HashAgg(group=[{0}], EXPR$1=[COUNT($0)])
00-08HashAgg(group=[{0}])
00-09  Scan(groupscan=[EasyGroupScan [selectionRoot=/test.json, 
numFiles=1, columns=[`a1`], files=[maprfs:/test.json]]])
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2093) Columns of time and timestamp data type are not stored correctly in json format on CTAS

[Victoria Markman (JIRA)]

Victoria Markman created DRILL-2093:
---

 Summary: Columns of time and timestamp data type are not stored 
correctly in json format on CTAS
 Key: DRILL-2093
 URL: https://issues.apache.org/jira/browse/DRILL-2093
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - JSON
Affects Versions: 0.8.0
Reporter: Victoria Markman
Assignee: Steven Phillips
Priority: Critical


I have a csv file and am trying to create matching file in json format.

{code}
alter session set `store.format` = 'json';

create table test_json(c_varchar, c_integer, c_bigint, c_smalldecimal, 
c_bigdecimal, c_float, c_date, c_time, c_timestamp, c_boolean) as
select
case when columns[0] = '' then cast(null as varchar(255)) else 
cast(columns[0] as varchar(255)) end,
case when columns[1] = '' then cast(null as integer) else 
cast(columns[1] as integer) end,
case when columns[2] = '' then cast(null as bigint) else 
cast(columns[2] as bigint) end,
case when columns[3] = '' then cast(null as decimal(18,4)) else 
cast(columns[3] as decimal(18, 4)) end,
case when columns[4] = '' then cast(null as decimal(38,4)) else 
cast(columns[4] as decimal(38, 4)) end,
case when columns[5] = '' then cast(null as float) else cast(columns[5] 
as float) end,
case when columns[6] = '' then cast(null as date) else cast(columns[6] 
as date) end,
case when columns[7] = '' then cast(null as time) else cast(columns[7] 
as time) end,
case when columns[8] = '' then cast(null as timestamp) else 
cast(columns[8] as timestamp) end,
case when columns[9] = '' then cast(null as boolean) else 
cast(columns[9] as boolean) end
from `t1.csv`;
{code}

Create table succeeds, but I can't read back time or timestamp:

{code}
0: jdbc:drill:schema=dfs> select cast(c_time as time) from test_json;
Query failed: RemoteRpcException: Failure while running fragment., Invalid 
format: "1970-01-01T08:13:16.000Z" is malformed at "70-01-01T08:13:16.000Z" [ 
b3f6c0c9-01e4-410f-919d-a899ede35ed9 on atsqa4-133.qa.lab:31010 ]
[ b3f6c0c9-01e4-410f-919d-a899ede35ed9 on atsqa4-133.qa.lab:31010 ]
Error: exception while executing query: Failure while executing query. 
(state=,code=0)

0: jdbc:drill:schema=dfs> select c_time from test_json;
++
|   c_time   |
++
| 1970-01-01T08:13:16.000Z |
| 1970-01-01T04:15:45.000Z |
| 1970-01-01T18:21:06.000Z |
| 1970-01-01T13:35:54.000Z |
| 1970-01-01T05:17:11.000Z |
++
5 rows selected (0.055 seconds)

0: jdbc:drill:schema=dfs> select cast(c_timestamp as timestamp) from test_json;
Query failed: RemoteRpcException: Failure while running fragment., Invalid 
format: "2014-03-16T03:55:21.000Z" is malformed at "T03:55:21.000Z" [ 
527a42e5-2eb7-41d7-a55b-c6cee0779db6 on atsqa4-133.qa.lab:31010 ]
[ 527a42e5-2eb7-41d7-a55b-c6cee0779db6 on atsqa4-133.qa.lab:31010 ]
Error: exception while executing query: Failure while executing query. 
(state=,code=0)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2094) Drill has problems with reading json fields when used in a subquery

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-2094:


 Summary: Drill has problems with reading json fields when used in 
a subquery
 Key: DRILL-2094
 URL: https://issues.apache.org/jira/browse/DRILL-2094
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization
Reporter: Rahul Challapalli
Assignee: Jinfeng Ni


git.commit.id.abbrev=3e33880

Data Set :
{code}
{
  "id" : 1,
  "list" : [1,2]
}
{code} 

The below query works
{code}
0: jdbc:drill:schema=dfs.drillTestDir> select id from `temp1.json` order by 
list[0];
++
| id |
++
| 1  |
++
1 row selected (0.146 seconds)
{code}

However when I used the same exact query as part of a sub-query, I get an error 
from drill

{code}
0: jdbc:drill:schema=dfs.drillTestDir> select s.id from (select id from 
`temp1.json` order by list[0]) s;
Query failed: SqlValidatorException: Table 'list' not found

Error: exception while executing query: Failure while executing query. 
(state=,code=0)
{code}

Explain plan also does not work and it returns the same problem



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 30362: DRILL-2090: Update HBase storage plugin to support HBase 0.98

[Steven Phillips]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30362/#review70037
---

Ship it!


Ship It!

- Steven Phillips


On Jan. 28, 2015, 10:50 a.m., Aditya Kishore wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30362/
> ---
> 
> (Updated Jan. 28, 2015, 10:50 a.m.)
> 
> 
> Review request for drill, Jacques Nadeau and Steven Phillips.
> 
> 
> Bugs: DRILL-2090
> https://issues.apache.org/jira/browse/DRILL-2090
> 
> 
> Repository: drill-git
> 
> 
> Description
> ---
> 
> Updated the storage plugin to use HBase 0.98 Classes and APIs.
> 
> HBase maven dependency version updated to 0.98.7-hadoop2.
> 
> 
> Diffs
> -
> 
>   contrib/storage-hbase/pom.xml 7dcf350 
>   
> contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseFilterBuilder.java
>  5c49f18 
>   
> contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java
>  16ccc15 
>   
> contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseUtils.java
>  3c8aee2 
>   
> contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/TableStatsCalculator.java
>  9c8fbad 
>   
> contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePStoreProvider.java
>  b3947b4 
>   
> contrib/storage-hbase/src/test/java/org/apache/drill/hbase/HBaseTestsSuite.java
>  96a887e 
>   
> contrib/storage-hbase/src/test/java/org/apache/drill/hbase/TestHBaseRegionScanAssignments.java
>  bfc0558 
>   distribution/pom.xml 2aefe08 
>   exec/java-exec/pom.xml 90734a5 
>   pom.xml 2087a0b 
> 
> Diff: https://reviews.apache.org/r/30362/diff/
> 
> 
> Testing
> ---
> 
> All HBase unit tests (and other Drill unit tests) passed. 
> 
> 
> Thanks,
> 
> Aditya Kishore
> 
>

[jira] [Created] (DRILL-2095) Order by on a repeated index inside a sub query results in an NPE

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-2095:


 Summary: Order by on a repeated index inside a sub query results 
in an NPE
 Key: DRILL-2095
 URL: https://issues.apache.org/jira/browse/DRILL-2095
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization
Reporter: Rahul Challapalli
Assignee: Jinfeng Ni
Priority: Critical


git.commit.id.abbrev=3e33880

Data Set :
{code}
{
  "id" : 1,
  "list" : [1,2]
}
{code}

The below query succeeds
{code}
0: jdbc:drill:schema=dfs.drillTestDir> select d.id id from `data.json` d order 
by d.list[0];
++
| id |
++
| 1  |
++
1 row selected (0.12 seconds)

{code}

However when we use the same query inside a sub-query we get an NPE
{code}
0: jdbc:drill:schema=dfs.drillTestDir> select s.id from (select d.id id from 
`data.json` d order by d.list[0]) s;
Query failed: NullPointerException: 

Error: exception while executing query: Failure while executing query. 
(state=,code=0)
{code}

Explain plan also results in the same NPE for the above query



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2096) Projecting a repeated list along with an order by on a subquery with flatten results in an error

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-2096:


 Summary: Projecting a repeated list along with an order by on a 
subquery with flatten results in an error
 Key: DRILL-2096
 URL: https://issues.apache.org/jira/browse/DRILL-2096
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Operators
Reporter: Rahul Challapalli
Assignee: Jason Altekruse
Priority: Critical


git.commit.id.abbrev=3e33880

Data Set :
{code}
{
"uid": 1,
"map": {
"rm": [
{
"mapid": "m1",
"mapvalue": {
"col1": 1,
"col2": [0,1,2,3,4,5]
},
"rptd": [
{ "a": "foo"},
{ "b": "boo"}
]
},
{
"mapid": "m2",
"mapvalue": {
"col1": 0,
"col2": []
},
"rptd": [
{ "a": "bar"},
{ "c": 1},
{ "d": 4.5}
]
}
]
}
}
{code}

{code}
0: jdbc:drill:schema=dfs.drillTestDir> select s.uid, s.rms.rptd from (select 
d.uid, flatten(d.map.rm) rms from `data.json` d) s order by s.rms.mapid;
Error: exception while executing query: null (state=,code=0)
{code}

Explain Plan for the above query
{code}
00-00Screen
00-01  Project(uid=[$0], EXPR$1=[$1])
00-02SelectionVectorRemover
00-03  Sort(sort0=[$2], dir0=[ASC])
00-04Project(uid=[$0], EXPR$1=[ITEM($1, 'rptd')], EXPR$2=[ITEM($1, 
'mapid')])
00-05  Project(uid=[$0], rms=[$2])
00-06Flatten(flattenField=[$2])
00-07  Project(EXPR$0=[$0], EXPR$1=[$1], EXPR$2=[ITEM($1, 
'rm')])
00-08Scan(groupscan=[EasyGroupScan 
[selectionRoot=/drill/testdata/flatten_operators/temp2.json, numFiles=1, 
columns=[`uid`, `map`.`rm`], 
files=[maprfs:/drill/testdata/flatten_operators/data.json]]])
{code}

However the below query works (It project an index in the repeated list)
{code}
select s.uid, s.rms.rptd[0] from (select d.uid, flatten(d.map.rm) rms from 
`data.json` d) s order by s.rms.mapid;
{code}

The logs are not entorely helpful here. Drill fails to propagate the error back 
for some reason. DRILL-2085 has been raised to track this
{code}
2015-01-28 19:21:10,232 [2b36c959-3087-63d6-9a20-ca45fdb875f1:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - State change requested.  PENDING --> 
RUNNING
2015-01-28 19:21:10,320 [UserServer-1] INFO  
o.a.drill.exec.work.foreman.Foreman - State change requested.  RUNNING --> 
CANCELED
2015-01-28 19:21:10,336 [UserServer-1] INFO  
o.a.drill.exec.work.foreman.Foreman - State change requested.  CANCELED --> 
COMPLETED
2015-01-28 19:21:10,336 [UserServer-1] WARN  
o.a.drill.exec.work.foreman.Foreman - Dropping request to move to COMPLETED 
state as query is already at CANCELED state (which is terminal).
{code} 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 30051: DRILL-1908: new window function implementation

[abdelhakim deneche]

> On Jan. 26, 2015, 5:51 p.m., Jacques Nadeau wrote:
> > exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/window/TestWindowFrame.java,
> >  line 38
> > 
> >
> > Please add a negative test case when using multiple partitions.  In 
> > that case, the failure should happen in the planning phase, not execution.

WindowPrel.getPhysicalOperator(PhysicalPlanCreator creator) has the following:
```
checkState(windows.size() == 1, "Only one window is expected in WindowPrel");
```


- abdelhakim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30051/#review69617
---


On Jan. 28, 2015, 7:50 p.m., abdelhakim deneche wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30051/
> ---
> 
> (Updated Jan. 28, 2015, 7:50 p.m.)
> 
> 
> Review request for drill.
> 
> 
> Bugs: DRILL-1908
> https://issues.apache.org/jira/browse/DRILL-1908
> 
> 
> Repository: drill-git
> 
> 
> Description
> ---
> 
> In order to fix DRILL-1487 a complete rewrite of the 
> StreamingWindowFrameRecordBatch was needed. This patch adds a new 
> WindowFrameRecordBatch that correctly handles window functions with or 
> without order by clauses.
> This code still lacks support for frame clauses and may be optimized to 
> reduce unneeded frame computations.
> 
> 
> Diffs
> -
> 
>   
> common/src/main/java/org/apache/drill/common/logical/data/AbstractBuilder.java
>  28424a5 
>   common/src/main/java/org/apache/drill/common/logical/data/Window.java 
> 6dba77c 
>   contrib/data/pom.xml 86075f2 
>   contrib/data/window-test-data/pom.xml PRE-CREATION 
>   exec/java-exec/pom.xml 90734a5 
>   exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java 
> 190c13f 
>   exec/java-exec/src/main/java/org/apache/drill/exec/opt/BasicOptimizer.java 
> 5288f5d 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/WindowPOP.java
>  17738ee 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/OverFinder.java
>  PRE-CREATION 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFrameBatchCreator.java
>  9b8929f 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFrameRecordBatch.java
>  26d23f2 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFrameTemplate.java
>  e2c7e9e 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFramer.java
>  9588cef 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameBatchCreator.java
>  PRE-CREATION 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameRecordBatch.java
>  PRE-CREATION 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameTemplate.java
>  PRE-CREATION 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFramer.java
>  PRE-CREATION 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java
>  3b7adca 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/StreamingWindowPrel.java
>  f1a8bc0 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/StreamingWindowPrule.java
>  00c20b2 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/WindowPrel.java
>  PRE-CREATION 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/WindowPrule.java
>  PRE-CREATION 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java
>  79603eb 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
>  f20627d 
>   
> exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/window/TestWindowFrame.java
>  a9d2ef8 
>   
> exec/java-exec/src/test/java/org/apache/drill/exec/sql/TestWindowFunctions.java
>  6eff6db 
> 
> Diff: https://reviews.apache.org/r/30051/diff/
> 
> 
> Testing
> ---
> 
> Unit tests are available to test window functions in mulitple scenarios:
> - b1.p1: single batch with a single partition
> - b1.p2: 2 batches, each containing a different parition
> - b2.p4: 2 batches and 4 partitions, one partition has rows in both batches
> - b3.p2: 3 batches and 2 partitions, one partition includes the whole 2nd 
> batch and has rows in 3 batches
> - b4.p4: 4 batches and 4 partitions, the partitions are arranged to test an 
> edge case: the 2nd time innerNext() is called, WindowFrameRecordBatch has 
> enough saved batches to call it's fram

FYI: "NoSchema Revolution" event with Hive Big Data Think Tank Meetup Group tonight

[Daniel Barclay]

For whomever is interested, the Hive Big Data Think Tank Meetup Group
is having a "NoSchema Revolution" event tonight.

See

http://www.meetup.com/SF-Bay-Areas-Big-Data-Think-Tank/events/219395393/


Daniel
--
Daniel Barclay
MapR Technologies

Re: Review Request 30051: DRILL-1908: new window function implementation

[abdelhakim deneche]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30051/
---

(Updated Jan. 28, 2015, 7:50 p.m.)


Review request for drill.


Changes
---

- updated GeneratorMapping and MappingSet variables to follow same convention 
used in HashJoinBatch and ChainedHashTable
- check and disable window function in sql parsing
- fail when schema changes
- use fail(Exception e) to report errors
- rename StreamingWindowPrel/StreamingWindowPrule to WindowPrel/WindowPrule


Bugs: DRILL-1908
https://issues.apache.org/jira/browse/DRILL-1908


Repository: drill-git


Description
---

In order to fix DRILL-1487 a complete rewrite of the 
StreamingWindowFrameRecordBatch was needed. This patch adds a new 
WindowFrameRecordBatch that correctly handles window functions with or without 
order by clauses.
This code still lacks support for frame clauses and may be optimized to reduce 
unneeded frame computations.


Diffs (updated)
-

  
common/src/main/java/org/apache/drill/common/logical/data/AbstractBuilder.java 
28424a5 
  common/src/main/java/org/apache/drill/common/logical/data/Window.java 6dba77c 
  contrib/data/pom.xml 86075f2 
  contrib/data/window-test-data/pom.xml PRE-CREATION 
  exec/java-exec/pom.xml 90734a5 
  exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java 190c13f 
  exec/java-exec/src/main/java/org/apache/drill/exec/opt/BasicOptimizer.java 
5288f5d 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/WindowPOP.java
 17738ee 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/OverFinder.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFrameBatchCreator.java
 9b8929f 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFrameRecordBatch.java
 26d23f2 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFrameTemplate.java
 e2c7e9e 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFramer.java
 9588cef 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameBatchCreator.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameRecordBatch.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameTemplate.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFramer.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java
 3b7adca 
  
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/StreamingWindowPrel.java
 f1a8bc0 
  
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/StreamingWindowPrule.java
 00c20b2 
  
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/WindowPrel.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/WindowPrule.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java
 79603eb 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
 f20627d 
  
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/window/TestWindowFrame.java
 a9d2ef8 
  
exec/java-exec/src/test/java/org/apache/drill/exec/sql/TestWindowFunctions.java 
6eff6db 

Diff: https://reviews.apache.org/r/30051/diff/


Testing
---

Unit tests are available to test window functions in mulitple scenarios:
- b1.p1: single batch with a single partition
- b1.p2: 2 batches, each containing a different parition
- b2.p4: 2 batches and 4 partitions, one partition has rows in both batches
- b3.p2: 3 batches and 2 partitions, one partition includes the whole 2nd batch 
and has rows in 3 batches
- b4.p4: 4 batches and 4 partitions, the partitions are arranged to test an 
edge case: the 2nd time innerNext() is called, WindowFrameRecordBatch has 
enough saved batches to call it's framer.doWork() without the need to call 
next(incoming)

All tests, except the last one, come in 2 variations: with and without "order 
by" clause


Thanks,

abdelhakim deneche

[jira] [Created] (DRILL-2097) Get rid of @Deprecated SimpleRootExec

[Chris Westin (JIRA)]

Chris Westin created DRILL-2097:
---

 Summary: Get rid of @Deprecated SimpleRootExec
 Key: DRILL-2097
 URL: https://issues.apache.org/jira/browse/DRILL-2097
 Project: Apache Drill
  Issue Type: Bug
Reporter: Chris Westin
Priority: Minor


SimpleRootExec is only used in tests. BaseRootExec is used in code, and not in 
any tests. We should be testing the class we're using (BaseRootExec), and not 
the one we've abandoned (SimpleRootExec).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2098) correct startup on windows

[Kristine (JIRA)]

Kristine created DRILL-2098:
---

 Summary: correct startup on windows
 Key: DRILL-2098
 URL: https://issues.apache.org/jira/browse/DRILL-2098
 Project: Apache Drill
  Issue Type: Bug
  Components: Documentation
Affects Versions: 0.7.0
Reporter: Kristine
Assignee: Bridget Bevens
 Fix For: 0.8.0


DRILL-2070 says 
{quote}
to execute the sqlline.bat on windows you must put quotes around the arguments, 
otherwise errors. Please update docs:
c:\drill\bin>sqlline.bat -u "jdbc:drill:zk=local"
https://cwiki.apache.org/confluence/display/DRILL/Installing+Drill+on+Windows
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2099) Order by has issues with repeated map within a repeated map

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-2099:


 Summary: Order by has issues with repeated map within a repeated 
map
 Key: DRILL-2099
 URL: https://issues.apache.org/jira/browse/DRILL-2099
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Operators
Reporter: Rahul Challapalli
Assignee: Mehant Baid
Priority: Critical


git.commit.id.abbrev=3e33880

Data Set :
{code}
{
  "uid" : 1,
  "map":{"rm": [
{"rptd": [{ "a": "foo"},{"b":"boo"}]},
{"rptd": [{ "a": "bar"},{"c":1},{"d":4.5}]}
  ]}
}
{code}

Query : The below query fails
{code}
select flatten(s1.rms.rptd) rptds from (select d.uid uid, flatten(d.map.rm) rms 
from `temp3.json` d order by d.uid) s1 ;
Query failed: RemoteRpcException: Failure while running fragment., index: -4, 
length: 4 (expected: range(0, 16384)) [ da0cb480-2fb8-4ea1-8d9a-ae7fc953c11d on 
qa-node190.qa.lab:31010 ]
[ da0cb480-2fb8-4ea1-8d9a-ae7fc953c11d on qa-node190.qa.lab:31010 ]


Error: exception while executing query: Failure while executing query. 
(state=,code=0)
{code}

The below 2 queries work as expected :
1. Use filter instead of order by
{code}
select flatten(s1.rms.rptd) rptds from (select d.uid uid, flatten(d.map.rm) rms 
from `temp3.json` d where d.uid=1) s1 ;
{code}

2. Remove the top level flatten
{code}
select s1.uid from (select d.uid uid, flatten(d.map.rm) rms from `temp3.json` d 
order by d.uid) s1 ;
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2100) Drill must clean up disk spill files after queries using an external sort complete.

[Abhishek Girish (JIRA)]

Abhishek Girish created DRILL-2100:
--

 Summary: Drill must clean up disk spill files after queries using 
an external sort complete.
 Key: DRILL-2100
 URL: https://issues.apache.org/jira/browse/DRILL-2100
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 0.8.0
Reporter: Abhishek Girish


Currently, after forcing queries to use an external sort by switching off hash 
join/agg causes spill-to-disk files accumulating. 

This causes issues with disk space availability when the spill is configured to 
be on the local file system (/tmp/drill). Also not optimal when configured to 
use DFS (custom). 

Drill must clean up all temporary files created after a query completes or 
after a drillbit restart. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2101) Incorrect data type is written to parquet file when NULL is cast to decimal

[Victoria Markman (JIRA)]

Victoria Markman created DRILL-2101:
---

 Summary: Incorrect data type is written to parquet file when NULL 
is cast to decimal
 Key: DRILL-2101
 URL: https://issues.apache.org/jira/browse/DRILL-2101
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Victoria Markman
Priority: Critical


{code}
create table t1(c1) as
select
cast(null as decimal(28,4))
from `t1.csv`;

message root {
  optional double c1; <-- Wrong, should be decimal
}
{code}

This is very commonly used construct to convert csv files to parquet files, 
that's why I'm marking this bug as critical.
{code}
create table t2 as 
select
case when columns[3] = '' then cast(null as decimal(28,4)) else 
cast(columns[3] as decimal(28, 4)) end
from `t1.csv`;
{code}


Correct - cast string literal to decimal
{code}
create table t3(c1) as
select
cast('12345678901234567890.1234' as decimal(28,4))
from `t1.csv`;

message root {
  required fixed_len_byte_array(12) c1 (DECIMAL(28,4));
}
{code}

Correct - cast literal from csv file as decimal

{code}
create table t4(c1) as
select
cast(columns[3] as decimal(28,4))
from `t1.csv`;

message root {
  optional fixed_len_byte_array(12) c1 (DECIMAL(28,4));
}
{code}

Correct - case statement (no null involved)

{code}
create table t5(c1) as
select
case when columns[3] = '' then cast('' as decimal(28,4)) else 
cast(columns[3] as decimal(28,4)) end
from `t1.csv`;

message root {
  optional fixed_len_byte_array(12) c1 (DECIMAL(28,4));
}
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2102) Empty query yields HTTP 500 error and poor error message.

[Daniel Barclay (Drill/MapR) (JIRA)]

Daniel Barclay (Drill/MapR) created DRILL-2102:
--

 Summary: Empty query yields HTTP 500 error and poor error message.
 Key: DRILL-2102
 URL: https://issues.apache.org/jira/browse/DRILL-2102
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - HTTP
Reporter: Daniel Barclay (Drill/MapR)
Assignee: Jason Altekruse


In the Drill web UI, going to the query page and clicking Submit (without 
entering any query text) yields a page saying:

HTTP ERROR 500

Problem accessing /query. Reason:

Request failed.

(rather than giving, say, a syntax error reporting that the query is empty).





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-2102) Empty query yields HTTP 500 error and poor error message.

[Jason Altekruse (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Altekruse resolved DRILL-2102.

Resolution: Duplicate

> Empty query yields HTTP 500 error and poor error message.
> -
>
> Key: DRILL-2102
> URL: https://issues.apache.org/jira/browse/DRILL-2102
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Reporter: Daniel Barclay (Drill/MapR)
>Assignee: Jason Altekruse
>
> In the Drill web UI, going to the query page and clicking Submit (without 
> entering any query text) yields a page saying:
> HTTP ERROR 500
> Problem accessing /query. Reason:
> Request failed.
> (rather than giving, say, a syntax error reporting that the query is empty).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2103) Queries with aggregate function on columns containing null values fail

[Abhishek Girish (JIRA)]

Abhishek Girish created DRILL-2103:
--

 Summary: Queries with aggregate function on columns containing 
null values fail
 Key: DRILL-2103
 URL: https://issues.apache.org/jira/browse/DRILL-2103
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Data Types
Reporter: Abhishek Girish
Assignee: Daniel Barclay (Drill/MapR)


> describe store_sales;
...
| ss_item_sk  | INTEGER| NO  |
...
| ss_sales_price | DOUBLE | NO  |
...

> select count(*) from store_sales where ss_sales_price is NULL;
129666 
> select count(*) from store_sales where ss_sales_price is NOT NULL;
2750738

> select avg(ss_item_sk) from store_sales;
Query failed: Query failed: Unexpected exception during fragment 
initialization: Internal error: Error while applying rule 
DrillReduceAggregatesRule, args 
[rel#852769:AggregateRel.NONE.ANY([]).[](child=rel#852768:Subset#3.NONE.ANY([]).[],group={},EXPR$0=AVG($0))]

Error: exception while executing query: Failure while executing query. 
(state=,code=0)

> select avg(ss_sales_price) from store_sales;
Query failed: Query failed: Unexpected exception during fragment 
initialization: Internal error: Error while applying rule 
DrillReduceAggregatesRule, args 
[rel#851712:AggregateRel.NONE.ANY([]).[](child=rel#851711:Subset#3.NONE.ANY([]).[],group={},EXPR$0=AVG($0))]

Error: exception while executing query: Failure while executing query. 
(state=,code=0)

Log attached. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2104) Web interface hangs after query-submission error

[Daniel Barclay (Drill/MapR) (JIRA)]

Daniel Barclay (Drill/MapR) created DRILL-2104:
--

 Summary: Web interface hangs after query-submission error
 Key: DRILL-2104
 URL: https://issues.apache.org/jira/browse/DRILL-2104
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - HTTP
Reporter: Daniel Barclay (Drill/MapR)
Assignee: Jason Altekruse


On the query page, getting an error seems to jam up the web application.

Example 1:
1. Go to query page.
2. Click on Submit button (with empty query, type defaulted to "SQL").
3. (Get "HTTP ERROR 500 \n Problem accessing /query. Reason: \nRequest 
failed." error.)
4. Click on browser Back button.
5. (Get query page.)
6. Click on Submit button.
7. Web application hangs--browser indicates it's waiting.  (In current case, 
Mozilla SeaMonkey says "Waiting for localhost..." and "Stop" icon is enabled.)

Example 2:
1.  Go to query page.
2.  Copy sample query ("SELECT * FROM cp.`employee.json` LIMIT 20") and paste 
into query box.
3.  Click Submit button.
4.  (Get tabular results page.)
5.  Click on browser Back button.
6.  (Get query page, with query still there.)
7.  Edit query, adding apparent syntax error (e.g., append ";" to end).
8.  Click Submit button.
9.  (Get "HTTP ERROR 500" page (rather than good "syntax error" message).)
10.  Click on browser Back button.
11.  (Get query page, with (bad) query still there.)
12.  Fix query (e.g., remove bad appended ";" from end).
13.  Click Submit button.
14.  Web application hangs--browser indicates it's waiting.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2105) Query fails when using flatten on JSON data where some documents have an empty array

[Andries Engelbrecht (JIRA)]

Andries Engelbrecht created DRILL-2105:
--

 Summary: Query fails when using flatten on JSON data where some 
documents have an empty array
 Key: DRILL-2105
 URL: https://issues.apache.org/jira/browse/DRILL-2105
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Operators
Affects Versions: 0.7.0
 Environment: MFS with JSON
Reporter: Andries Engelbrecht
Assignee: Chris Westin


Drill query fails when using flatten on an array, where some records contain an 
empty array. Especially with larger data sets where the number of JSON 
documents are greater than 100k.

Using twitter data as sample.

select flatten (entities.hashtags) from dfs.foo.`file.json`;

Empty array
  "entities": {
"trends": [],
"symbols": [],
"urls": [
  {
"expanded_url": "http://on.nfl.com/1BkThQF";,
"indices": [
  118,
  140
],
"display_url": "on.nfl.com/1BkThQF",
"url": "http://t.co/Unr5KFy6hG";
  }
],
"hashtags": [],
"user_mentions": [
  {
"id": 19362299,
"name": "NFL Network",
"indices": [
  3,
  14
],
"screen_name": "nflnetwork",
"id_str": "19362299"
  }
]
  },

Array with content

  "entities": {
"trends": [],
"symbols": [],
"urls": [],
"hashtags": [
  {
"text": "djpreps",
"indices": [
  47,
  55
]
  },
  {
"text": "MSPreps",
"indices": [
  56,
  64
]
  }
],
"user_mentions": []
  },


Log output

2015-01-27 02:26:13,478 [2b3908b9-cf08-3fd5-3bd8-ebb6bb5b70f1:foreman] INFO  
o.a.d.e.store.mock.MockStorageEngine - Failure while attempting to check for 
Parquet metadata file.
java.io.IOException: Open failed for file: /data/twitter/nfl/2015, error: 
Invalid argument (22)
at com.mapr.fs.MapRClientImpl.open(MapRClientImpl.java:191) 
~[maprfs-4.0.1.28318-mapr.jar:4.0.1.28318-mapr]
at com.mapr.fs.MapRFileSystem.open(MapRFileSystem.java:776) 
~[maprfs-4.0.1.28318-mapr.jar:4.0.1.28318-mapr]
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:800) 
~[hadoop-common-2.4.1-mapr-1408.jar:na]
at 
org.apache.drill.exec.store.dfs.shim.fallback.FallbackFileSystem.open(FallbackFileSystem.java:94)
 ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at 
org.apache.drill.exec.store.dfs.BasicFormatMatcher$MagicStringMatcher.matches(BasicFormatMatcher.java:138)
 ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at 
org.apache.drill.exec.store.dfs.BasicFormatMatcher.isReadable(BasicFormatMatcher.java:107)
 ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at 
org.apache.drill.exec.store.parquet.ParquetFormatPlugin$ParquetFormatMatcher.isDirReadable(ParquetFormatPlugin.java:232)
 [drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at 
org.apache.drill.exec.store.parquet.ParquetFormatPlugin$ParquetFormatMatcher.isReadable(ParquetFormatPlugin.java:212)
 [drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at 
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.create(WorkspaceSchemaFactory.java:141)
 [drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at 
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.create(WorkspaceSchemaFactory.java:58)
 [drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at 
org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry(ExpandingConcurrentMap.java:96)
 [drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at 
org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get(ExpandingConcurrentMap.java:90)
 [drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at 
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable(WorkspaceSchemaFactory.java:273)
 [drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at 
net.hydromatic.optiq.jdbc.SimpleOptiqSchema.getTable(SimpleOptiqSchema.java:75) 
[optiq-core-0.9-drill-r12.jar:na]
at 
net.hydromatic.optiq.prepare.OptiqCatalogReader.getTableFrom(OptiqCatalogReader.java:87)
 [optiq-core-0.9-drill-r12.jar:na]
at 
net.hydromatic.optiq.prepare.OptiqCatalogReader.getTable(OptiqCatalogReader.java:70)
 [optiq-core-0.9-drill-r12.jar:na]
at 
net.hydromatic.optiq.prepare.OptiqCatalogReader.getTable(OptiqCatalogReader.java:42)
 [optiq-core-0.9-drill-r12.jar:na]
at 
org.eigenbase.sql.validate.EmptyScope.getTableNamespace(EmptyScope.java:67) 
[optiq-core-0.9-drill-r12.jar:na]
at 
org.eigenbase.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:75)
 [optiq-core-0.9-drill-r12.jar:na]
at 
org.eigenbase.sql.validate.AbstractNamespace.validate(AbstractNamespac

[jira] [Created] (DRILL-2106) Flatten together with order by on a list within a list is causing an error

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-2106:


 Summary: Flatten together with order by on a list within a list is 
causing an error
 Key: DRILL-2106
 URL: https://issues.apache.org/jira/browse/DRILL-2106
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Operators
Reporter: Rahul Challapalli
Assignee: Mehant Baid
Priority: Critical


git.commit.id.abbrev=3e33880

Data Set :
{code}
{
  "uid" : 1,
  "lst_lst" : [[1,2,3,4,5],[2,3,4,5,6]]
}
{code}

The below query fails
{code}
select flatten(lst) from (select uid, flatten(d.lst_lst) lst from `temp4.json` 
d) s1 order by s1.lst[3];
{code}

When we remove the top level flatten the below query works
{code}
select uid from (select uid, flatten(d.lst_lst) lst from `temp4.json` d) s1 
order by s1.lst[3];
{code}

Attached the error log



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2107) Hash Join throw IOBE for a query with exists subquery.

[Jinfeng Ni (JIRA)]

Jinfeng Ni created DRILL-2107:
-

 Summary: Hash Join throw IOBE for a query with exists subquery. 
 Key: DRILL-2107
 URL: https://issues.apache.org/jira/browse/DRILL-2107
 Project: Apache Drill
  Issue Type: New Feature
  Components: Execution - Operators
Reporter: Jinfeng Ni
Assignee: Chris Westin
Priority: Critical


I hit an IOBE for TestTpchDistributed Q4, when I tried to enable an optimizer 
rule.  Then, I simplified Q4 to the following, and still re-produce the same 
IOBE.

{code}
select
  o.o_orderpriority
from
  cp.`tpch/orders.parquet` o
where
  exists (
select
  *
from
  cp.`tpch/lineitem.parquet` l
where
  l.l_orderkey = o.o_orderkey
  )
;
{code}

Stack trace of the exception:
{code}
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
 at java.util.ArrayList.rangeCheck(ArrayList.java:635) ~[na:1.7.0_45]
 at java.util.ArrayList.get(ArrayList.java:411) ~[na:1.7.0_45]
 at 
org.apache.drill.exec.record.VectorContainer.getValueAccessorById(VectorContainer.java:232)
 ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.RecordBatchLoader.getValueAccessorById(RecordBatchLoader.java:149)
 ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
 at 
org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.getValueAccessorById(UnorderedReceiverBatch.java:132)
 ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
 at 
org.apache.drill.exec.test.generated.HashTableGen307.doSetup(HashTableTemplate.java:71)
 ~[na:na]
 at 
org.apache.drill.exec.test.generated.HashTableGen307.updateBatches(HashTableTemplate.java:473)
 ~[na:na]
 at 
org.apache.drill.exec.test.generated.HashJoinProbeGen313.executeProbePhase(HashJoinProbeTemplate.java:139)
 ~[na:na]
 at 
org.apache.drill.exec.test.generated.HashJoinProbeGen313.probeAndProject(HashJoinProbeTemplate.java:223)
 ~[na:na]
 at 
org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext(HashJoinBatch.java:227)
 ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]

{code}

The physical plan seems to be correct, after enabling the new rule. Actually, 
if I disable HashJoin, and use merge join for the query, it works fine. So, 
seems the IOBE exposes some bug in HashJoin.

To re-produce this issue,  two options:
 1 )  - Modify DrillRuleSets.java, remove the comment before SwapJoinRule 
   - alter session set `planner.slice_target` = 10;
   - run the query
 
 2) use the attached physical plan in json file, and use "submitplan" to submit 
the physical plan.

For comparison, I also attached the physical plan when disabling hashjoin (use 
merge join), and the explain plan at physical operator level. 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2108) Running SQLLine with bad command syntax yields index exception (not usage message)

[Daniel Barclay (Drill/MapR) (JIRA)]

Daniel Barclay (Drill/MapR) created DRILL-2108:
--

 Summary: Running SQLLine with bad command syntax yields index 
exception (not usage message)
 Key: DRILL-2108
 URL: https://issues.apache.org/jira/browse/DRILL-2108
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - CLI
Reporter: Daniel Barclay (Drill/MapR)
Assignee: Jacques Nadeau


Running ".../sqlline -u" yields an ArrayIndexOutOfBoundsException and a Java 
stack trace (rather than reporting the command syntax error and giving a usage 
message).




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2109) Parquet reader does not return correct data for required dictionary encoded fields

[Parth Chandra (JIRA)]

Parth Chandra created DRILL-2109:


 Summary: Parquet reader does not return correct data for required 
dictionary encoded fields
 Key: DRILL-2109
 URL: https://issues.apache.org/jira/browse/DRILL-2109
 Project: Apache Drill
  Issue Type: Bug
Reporter: Parth Chandra
Assignee: Jason Altekruse


The reader returns all the data for the column but the records are in the wrong 
order. If you read more than one column, then the values do not match up.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

HBase storage plugin updated to support HBase 0.98

[Aditya]

Folks,

With the commit of DRILL-2090[1] patch, I have updated the HBase storage
plugin to support HBase 0.98. As a result, from now[2] onward Drill will
not work with HBase 0.94.

Regards,
aditya...

[1] https://issues.apache.org/jira/browse/DRILL-2090
[2] Release 0.8.0

-- Forwarded message --
From: 
Date: Wed, Jan 28, 2015 at 6:55 PM
Subject: drill git commit: DRILL-2090: Update HBase storage plugin to
support HBase 0.98
To: comm...@drill.apache.org


Repository: drill
Updated Branches:
  refs/heads/master b6a387878 -> 54b5f80a5


DRILL-2090: Update HBase storage plugin to support HBase 0.98
​...​

[jira] [Created] (DRILL-2110) all_text_mode for json works only with alter system not with alter session

[Sudhakar Thota (JIRA)]

Sudhakar Thota created DRILL-2110:
-

 Summary: all_text_mode for json works only with alter system not 
with alter session
 Key: DRILL-2110
 URL: https://issues.apache.org/jira/browse/DRILL-2110
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - CLI
Affects Versions: 0.7.0
 Environment: MapR 4.0.2 sandbox
Reporter: Sudhakar Thota
Assignee: Jacques Nadeau


1. alter session set `store.json.all_text_mode` = true;
fails to show the json document but 
alter system set `store.json.all_text_mode` = true;
shows the documents on MapR 4.0.2 sandbox with drill-0.7. 

2. Exhibiting same behavior with Drill Explorer. 
-
[root@maprdemo ~]# ls -l /tmp/anonym.json 
-rwxr-xr-x 1 root root 9966583 Jan 28 18:26 /tmp/anonym.json
[root@maprdemo christian]# pwd
/mapr/demo.mapr.com/user/sthota/christian
[root@maprdemo christian]# cp /tmp/anonym.json .
[root@maprdemo christian]# ls -l 
/mapr/demo.mapr.com/user/sthota/christian/anonym.json 
-rwxr-xr-x 1 root root 9966583 Jan 28 18:28 
/mapr/demo.mapr.com/user/sthota/christian/anonym.json
[root@maprdemo christian]# su - mapr
[mapr@maprdemo ~]$ sqlline
sqlline version 1.1.6
0: jdbc:drill:> show databases;
+-+
| SCHEMA_NAME |
+-+
| hive.default |
| dfs.default |
| dfs.logs|
| dfs.root|
| dfs.views   |
| dfs.clicks  |
| dfs.tmp |
| sys |
| maprdb  |
| cp.default  |
| INFORMATION_SCHEMA |
+-+
11 rows selected (0.222 seconds)
0: jdbc:drill:> select * from dfs.`/user/sthota/christian/anonym.json` limit 4;
Query failed: Query stopped., You tried to write a BigInt type when you are 
using a ValueWriter of type NullableFloat8WriterImpl. [ 
e3a8c338-ba88-435c-b7d4-7b8d60d61fb4 on maprdemo:31010 ]


Error: exception while executing query: Failure while executing query. 
(state=,code=0)
0: jdbc:drill:> alter session set `store.json.all_text_mode`=true;
+++
| ok |  summary   |
+++
| true   | store.json.all_text_mode updated. |
+++
1 row selected (0.088 seconds)
0: jdbc:drill:> 

0: jdbc:drill:> alter system set `store.json.all_text_mode`=true;
+++
| ok |  summary   |
+++
| true   | store.json.all_text_mode updated. |
+++
1 row selected (0.064 seconds)
0: jdbc:drill:> select * from dfs.`/user/sthota/christian/anonym.json` limit 4;
+++++++
| eq | tm |seq |guid|val |log |
+++++++
| in4ktVxSI  | 133614.84 | 108| 
61429740-6d15-165e-3388-c6d4e19af9c1 | …….. |
+++++++
4 rows selected (0.997 seconds)
0: jdbc:drill:> 

--



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2111) Export HBase namespaces as sub-schemas

[Aditya Kishore (JIRA)]

Aditya Kishore created DRILL-2111:
-

 Summary: Export HBase namespaces as sub-schemas
 Key: DRILL-2111
 URL: https://issues.apache.org/jira/browse/DRILL-2111
 Project: Apache Drill
  Issue Type: Sub-task
  Components: Storage - HBase
Reporter: Aditya Kishore
Assignee: Aditya Kishore


HBase 0.98 introduces support for 
[namespace|https://issues.apache.org/jira/browse/HBASE-8015], which Drill 
should export as sub-schemas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 28686: For DRILL-1735, hook up JDBC connection closing, and fix follow-on resource bugs.

[Daniel Barclay]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28686/
---

Review request for drill, Mehant Baid and Parth Chandra.


Bugs: DRILL-1735
https://issues.apache.org/jira/browse/DRILL-1735


Repository: drill-git


Description
---

This change hooks in the previously written but unused DrillHandler so that a 
local-mode JDBC connection shuts down its embedded DrillBit. 

It also fixes a chain of follow-on bugs (mostly resource-deallocation 
omissions) exposed by the original bug fixes.  (See the commit message for the 
list of bugs/fixes and other changes.)


Diffs
-

  common/src/main/java/org/apache/drill/common/config/DrillConfig.java 4c6c766 
  exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java aa5b702 
  exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java 8e9d395 
  exec/java-exec/src/main/java/io/netty/buffer/FakeAllocator.java 3de0a75 
  exec/java-exec/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java 
c0de544 
  exec/java-exec/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java 
dfdc114 
  exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java 
04b955b 
  
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferAllocator.java 
83d9d1e 
  exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 
dc47f4e 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/FragmentRoot.java
 1721fcf 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/RootExec.java 
a644c34 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java 
d68a5b5 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScreenCreator.java
 d884200 
  
exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatch.java
 2bb29e5 
  
exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractSingleRecordBatch.java
 f895f47 
  exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java 
0a8ece5 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/BasicServer.java 
c00df4e 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcBus.java 96c9911 
  
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryResultHandler.java
 9015a16 
  
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserResultsListener.java
 9f83a4f 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/BootStrapContext.java 
3da2ea9 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/accessor/AbstractSqlAccessor.java
 1605c7d 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/accessor/SqlAccessor.java
 b8480b4 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/CancelableQuery.java 
5b11943 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java 
5bc3da1 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 
b33042b 
  
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java
 2de3592 
  
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryStatus.java
 4e18da6 
  
exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentExecutor.java
 9ffe643 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/user/UserWorker.java 
854f474 
  exec/java-exec/src/main/java/parquet/hadoop/CodecFactoryExposer.java 4d107e4 
  exec/java-exec/src/main/java/parquet/hadoop/ColumnChunkIncReadStore.java 
2a3124c 
  
exec/java-exec/src/main/java/parquet/hadoop/ColumnChunkPageWriteStoreExposer.java
 0e9dec0 
  exec/java-exec/src/test/java/org/apache/drill/exec/ExecTest.java 0272b23 
  
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestComparisonFunctions.java
 609bc14 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/AvaticaDrillSqlAccessor.java 
cf5829a 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillAccessorList.java 7594783 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillConnectionConfig.java 
584bbb4 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillConnectionImpl.java 
4c54780 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillCursor.java fbe611f 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillResultSet.java 88a6c6d 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/Driver.java 974e786 
  
exec/jdbc/src/main/java/org/apache/drill/jdbc/InvalidCursorStateSqlException.java
 PRE-CREATION 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/JdbcApiSqlException.java 
PRE-CREATION 
  exec/jdbc/src/test/java/org/apache/drill/jdbc/DrillResultSetTest.java 
PRE-CREATION 
  exec/jdbc/src/test/java/org/apache/drill/jdbc/DriverTest.java PRE-CREATION 
  
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Bug1735ConnectionCloseTest.java
 PRE-CREATION 
  
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Bug1735ResultSetCloseReleasesBuffersTest.java
 PRE-

[jira] [Created] (DRILL-2112) Throw helpful error message with unsupported KVGen functionality

[Neeraja (JIRA)]

Neeraja created DRILL-2112:
--

 Summary: Throw helpful error message with unsupported KVGen 
functionality 
 Key: DRILL-2112
 URL: https://issues.apache.org/jira/browse/DRILL-2112
 Project: Apache Drill
  Issue Type: Bug
Reporter: Neeraja
 Fix For: 0.9.0


KVGen function currently doesn't support maps with changing data types on 
values.  We should throw helpful error message indicating that this is not an 
unsupported functionality rather than execution exception.

0: jdbc:drill:zk=local> select attributes from 
dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json` 
limit 2;
++
| attributes |
++
| {"By Appointment Only":true,"Good 
For":{},"Ambience":{},"Parking":{},"Music":{},"Hair Types Specialized 
In":{},"Payment Types":{},"Dietary Restrictions":{}} |
| {"Take-out":true,"Good 
For":{"dessert":false,"latenight":false,"lunch":true,"dinner":false,"breakfast":false,"brunch":false},"Caters":false,"Noise
 Level":"average","Takes 
Reservations":false,"Delivery":false,"Ambience":{"romantic":false,"intimate":false,"touristy":false,"hipster":false,"divey":false,"classy":false,"trendy":false,"upscale":false,"casual":false},"Parking":{"garage":false,"street":false,"validated":false,"lot":true,"valet":false},"Has
 TV":true,"Outdoor Seating":false,"Attire":"casual","Alcohol":"none","Waiter 
Service":true,"Accepts Credit Cards":true,"Good for Kids":true,"Good For 
Groups":true,"Price Range":1,"Music":{},"Hair Types Specialized In":{},"Payment 
Types":{},"Dietary Restrictions":{}} |
++
2 rows selected (0.274 seconds)
0: jdbc:drill:zk=local> select kvgen(attributes) from 
dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json` 
limit 2;
Query failed: Query failed: Failure while running fragment., 
org.apache.drill.exec.vector.complex.impl.NullableBitWriterImpl cannot be cast 
to org.apache.drill.exec.vector.complex.impl.SingleMapWriter [ 
24bcf2c3-c995-4451-8e11-4e804050388f on 10.0.0.6:31010 ]
[ 24bcf2c3-c995-4451-8e11-4e804050388f on 10.0.0.6:31010 ]


Error: exception while executing query: Failure while executing query. 
(state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-2113) Support group by on aliases

[Neeraja (JIRA)]

Neeraja created DRILL-2113:
--

 Summary: Support group by on aliases
 Key: DRILL-2113
 URL: https://issues.apache.org/jira/browse/DRILL-2113
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Neeraja
 Fix For: Future


Currently Drill doesn't support group by on aliases. This however works for 
order by clauses. Using group by based on alias is a common pattern especially 
if there are expressions used for grouping.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 28686: For DRILL-1735, hook up JDBC connection closing, and fix follow-on resource bugs.

[Daniel Barclay]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28686/
---

(Updated Jan. 29, 2015, 5:47 a.m.)


Review request for drill, Mehant Baid and Parth Chandra.


Bugs: DRILL-1735
https://issues.apache.org/jira/browse/DRILL-1735


Repository: drill-git


Description
---

This change hooks in the previously written but unused DrillHandler so that a 
local-mode JDBC connection shuts down its embedded DrillBit. 

It also fixes a chain of follow-on bugs (mostly resource-deallocation 
omissions) exposed by the original bug fixes.  (See the commit message for the 
list of bugs/fixes and other changes.)


Diffs
-

  common/src/main/java/org/apache/drill/common/config/DrillConfig.java 4c6c766 
  exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java aa5b702 
  exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java 8e9d395 
  exec/java-exec/src/main/java/io/netty/buffer/FakeAllocator.java 3de0a75 
  exec/java-exec/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java 
c0de544 
  exec/java-exec/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java 
dfdc114 
  exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java 
04b955b 
  
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferAllocator.java 
83d9d1e 
  exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 
dc47f4e 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/FragmentRoot.java
 1721fcf 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/RootExec.java 
a644c34 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java 
d68a5b5 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScreenCreator.java
 d884200 
  
exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatch.java
 2bb29e5 
  
exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractSingleRecordBatch.java
 f895f47 
  exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java 
0a8ece5 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/BasicServer.java 
c00df4e 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcBus.java 96c9911 
  
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryResultHandler.java
 9015a16 
  
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserResultsListener.java
 9f83a4f 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/BootStrapContext.java 
3da2ea9 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/accessor/AbstractSqlAccessor.java
 1605c7d 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/accessor/SqlAccessor.java
 b8480b4 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/CancelableQuery.java 
5b11943 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java 
5bc3da1 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 
b33042b 
  
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java
 2de3592 
  
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryStatus.java
 4e18da6 
  
exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentExecutor.java
 9ffe643 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/user/UserWorker.java 
854f474 
  exec/java-exec/src/main/java/parquet/hadoop/CodecFactoryExposer.java 4d107e4 
  exec/java-exec/src/main/java/parquet/hadoop/ColumnChunkIncReadStore.java 
2a3124c 
  
exec/java-exec/src/main/java/parquet/hadoop/ColumnChunkPageWriteStoreExposer.java
 0e9dec0 
  exec/java-exec/src/test/java/org/apache/drill/exec/ExecTest.java 0272b23 
  
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestComparisonFunctions.java
 609bc14 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/AvaticaDrillSqlAccessor.java 
cf5829a 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillAccessorList.java 7594783 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillConnectionConfig.java 
584bbb4 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillConnectionImpl.java 
4c54780 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillCursor.java fbe611f 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillResultSet.java 88a6c6d 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/Driver.java 974e786 
  
exec/jdbc/src/main/java/org/apache/drill/jdbc/InvalidCursorStateSqlException.java
 PRE-CREATION 
  exec/jdbc/src/main/java/org/apache/drill/jdbc/JdbcApiSqlException.java 
PRE-CREATION 
  exec/jdbc/src/test/java/org/apache/drill/jdbc/DrillResultSetTest.java 
PRE-CREATION 
  exec/jdbc/src/test/java/org/apache/drill/jdbc/DriverTest.java PRE-CREATION 
  
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Bug1735ConnectionCloseTest.java
 PRE-CREATION 
  
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Bug1735ResultS

[jira] [Created] (DRILL-2114) Improve profile page in web ui to show additional memory stats and fragment status

[Jacques Nadeau (JIRA)]

Jacques Nadeau created DRILL-2114:
-

 Summary: Improve profile page in web ui to show additional memory 
stats and fragment status
 Key: DRILL-2114
 URL: https://issues.apache.org/jira/browse/DRILL-2114
 Project: Apache Drill
  Issue Type: Improvement
  Components: Client - HTTP
Reporter: Jacques Nadeau
Assignee: Jacques Nadeau
 Fix For: 0.8.0






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 30404: DRILL-1747

[Jacques Nadeau]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30404/
---

Review request for drill, Chris Westin and Steven Phillips.


Repository: drill-git


Description
---

Update Workmanager to report status of fragments back to Foreman so that 
profiles show intermediate status.


Diffs
-

  exec/java-exec/src/main/java/org/apache/drill/exec/work/StatusProvider.java 
9215f43 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java 
5bc3da1 
  
exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/AbstractStatusReporter.java
 c7ac311 
  
exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentExecutor.java
 9ffe643 

Diff: https://reviews.apache.org/r/30404/diff/


Testing
---

Hand testing on long running queries to ensure query stats were updated during 
query.


Thanks,

Jacques Nadeau

Review Request 30405: DRILL-2114: Update Profile WebUI

[Jacques Nadeau]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30405/
---

Review request for drill, Jason Altekruse and Steven Phillips.


Repository: drill-git


Description
---

Enhance web ui to provide more information including hostname, memory 
allocation and fragment status


Diffs
-

  exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 
dc47f4e 
  exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentStats.java 
4431235 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRestServer.java
 e8cb4ba 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/ProfileResources.java
 58b3d4e 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/ProfileWrapper.java
 80c08d3 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/Comparators.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/Filters.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/FragmentWrapper.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/OperatorPathBuilder.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/OperatorWrapper.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileWrapper.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/TableBuilder.java
 PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 
b33042b 
  
exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/AbstractStatusReporter.java
 c7ac311 

Diff: https://reviews.apache.org/r/30405/diff/


Testing
---

manual verification of profile layout on test queries.


Thanks,

Jacques Nadeau

Re: Review Request 30404: DRILL-1747: Intermediate query profile reporting

[Jacques Nadeau]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30404/
---

(Updated Jan. 29, 2015, 7:35 a.m.)


Review request for drill, Chris Westin and Steven Phillips.


Summary (updated)
-

DRILL-1747: Intermediate query profile reporting


Repository: drill-git


Description
---

Update Workmanager to report status of fragments back to Foreman so that 
profiles show intermediate status.


Diffs
-

  exec/java-exec/src/main/java/org/apache/drill/exec/work/StatusProvider.java 
9215f43 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java 
5bc3da1 
  
exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/AbstractStatusReporter.java
 c7ac311 
  
exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentExecutor.java
 9ffe643 

Diff: https://reviews.apache.org/r/30404/diff/


Testing
---

Hand testing on long running queries to ensure query stats were updated during 
query.


Thanks,

Jacques Nadeau