[jira] [Commented] (DRILL-4919) Fix select count(1) / count(*) on csv with header

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820071#comment-15820071
 ] 

ASF GitHub Bot commented on DRILL-4919:
---

Github user gparai commented on a diff in the pull request:

https://github.com/apache/drill/pull/714#discussion_r95721099
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/text/compliant/CompliantTextRecordReader.java
 ---
@@ -91,6 +96,9 @@ public boolean apply(@Nullable SchemaPath path) {
 
   @Override
   protected List getDefaultColumnsToRead() {
+if (settings.isHeaderExtractionEnabled()) {
--- End diff --

Please add a comment describing the change - something similar to your 
comment 
> ... currently text files with headers do not support columns syntax, so 
for these type of files we need to replace empty list of columns with default 
named column when header extraction is enabled


> Fix select count(1) / count(*) on csv with header
> -
>
> Key: DRILL-4919
> URL: https://issues.apache.org/jira/browse/DRILL-4919
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.8.0
>Reporter: F Méthot
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: Future
>
>
> This happens since  1.8
> Dataset (I used extended char for display purpose) test.csvh:
> a,b,c,d\n
> 1,2,3,4\n
> 5,6,7,8\n
> Storage config:
> "csvh": {
>   "type": "text",
>   "extensions" : [
>   "csvh"
>],
>"extractHeader": true,
>"delimiter": ","
>   }
> select count(1) from dfs.`test.csvh`
> Error: UNSUPPORTED_OPERATION ERROR: With extractHeader enabled, only header 
> names are supported
> coumn name columns
> column index
> Fragment 0:0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5088) Error when reading DBRef column

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820028#comment-15820028
 ] 

ASF GitHub Bot commented on DRILL-5088:
---

Github user gparai commented on a diff in the pull request:

https://github.com/apache/drill/pull/702#discussion_r95719700
  
--- Diff: 
contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoGroupScan.java
 ---
@@ -503,7 +505,9 @@ public ScanStats getScanStats() {
   long numDocs = collection.count();
   float approxDiskCost = 0;
   if (numDocs != 0) {
-String json = collection.find().first().toJson();
+final DocumentCodec codec =
--- End diff --

Please add a comment to describe the need for the codec along with the 
example from the bug.


> Error when reading DBRef column
> ---
>
> Key: DRILL-5088
> URL: https://issues.apache.org/jira/browse/DRILL-5088
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
> Environment: drill 1.9.0
> mongo 3.2
>Reporter: Guillaume Champion
>Assignee: Chunhui Shi
>
> In a mongo database with DBRef, when a DBRef is inserted in the first line of 
> a mongo's collection drill query failed :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> {code}
> Simple example to reproduce:
> In mongo instance
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> [Error Id: 2944d766-e483-4453-a706-3d481397b186 on Analytics-Biznet:31010] 
> (state=,code=0)
> {code}
> If the first line doesn't contain de DBRef, drill will querying correctly :
> In a mongo instance :
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8939") });
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> +--+---+
> | _id  |account   
>  |
> +--+---+
> | {"$oid":"582081d96b69060001fd8939"}  | {"$id":{}}   
>  |
> | {"$oid":"582081d96b69060001fd8938"}  | 
> {"$ref":"contact","$id":{"$oid":"999cbf116b69060001fd8611"}}  |
> +--+---+
> 2 rows selected (0,563 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5172) Display elapsed time for queries in the UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819712#comment-15819712
 ] 

ASF GitHub Bot commented on DRILL-5172:
---

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/721
  
@arina-ielchiieva , @paul-rogers , @sudheeshkatkam 
Closed the previous PR due to mixup with squashing and rebasing. Please 
review this PR. 


> Display elapsed time for queries in the UI
> --
>
> Key: DRILL-5172
> URL: https://issues.apache.org/jira/browse/DRILL-5172
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, the Web UI does not display the runtime for a query either in the 
> list of queries or the query profile page itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5172) Display elapsed time for queries in the UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819706#comment-15819706
 ] 

ASF GitHub Bot commented on DRILL-5172:
---

GitHub user kkhatua opened a pull request:

https://github.com/apache/drill/pull/721

DRILL-5172: Display elapsed time for queries in the UI

Displays the elapsed time for running queries and the total duration of 
completed/failed/cancelled queries in the list of query profiles displayed, and 
within a query's profile page as well.
The query runtime is  displayed in `[hr] [min] sec`.
e.g. A duration of `25,254,321ms` is displayed  `7 hr 00 min 54.321 sec` 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kkhatua/drill DRILL-5172

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/721.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #721


commit a92f12668dbcd881468b2b128373866df8d115ae
Author: Kunal Khatua 
Date:   2017-01-12T00:45:15Z

DRILL-5172: Display elapsed time for queries in the UI

Displays the elapsed time for running queries and the total duration of 
completed/failed/cancelled queries in the list of query profiles displayed, and 
within a query's profile page as well.
The query runtime is  displayed in '[hr] [min] sec'.
e.g. A duration of 25,254,321ms is displayed  7 hr 00 min 54.321 sec




> Display elapsed time for queries in the UI
> --
>
> Key: DRILL-5172
> URL: https://issues.apache.org/jira/browse/DRILL-5172
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, the Web UI does not display the runtime for a query either in the 
> list of queries or the query profile page itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5172) Display elapsed time for queries in the UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819660#comment-15819660
 ] 

ASF GitHub Bot commented on DRILL-5172:
---

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/719
  
Closing the pull request. Attempted to rebase and squash, but got 
interleaved commits with master. Will send out a new pull request. 


> Display elapsed time for queries in the UI
> --
>
> Key: DRILL-5172
> URL: https://issues.apache.org/jira/browse/DRILL-5172
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, the Web UI does not display the runtime for a query either in the 
> list of queries or the query profile page itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5172) Display elapsed time for queries in the UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819659#comment-15819659
 ] 

ASF GitHub Bot commented on DRILL-5172:
---

Github user kkhatua closed the pull request at:

https://github.com/apache/drill/pull/719


> Display elapsed time for queries in the UI
> --
>
> Key: DRILL-5172
> URL: https://issues.apache.org/jira/browse/DRILL-5172
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, the Web UI does not display the runtime for a query either in the 
> list of queries or the query profile page itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5172) Display elapsed time for queries in the UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819464#comment-15819464
 ] 

ASF GitHub Bot commented on DRILL-5172:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95692761
  
--- Diff: exec/java-exec/src/main/resources/rest/profile/profile.ftl ---
@@ -106,6 +106,7 @@
   STATE: ${model.getProfile().getState().name()}
   FOREMAN: ${model.getProfile().getForeman().getAddress()}
   TOTAL FRAGMENTS: ${model.getProfile().getTotalFragments()}
+  DURATION: ${model.getProfileDuration()}
--- End diff --

The end time for a query is not set until the very end of the query, so the 
default value in 0L. For the `/profiles ` template, the elapsed time is 
computed using the current time in millis for running queries. Hence it shows 
correctly here. As per your suggestion, fixed it now by changing the signature 
to take the start and end times as arguments.  


> Display elapsed time for queries in the UI
> --
>
> Key: DRILL-5172
> URL: https://issues.apache.org/jira/browse/DRILL-5172
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, the Web UI does not display the runtime for a query either in the 
> list of queries or the query profile page itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4525) Query with BETWEEN clause on Date and Timestamp values fails with Validation Error

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819337#comment-15819337
 ] 

Kunal Khatua commented on DRILL-4525:
-

[~zfong] Can we assign a dev resource for this?

> Query with BETWEEN clause on Date and Timestamp values fails with Validation 
> Error
> --
>
> Key: DRILL-4525
> URL: https://issues.apache.org/jira/browse/DRILL-4525
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Abhishek Girish
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Fix For: 1.9.0
>
>
> Query: (simplified variant of TPC-DS Query37)
> {code}
> SELECT
>*
> FROM   
>date_dim
> WHERE   
>d_date BETWEEN Cast('1999-03-06' AS DATE) AND  (
>   Cast('1999-03-06' AS DATE) + INTERVAL '60' day)
> LIMIT 10;
> {code}
> Error:
> {code}
> Error: VALIDATION ERROR: From line 6, column 8 to line 7, column 64: Cannot 
> apply 'BETWEEN ASYMMETRIC' to arguments of type ' BETWEEN ASYMMETRIC 
>  AND '. Supported form(s): ' BETWEEN 
>  AND '
> SQL Query null
> [Error Id: 223fb37c-f561-4a37-9283-871dc6f4d6d0 on abhi2:31010] 
> (state=,code=0)
> {code}
> This is a regression from 1.6.0. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5192) REFRESH table METADATA as default system option

[bharath kumar (JIRA)]

bharath kumar created DRILL-5192:


 Summary: REFRESH table METADATA as default system option 
 Key: DRILL-5192
 URL: https://issues.apache.org/jira/browse/DRILL-5192
 Project: Apache Drill
  Issue Type: Improvement
  Components: Functions - Drill
Affects Versions: 1.8.0, 1.7.0, 1.6.0, 1.5.0, 1.4.0, 1.3.0, 1.2.0
 Environment: maprfs
Reporter: bharath kumar


Hi,

We have a batch use case where files are loaded daily , we run REFRESH METADATA 
TABLE; for more performance on these files. How can we automate this refresh 
metadata table; at drill configuration level for specific table.


Thanks,
Bharath



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5191) OutOfMemoryException - TPC-DS query4 on SF100

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818983#comment-15818983
 ] 

Khurram Faraaz commented on DRILL-5191:
---

TPC-DS query 11 also fails with OOM on Drill 1.10.0 (ee399317) SF100 data. 

{noformat}
2017-01-11 18:05:51,106 [BitServer-5] ERROR o.a.d.exec.rpc.RpcExceptionHandler 
- Exception in RPC communication.  Connection: /10.10.100.204:31012 <--> 
/10.10.100.203:52949 (data server).  Closing connection.
org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer 
of size 32768 due to memory limit. Current allocation: 4194304
at 
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) 
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191) 
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:49)
 ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.memory.DrillByteBufAllocator.ioBuffer(DrillByteBufAllocator.java:64)
 ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
 ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
 ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) 
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
 [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) 
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) 
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
 [netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
2017-01-11 18:05:51,112 [27898c35-dca7-a331-7ea9-61128319d4e9:frag:20:36] INFO  
o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes 
ran out of memory while executing the query. (Unable to allocate buffer of size 
262144 due to memory limit. Current allocation: 53379072)
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
nodes ran out of memory while executing the query.

Unable to allocate buffer of size 262144 due to memory limit. Current 
allocation: 53379072

[Error Id: e9ee45e8-9031-4366-a684-0fc4663951ac ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
 ~[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:242)
 [drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_65]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to 
allocate buffer of size 262144 due to memory limit. Current allocation: 53379072
at 
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) 
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191) 
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.IntVector.allocateBytes(IntVector.java:200) 
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.IntVector.allocateNew(IntVector.java:182) 
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.test.generated.HashTableGen6705.allocMetadataVector(HashTableTemplate.java:769)
 ~[na:na]
at 
org.apache.drill.exec.test.generated.HashTableGen6705.access$200(HashTableTemplate.java:42)
 ~[na:na]
at 
org.apache.drill.exec.test.generated.HashTableGen6705$BatchHolder.(HashTableTemplate.java:146)
 ~[na:na]
at 
org.apache.drill.exec.test.generated.HashTableGen6705.newBatchHolder(HashTableTemplate.java:697)
 ~[na:na]
at 
org.apache.drill.exec.test.generated.HashTableGen6705.addBatchHolder(HashTableTemplate.java:690)
 ~[na:na]
at 

[jira] [Commented] (DRILL-4525) Query with BETWEEN clause on Date and Timestamp values fails with Validation Error

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818903#comment-15818903
 ] 

Khurram Faraaz commented on DRILL-4525:
---

List of TPC-DS queries that FAIL on SF1 Drill 1.10.0, are query5, query12, 
query20, query21, query37, query40, query77, query80, query82, query98.

Fixing this issue will fix the above 10 TPC-DS queries.

> Query with BETWEEN clause on Date and Timestamp values fails with Validation 
> Error
> --
>
> Key: DRILL-4525
> URL: https://issues.apache.org/jira/browse/DRILL-4525
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Abhishek Girish
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Fix For: 1.9.0
>
>
> Query: (simplified variant of TPC-DS Query37)
> {code}
> SELECT
>*
> FROM   
>date_dim
> WHERE   
>d_date BETWEEN Cast('1999-03-06' AS DATE) AND  (
>   Cast('1999-03-06' AS DATE) + INTERVAL '60' day)
> LIMIT 10;
> {code}
> Error:
> {code}
> Error: VALIDATION ERROR: From line 6, column 8 to line 7, column 64: Cannot 
> apply 'BETWEEN ASYMMETRIC' to arguments of type ' BETWEEN ASYMMETRIC 
>  AND '. Supported form(s): ' BETWEEN 
>  AND '
> SQL Query null
> [Error Id: 223fb37c-f561-4a37-9283-871dc6f4d6d0 on abhi2:31010] 
> (state=,code=0)
> {code}
> This is a regression from 1.6.0. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5039) NPE - CTAS PARTITION BY ()

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818880#comment-15818880
 ] 

Khurram Faraaz commented on DRILL-5039:
---

Verified on Drill 1.10.0 (ee399317). Once tests are enabled, this JIRA will be 
marked as verified & closed.

{noformat}
0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> CREATE TABLE tprtnby_vrchrcl11 
PARTITION BY( col_vrchr2 ) AS SELECT * FROM typeall_l;
+---++
| Fragment  | Number of records written  |
+---++
| 0_0   | 105|
+---++
1 row selected (1.433 seconds)
0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> select count(*) from 
tprtnby_vrchrcl11;
+-+
| EXPR$0  |
+-+
| 105 |
+-+
1 row selected (0.258 seconds)
{noformat}

> NPE - CTAS PARTITION BY ()
> 
>
> Key: DRILL-5039
> URL: https://issues.apache.org/jira/browse/DRILL-5039
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> We are seeing an NPE, when CTAS is used with PARTITION BY 
> () and all columns are projected in SELECT of CTAS.
> Drill 1.9.0 
> git commit ID : db30854
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE dfs.tmp.temp_tbl
> . . . . . . . . . . . . . . > PARTITION BY (col_chr)
> . . . . . . . . . . . . . . > AS
> . . . . . . . . . . . . . . > SELECT col_int, col_chr, col_vrchr1, col_vrchr2 
> ,  col_dt, col_tim, col_tmstmp , col_flt, col_intrvl_yr , col_intrvl_day , 
> col_bln
> . . . . . . . . . . . . . . > FROM typeall_l;
> Error: SYSTEM ERROR: NullPointerException
> Fragment 0:0
> [Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> {noformat}
> 2016-11-12 19:54:14,901 [27d88c99-a64d-0317-ba3b-d78195cf85cc:frag:0:0] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment 0:0
> [Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> NullPointerException
> Fragment 0:0
> [Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
>  [drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
>  [drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.9.0.jar:1.9.0]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_91]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.lang.NullPointerException: null
> at 
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:100)
>  ~[vector-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.test.generated.ProjectorGen3.doEval(ProjectorTemplate.java:88)
>  ~[na:na]
> at 
> org.apache.drill.exec.test.generated.ProjectorGen3.projectRecords(ProjectorTemplate.java:62)
>  ~[na:na]
> at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:199)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:91)
>  

[jira] [Commented] (DRILL-5172) Display elapsed time for queries in the UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818240#comment-15818240
 ] 

ASF GitHub Bot commented on DRILL-5172:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95569784
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 ---
@@ -68,20 +68,35 @@
   @Inject DrillUserPrincipal principal;
   @Inject SecurityContext sc;
 
+  public static String getPrettyDuration(long durationInMillis) {
+long hours = TimeUnit.MILLISECONDS.toHours(durationInMillis);
--- End diff --

Please add description with output example.


> Display elapsed time for queries in the UI
> --
>
> Key: DRILL-5172
> URL: https://issues.apache.org/jira/browse/DRILL-5172
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, the Web UI does not display the runtime for a query either in the 
> list of queries or the query profile page itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5172) Display elapsed time for queries in the UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818237#comment-15818237
 ] 

ASF GitHub Bot commented on DRILL-5172:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95570474
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 ---
@@ -105,6 +120,17 @@ public String getTime() {
   return format.format(time);
 }
 
+public long getStartTime() {
+  return startTime;
+}
+
+public long getEndTime() {
+  return endTime;
+}
+
+public String getDuration() {
+  return getPrettyDuration(endTime - startTime);
+}
--- End diff --

May be we can factor out calculation? Please see comment above.


> Display elapsed time for queries in the UI
> --
>
> Key: DRILL-5172
> URL: https://issues.apache.org/jira/browse/DRILL-5172
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, the Web UI does not display the runtime for a query either in the 
> list of queries or the query profile page itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5172) Display elapsed time for queries in the UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818236#comment-15818236
 ] 

ASF GitHub Bot commented on DRILL-5172:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95570493
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileWrapper.java
 ---
@@ -114,6 +114,10 @@ public QueryProfile getProfile() {
 return profile;
   }
 
+  public String getProfileDuration() {
+return 
ProfileResources.getPrettyDuration(profile.getEnd()-profile.getStart());
+  }
--- End diff --

May be we can factor out calculation? Please see comment above.


> Display elapsed time for queries in the UI
> --
>
> Key: DRILL-5172
> URL: https://issues.apache.org/jira/browse/DRILL-5172
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, the Web UI does not display the runtime for a query either in the 
> list of queries or the query profile page itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5172) Display elapsed time for queries in the UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818238#comment-15818238
 ] 

ASF GitHub Bot commented on DRILL-5172:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95571159
  
--- Diff: exec/java-exec/src/main/resources/rest/profile/profile.ftl ---
@@ -106,6 +106,7 @@
   STATE: ${model.getProfile().getState().name()}
   FOREMAN: ${model.getProfile().getForeman().getAddress()}
   TOTAL FRAGMENTS: ${model.getProfile().getTotalFragments()}
+  DURATION: ${model.getProfileDuration()}
--- End diff --

For running queries, I see incorrect duration calculation. On `/profiles` 
elapsed time is shown correctly but when you step into running profile, 
duration is shown incorrectly.

![image](https://cloud.githubusercontent.com/assets/15086720/21849179/259dc0ae-d80d-11e6-990d-cbb9e2d072ad.png)

![image](https://cloud.githubusercontent.com/assets/15086720/21849198/3c26d9f0-d80d-11e6-873d-8bd439cc4351.png)




> Display elapsed time for queries in the UI
> --
>
> Key: DRILL-5172
> URL: https://issues.apache.org/jira/browse/DRILL-5172
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, the Web UI does not display the runtime for a query either in the 
> list of queries or the query profile page itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5172) Display elapsed time for queries in the UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818239#comment-15818239
 ] 

ASF GitHub Bot commented on DRILL-5172:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/719#discussion_r95570379
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
 ---
@@ -68,20 +68,35 @@
   @Inject DrillUserPrincipal principal;
   @Inject SecurityContext sc;
 
+  public static String getPrettyDuration(long durationInMillis) {
+long hours = TimeUnit.MILLISECONDS.toHours(durationInMillis);
--- End diff --

May be this utility method can accept two parameters: `long start, long 
end`, so you won't have to repeat the following logic 
`profile.getEnd()-profile.getStart()` or `endTime - startTime`?


> Display elapsed time for queries in the UI
> --
>
> Key: DRILL-5172
> URL: https://issues.apache.org/jira/browse/DRILL-5172
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, the Web UI does not display the runtime for a query either in the 
> list of queries or the query profile page itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5191) OutOfMemoryException - TPC-DS query4 on SF100

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz updated DRILL-5191:
--
Summary: OutOfMemoryException - TPC-DS query4 on SF100  (was: 
OutOfMemoryException - TPCDS query4 )

> OutOfMemoryException - TPC-DS query4 on SF100
> -
>
> Key: DRILL-5191
> URL: https://issues.apache.org/jira/browse/DRILL-5191
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.10.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Priority: Critical
> Attachments: 2789eba3-60f0-0b2f-eba8-82331735d5c4.sys.drill
>
>
> TPC-DS Query4 against SF100 on Drill 1.10.0 (ee399317), on a 4 node CentOS 
> cluster
> Query4 => https://raw.githubusercontent.com/Agirish/tpcds/master/query4.sql
> total number of fragments : 1,125
> Stack trace from drillbit.log
> {noformat}
> 2017-01-11 11:17:57,007 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2017-01-11 11:17:57,008 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State to report: RUNNING
> 2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested RUNNING --> 
> FAILED
> 2017-01-11 11:17:57,009 [BitServer-6] ERROR 
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  
> Connection: /10.10.100.202:31012 <--> /10.10.100.201:44712 (data server).  
> Closing connection.
> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate 
> buffer of size 16384 due to memory limit. Current allocation: 16777216
> at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) 
> ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191) 
> ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:49)
>  ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.memory.DrillByteBufAllocator.ioBuffer(DrillByteBufAllocator.java:64)
>  ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
>  ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
>  ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) 
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>  [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) 
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) 
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>  [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> 2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested 
> AWAITING_ALLOCATION --> FAILED
> 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: State to report: RUNNING
> 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested FAILED --> 
> FINISHED
> 2017-01-11 11:17:57,010 [BitServer-6] INFO  
> o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding 
> remaining 3240924 byte(s) in buffer.
> 2017-01-11 11:17:57,011 [BitServer-10] ERROR 
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  
> Connection: 

[jira] [Updated] (DRILL-5191) OutOfMemoryException - TPCDS query4

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz updated DRILL-5191:
--
Attachment: 2789eba3-60f0-0b2f-eba8-82331735d5c4.sys.drill

query profile attached

> OutOfMemoryException - TPCDS query4 
> 
>
> Key: DRILL-5191
> URL: https://issues.apache.org/jira/browse/DRILL-5191
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.10.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Priority: Critical
> Attachments: 2789eba3-60f0-0b2f-eba8-82331735d5c4.sys.drill
>
>
> TPC-DS Query4 against SF100 on Drill 1.10.0 (ee399317), on a 4 node CentOS 
> cluster
> Query4 => https://raw.githubusercontent.com/Agirish/tpcds/master/query4.sql
> total number of fragments : 1,125
> Stack trace from drillbit.log
> {noformat}
> 2017-01-11 11:17:57,007 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2017-01-11 11:17:57,008 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State to report: RUNNING
> 2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested RUNNING --> 
> FAILED
> 2017-01-11 11:17:57,009 [BitServer-6] ERROR 
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  
> Connection: /10.10.100.202:31012 <--> /10.10.100.201:44712 (data server).  
> Closing connection.
> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate 
> buffer of size 16384 due to memory limit. Current allocation: 16777216
> at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) 
> ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191) 
> ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:49)
>  ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.memory.DrillByteBufAllocator.ioBuffer(DrillByteBufAllocator.java:64)
>  ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
>  ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
>  ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) 
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>  [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) 
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) 
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>  [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> 2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested 
> AWAITING_ALLOCATION --> FAILED
> 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: State to report: RUNNING
> 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested FAILED --> 
> FINISHED
> 2017-01-11 11:17:57,010 [BitServer-6] INFO  
> o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding 
> remaining 3240924 byte(s) in buffer.
> 2017-01-11 11:17:57,011 [BitServer-10] ERROR 
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  
> Connection: /10.10.100.202:31012 <--> /10.10.100.202:52127 

[jira] [Created] (DRILL-5191) OutOfMemoryException - TPCDS query4

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5191:
-

 Summary: OutOfMemoryException - TPCDS query4 
 Key: DRILL-5191
 URL: https://issues.apache.org/jira/browse/DRILL-5191
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.10.0
 Environment: 4 node cluster CentOS
Reporter: Khurram Faraaz
Priority: Critical


TPC-DS Query4 against SF100 on Drill 1.10.0 (ee399317), on a 4 node CentOS 
cluster

Query4 => https://raw.githubusercontent.com/Agirish/tpcds/master/query4.sql

total number of fragments : 1,125

Stack trace from drillbit.log

{noformat}
2017-01-11 11:17:57,007 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 
2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested 
AWAITING_ALLOCATION --> RUNNING
2017-01-11 11:17:57,008 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: 
State to report: RUNNING
2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 
2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested RUNNING --> 
FAILED
2017-01-11 11:17:57,009 [BitServer-6] ERROR o.a.d.exec.rpc.RpcExceptionHandler 
- Exception in RPC communication.  Connection: /10.10.100.202:31012 <--> 
/10.10.100.201:44712 (data server).  Closing connection.
org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer 
of size 16384 due to memory limit. Current allocation: 16777216
at 
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) 
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191) 
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:49)
 ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.memory.DrillByteBufAllocator.ioBuffer(DrillByteBufAllocator.java:64)
 ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
 ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
 ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) 
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
 [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) 
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) 
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
 [netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 
2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested 
AWAITING_ALLOCATION --> FAILED
2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 
2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: State change requested 
AWAITING_ALLOCATION --> RUNNING
2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: 
State to report: RUNNING
2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 
2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested FAILED --> 
FINISHED
2017-01-11 11:17:57,010 [BitServer-6] INFO  
o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding remaining 
3240924 byte(s) in buffer.
2017-01-11 11:17:57,011 [BitServer-10] ERROR o.a.d.exec.rpc.RpcExceptionHandler 
- Exception in RPC communication.  Connection: /10.10.100.202:31012 <--> 
/10.10.100.202:52127 (data server).  Closing connection.
org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer 
of size 4096 due to memory limit. Current allocation: 16777216
at 
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) 
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at 
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191) 
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
  

[jira] [Commented] (DRILL-3562) Query fails when using flatten on JSON data where some documents have an empty array

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818118#comment-15818118
 ] 

Serhii Harnyk commented on DRILL-3562:
--

Besides the initialization of empty arrays we have the problem with ordering of 
columns with arrays. 
Query 
{code}
select * from example 
{code}
for Json
{noformat}
{ "a": [], "c": [], "c1": 1 }
{ "a": [1], "c": [1], "c1": 1 }
{noformat}
returns result
{noformat}
---
| c1| a | c |
---
| 1   | []  | []  |
| 1   | [1] | [1] |
---
{noformat}
with wrong columns order.

> Query fails when using flatten on JSON data where some documents have an 
> empty array
> 
>
> Key: DRILL-3562
> URL: https://issues.apache.org/jira/browse/DRILL-3562
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JSON
>Affects Versions: 1.1.0
>Reporter: Philip Deegan
>Assignee: Serhii Harnyk
> Fix For: Future
>
>
> Drill query fails when using flatten when some records contain an empty array 
> {noformat}
> SELECT COUNT(*) FROM (SELECT FLATTEN(t.a.b.c) AS c FROM dfs.`flat.json` t) 
> flat WHERE flat.c.d.e = 'f' limit 1;
> {noformat}
> Succeeds on 
> { "a": { "b": { "c": [  { "d": {  "e": "f" } } ] } } }
> Fails on
> { "a": { "b": { "c": [] } } }
> Error
> {noformat}
> Error: SYSTEM ERROR: ClassCastException: Cannot cast 
> org.apache.drill.exec.vector.NullableIntVector to 
> org.apache.drill.exec.vector.complex.RepeatedValueVector
> {noformat}
> Is it possible to ignore the empty arrays, or do they need to be populated 
> with dummy data?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4469) SUM window query returns incorrect results over integer data

[Nitin Pawar (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818022#comment-15818022
 ] 

Nitin Pawar commented on DRILL-4469:


[~khfaraaz] would it be a good idea to disable this feature if it is giving 
wrong results? 


> SUM window query returns incorrect results over integer data
> 
>
> Key: DRILL-4469
> URL: https://issues.apache.org/jira/browse/DRILL-4469
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.6.0
> Environment: 4 node CentOS cluster
>Reporter: Khurram Faraaz
>Priority: Critical
>  Labels: window_function
> Attachments: t_alltype.csv, t_alltype.parquet
>
>
> SUM window query returns incorrect results as compared to Postgres, with or 
> without the frame clause in the window definition. Note that there is a sub 
> query involved and data in column c1 is sorted integer data with no nulls.
> Drill 1.6.0 commit ID: 6d5f4983
> Results from Drill 1.6.0
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT SUM(c1) OVER w FROM (select * from 
> dfs.tmp.`t_alltype`) subQry WINDOW w AS (PARTITION BY c8 ORDER BY c1 RANGE 
> BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING);
> +-+
> | EXPR$0  |
> +-+
> | 10585   |
> | 10585   |
> | 10585   |
> | 10585   |
> | 10585   |
> | 10585   |
> ...
> | 10585  |
> | 10585  |
> | 10585  |
> ++
> 145 rows selected (0.257 seconds)
> {noformat}
> results from Postgres 9.3
> {noformat}
> postgres=# SELECT SUM(c1) OVER w FROM (select * from t_alltype) subQry WINDOW 
> w AS (PARTITION BY c8 ORDER BY c1 RANGE BETWEEN UNBOUNDED PRECEDING AND 
> UNBOUNDED FOLLOWING);
>  sum
> --
>  4499
>  4499
>  4499
>  4499
>  4499
>  4499
> ...
>  5613
>  5613
>  5613
>   473
>   473
>   473
>   473
>   473
> (145 rows)
> {noformat}
> Removing the frame clause from window definition, still results in completely 
> different results on Postgres vs Drill
> Results from Drill 1.6.0
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp>SELECT SUM(c1) OVER w FROM (select * from 
> t_alltype) subQry WINDOW w AS (PARTITION BY c8 ORDER BY c1);
> +-+
> | EXPR$0  |
> +-+
> | 10585   |
> | 10585   |
> | 10585   |
> | 10585   |
> | 10585   |
> | 10585   |
> | 10585   |
> | 10585   |
> | 10585   |
> ...
> | 10585  |
> | 10585  |
> | 10585  |
> | 10585  |
> | 10585  |
> ++
> 145 rows selected (0.28 seconds)
> {noformat}
> Results from Postgres
> {noformat}
> postgres=# SELECT SUM(c1) OVER w FROM (select * from t_alltype) subQry WINDOW 
> w AS (PARTITION BY c8 ORDER BY c1);
>  sum
> --
> 5
>12
>21
>33
>47
>62
>78
>96
>   115
>   135
>   158
>   182
>   207
>   233
>   260
>   289
> ...
> 4914
>  5051
>  5189
>  5328
>  5470
>  5613
> 8
>70
>   198
>   332
>   473
> (145 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4578) "children" missing from results of full scan over JSON data

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk reassigned DRILL-4578:


Assignee: Serhii Harnyk

> "children" missing from results of full scan over JSON data
> ---
>
> Key: DRILL-4578
> URL: https://issues.apache.org/jira/browse/DRILL-4578
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JSON
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>Priority: Critical
>
> One of the fields named "children" is missing from the output of SELECT * 
> over the JSON data, with or without enabling all_text_mode for JSON data.
> Projecting just the "children" field returns a null.
> Note that children field holds an empty array.
>  Drill 1.7.0-SNAPSHOT  commit ID e7e9b73c
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> alter session set 
> `store.json.all_text_mode`=true;
> +---++
> |  ok   |  summary   |
> +---++
> | true  | store.json.all_text_mode updated.  |
> +---++
> 1 row selected (0.118 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select * from `employee.json`;
> ++---+--+--++-+--+---+
> | firstName  | lastName  | isAlive  | age  | height_cm  | 
>   address   | 
> phoneNumbers  
>|hobbies|
> ++---+--+--++-+--+---+
> | John   | Smith | true | 45   | 177.6  | 
> {"streetAddress":"29 4th Street","city":"New 
> York","state":"NY","postalCode":"10021-3100"}  | 
> [{"type":"home","number":"212 555-1234"},{"type":"office","number":"646 
> 555-4567"}]  | ["scuba diving","hiking","biking","rock climbing","surfing"]  |
> ++---+--+--++-+--+---+
> 1 row selected (0.214 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select t.children from `employee.json` t;
> +---+
> | children  |
> +---+
> | null  |
> +---+
> 1 row selected (0.148 seconds)
> {noformat}
> JSON data used in test
> {noformat}
> [root@centos-01 ~]# cat employee.json
> {
>   "firstName": "John",
>   "lastName": "Smith",
>   "isAlive": true,
>   "age": 45,
>   "height_cm": 177.6,
>   "address": {
> "streetAddress": "29 4th Street",
> "city": "New York",
> "state": "NY",
> "postalCode": "10021-3100"
>   },
>   "phoneNumbers": [
> {
>   "type": "home",
>   "number": "212 555-1234"
> },
> {
>   "type": "office",
>   "number": "646 555-4567"
> }
>   ],
>   "children": [],
>   "hobbies": ["scuba diving","hiking","biking","rock climbing","surfing"]
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)