[GitHub] drill pull request #584: DRILL-4884: Fix bug that drill sometimes produced I...

2016-09-09 Thread zbdzzg 
GitHub user zbdzzg opened a pull request:

https://github.com/apache/drill/pull/584

DRILL-4884: Fix bug that drill sometimes produced IOB exception while 
querying data of 65536 limitation

Drill produces IOB while using a non batched scanner and limiting SQL by 
65536.

SQL:

```
select id from isearch.tmall_auction_cluster limit 1 offset 65535
```

Result:

```
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
 ~[classes/:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:324)
 [classes/:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
 [classes/:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
 [classes/:na]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[classes/:na]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_101]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.lang.IndexOutOfBoundsException: index: 131072, length: 2 
(expected: range(0, 131072))
at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:175) 
~[classes/:4.0.27.Final]
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:197) 
~[classes/:4.0.27.Final]
at io.netty.buffer.DrillBuf.setChar(DrillBuf.java:517) 
~[classes/:4.0.27.Final]
at 
org.apache.drill.exec.record.selection.SelectionVector2.setIndex(SelectionVector2.java:79)
 ~[classes/:na]
at 
org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.limitWithNoSV(LimitRecordBatch.java:167)
 ~[classes/:na]
at 
org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.doWork(LimitRecordBatch.java:145)
 ~[classes/:na]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
 ~[classes/:na]
at 
org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:115)
 ~[classes/:na]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
 ~[classes/:na]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
 ~[classes/:na]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[classes/:na]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 ~[classes/:na]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 ~[classes/:na]
at 
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:94)
 ~[classes/:na]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
 ~[classes/:na]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
 ~[classes/:na]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[classes/:na]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 ~[classes/:na]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 ~[classes/:na]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
 ~[classes/:na]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
 ~[classes/:na]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
 ~[classes/:na]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
~[classes/:na]
at 
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
 ~[classes/:na]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
~[classes/:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:256)
 ~[classes/:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:250)
 ~[classes/:na]
at java.security.AccessController.doPrivileged(Native Method) 
~[na:1.8.0_101]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_101]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
 ~[hadoop-common-2.7.1.jar:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250)
 [classes/:na]
... 4 common

[GitHub] drill issue #584: DRILL-4884: Fix bug that drill sometimes produced IOB exce...

2016-09-18 Thread zbdzzg 
Github user zbdzzg commented on the issue:

https://github.com/apache/drill/pull/584
  
 @dsbos 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #584: DRILL-4884: Fix bug that drill sometimes produced I...

2016-10-24 Thread zbdzzg 
Github user zbdzzg commented on a diff in the pull request:

https://github.com/apache/drill/pull/584#discussion_r84828174
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/validate/IteratorValidatorBatchIterator.java
 ---
@@ -301,7 +301,7 @@ public IterOutcome next() {
   "Incoming batch [#%d, %s] has an empty schema. This is 
not allowed.",
   instNum, batchTypeName));
 }
-if (incoming.getRecordCount() > MAX_BATCH_SIZE) {
+if (incoming.getRecordCount() >= MAX_BATCH_SIZE) {
--- End diff --

Thanks for reply.

I tried turn assertion off and rerun, the IOB is produced again. The 
IllegalStateException is thrown when turning it on.

The built-in storage plug-ins always have smaller batch limitation (smaller 
than 65536) in reader implementations, so this problem is supposed to be 
avoided by design. The scene to us running into this problem is that we have a 
customized storage plug-in with a large batch-size set in the reader part. I 
personally guess that the assertion is in technically enabled in testing / 
debugging for a new written storage plug-in, so the IllegalStateException 
should be thrown in which the assertion is on, then we know that our batch-size 
is too large to execute all stuffs. As a comparison, a wired IOB exception 
makes us confused. 

If we consider the scene disabling assertion, another bound check should be 
inserted to LimitRecordBatch.java, but what I am seeing is that most check 
logics is written at IteratorValidator, I suppose that author wants storage 
developers make all things work in the early debugging, the assertion can be 
off after the exceptions like this IOB are resolved.

So I don t think that making any changes to LimitRecordBatch.java is 
reasonable rather than Doing same thing to validator part. This change benefits 
the plugin development. Do you agree?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #584: DRILL-4884: Fix bug that drill sometimes produced I...

2016-10-24 Thread zbdzzg 
Github user zbdzzg commented on a diff in the pull request:

https://github.com/apache/drill/pull/584#discussion_r84835453
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/validate/IteratorValidatorBatchIterator.java
 ---
@@ -301,7 +301,7 @@ public IterOutcome next() {
   "Incoming batch [#%d, %s] has an empty schema. This is 
not allowed.",
   instNum, batchTypeName));
 }
-if (incoming.getRecordCount() > MAX_BATCH_SIZE) {
+if (incoming.getRecordCount() >= MAX_BATCH_SIZE) {
--- End diff --

Pretty clear, see commit ccab954.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #584: DRILL-4884: Fix bug that drill sometimes produced IOB exce...

2016-10-26 Thread zbdzzg 
Github user zbdzzg commented on the issue:

https://github.com/apache/drill/pull/584
  
@jinfengni A test case has been added, See 
2c67dd78e463514393fd33779b64fa88c7c22ebd


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #658: DRILL-5051: Fix incorrect result returned in nest q...

2016-11-18 Thread zbdzzg 
GitHub user zbdzzg opened a pull request:

https://github.com/apache/drill/pull/658

DRILL-5051: Fix incorrect result returned in nest query with offset 
specified

1. Fix this problem.
2. Merge two limit methods, most of code is the same.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zbdzzg/drill master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/658.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #658


commit 982b8c2bfa742feffb551ba5bfb66ba01b38d839
Author: hongze.zhz 
Date:   2016-11-18T12:11:38Z

DRILL-5051: Fix incorrect result returned in nest query with offset 
specified




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #658: DRILL-5051: Fix incorrect result returned in nest q...

2016-11-21 Thread zbdzzg 
Github user zbdzzg closed the pull request at:

https://github.com/apache/drill/pull/658


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #662: DRILL-5051: Fix incorrect result returned in nest q...

2016-11-21 Thread zbdzzg 
GitHub user zbdzzg opened a pull request:

https://github.com/apache/drill/pull/662

DRILL-5051: Fix incorrect result returned in nest query with offset s…

…pecified

1. Fix this problem.
2. Merge two limit methods, most of code is the same.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zbdzzg/drill 5051

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/662.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #662


commit 102ad402250c0d74c9ab7a724aef3262b92069d8
Author: hongze.zhz 
Date:   2016-11-18T12:11:38Z

DRILL-5051: Fix incorrect result returned in nest query with offset 
specified




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #663: DRILL-5054: Provide source files of jquery-ui and j...

2016-11-21 Thread zbdzzg 
GitHub user zbdzzg opened a pull request:

https://github.com/apache/drill/pull/663

DRILL-5054: Provide source files of jquery-ui and jquery-dataTables locally

Hi,

Currently Drill uses CDN for serving source files of jquery-ui and 
jquery-dataTables. This is OK for most cases, but not working in an isolated 
environment.

This is a patch adding these files so that Drill will work fine in intranet.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zbdzzg/drill 5054

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/663.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #663


commit 55701b7494cf4b505f41a78179a4214090365bdf
Author: hongze.zhz 
Date:   2016-11-21T12:42:26Z

DRILL-5054: Provide source files of jquery-ui and jquery-dataTables locally




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #663: DRILL-5054: Provide source files of jquery-ui and j...

2016-11-23 Thread zbdzzg 
Github user zbdzzg commented on a diff in the pull request:

https://github.com/apache/drill/pull/663#discussion_r89448653
  
--- Diff: exec/java-exec/src/main/resources/rest/query/result.ftl ---
@@ -11,14 +11,13 @@
 
 <#include "*/generic.ftl">
 <#macro page_head>
-  
-  
+  
--- End diff --

Hi Parthchandra,

I have reconsidered this.
 
Simply putting these things to source code is indeed not a graceful 
solution, even this method is just temporarily working in my case.

I agree with your second advice: to make these stuffs downloadable and 
configurable. But as the typical scene for now is that users can already have 
them in cache, so may be just customizing and recompiling the code for special 
requirements like this is more reasonable for users like me (the scripts are 
not fetchable from my area, the network hangs while accessing cdn, util click 
"stop" actively. To have both the cdn and local files is therefore not work for 
me), I am so going to shelve this issue util someone report the same problem. 
Thank you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #668: DRILL-5068: Add a new system table for completed pr...

2016-11-24 Thread zbdzzg 
GitHub user zbdzzg opened a pull request:

https://github.com/apache/drill/pull/668

DRILL-5068: Add a new system table for completed profiles

Add table "sys.profiles" for completed queries.

Following fields added:

1. queryID (String)
2. time (Timestamp)
3. latency (long)
4. user (String)
5. query (String)
6. state (String)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zbdzzg/drill profile_query

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/668.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #668


commit 36b4cbfe8c31f789d225c23e13036c6f661d4895
Author: hongze.zhz 
Date:   2016-11-24T13:19:30Z

DRILL-5068: Add a new system table for completed profiles




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #663: DRILL-5054: Provide source files of jquery-ui and j...

2016-11-24 Thread zbdzzg 
Github user zbdzzg closed the pull request at:

https://github.com/apache/drill/pull/663


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #662: DRILL-5051: Fix incorrect result returned in nest query wi...

2016-12-13 Thread zbdzzg 
Github user zbdzzg commented on the issue:

https://github.com/apache/drill/pull/662
  
@sudheeshkatkam Could you please review this? Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #662: DRILL-5051: Fix incorrect result returned in nest query wi...

2016-12-14 Thread zbdzzg 
Github user zbdzzg commented on the issue:

https://github.com/apache/drill/pull/662
  
@sudheeshkatkam Thanks for your review, commit message has been changed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #662: DRILL-5051: Fix incorrect result returned in nest q...

2016-12-19 Thread zbdzzg 
Github user zbdzzg commented on a diff in the pull request:

https://github.com/apache/drill/pull/662#discussion_r93163164
  
--- Diff: exec/java-exec/src/test/java/org/apache/drill/TestBugFixes.java 
---
@@ -221,4 +221,14 @@ public void testDRILL4884() throws Exception {
 .baselineRecords(baseline)
 .go();
   }
+
+  @Test
+  public void testDRILL5051() throws Exception {
+testBuilder()
+.sqlQuery("select count(1) as cnt from (select l_orderkey from 
(select id from cp.`tpch/lineitem.parquet` limit 2) limit 1 offset 1)")
--- End diff --

Thank you for the catch!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #668: DRILL-5068: Add a new system table for completed pr...

2016-12-21 Thread zbdzzg 
Github user zbdzzg closed the pull request at:

https://github.com/apache/drill/pull/668


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #668: DRILL-5068: Add a new system table for completed pr...

2016-12-21 Thread zbdzzg 
GitHub user zbdzzg reopened a pull request:

https://github.com/apache/drill/pull/668

DRILL-5068: Add a new system table for completed profiles

Add table "sys.profiles" for completed queries.

Following fields added:

1. queryID (String)
2. time (Timestamp)
3. latency (long)
4. user (String)
5. query (String)
6. state (String)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zbdzzg/drill profile_query

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/668.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #668


commit e05a999dc8ace315966cbbdb72b3e52d3d956bbd
Author: hongze.zhz 
Date:   2016-12-22T07:47:42Z

DRILL-5068: Add a new system table for completed profiles




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---