[GitHub] drill pull request #719: DRILL-5172: Display elapsed time for queries in the...

[kkhatua]

GitHub user kkhatua opened a pull request:

https://github.com/apache/drill/pull/719

DRILL-5172: Display elapsed time for queries in the UI

Displays the elapsed time for running queries and the total duration of
completed/failed/cancelled queries in the list of query profiles
displayed, and within a query's profile page as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kkhatua/drill master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/719.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #719


commit 48ef739fffa9ba4f2b05412d7b846b48161fdedc
Author: Kunal Khatua 
Date:   2017-01-04T07:24:36Z

DRILL-5172: Display elapsed time for queries in the UI

Displays the elapsed time for running queries and the total duration of
completed/failed/cancelled queries in the list of query profiles
displayed, and within a query's profile page as well.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #718: DRILL-5127: Revert the fix for DRILL-4831

[ppadma]

GitHub user ppadma opened a pull request:

https://github.com/apache/drill/pull/718

DRILL-5127: Revert the fix for DRILL-4831

Please see DRILL-5127 for details.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ppadma/drill DRILL-5127

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/718.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #718


commit e24a269bcfeee3e9d078a91b56aa36fd321d46be
Author: Padma Penumarthy 
Date:   2017-01-03T22:01:00Z

DRILL-5127: Revert the fix for DRILL-4831




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #706: DRILL-5039: NPE - CTAS PARTITION BY (

[chunhui-shi]

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/706#discussion_r94492239
  
--- Diff: exec/java-exec/src/main/codegen/templates/NewValueFunctions.java 
---
@@ -17,6 +17,12 @@
  */
 <@pp.dropOutputFile />
 
+<#macro reassignHolder>
+previous.buffer = buf.reallocIfNeeded(length);
+previous.buffer.setBytes(0, in.buffer, in.start, in.end - 
in.start);
--- End diff --

length=(in.end - in.start) here. I would suggest always use length.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #715: DRILL-5105: remove buffer size checking in getBuffe...

[chunhui-shi]

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/715#discussion_r94471881
  
--- Diff: 
exec/vector/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java 
---
@@ -134,10 +134,6 @@ public int getBufferSizeFor(final int valueCount) {
 
   @Override
   public DrillBuf[] getBuffers(boolean clear) {
--- End diff --

Thanks for suggesting to use assert. It is a nice way to avoid doing this 
check in production. My thoughts is we don't need any check here, the reasons 
are:

1, mapVector does not have its own 'real' data vectors but just go to 
underlying vectors and get a sum of buffer sizes.
2, for non-mapVectors, there may be multiple vectors(offsets, bits, etc, 
such as ListVector) or the getBufferSize() is just simply get writeIndex, for 
which super.getBufferSize() is identical to this.getBufferSize(). And if there 
is any issues that non-mapVector did not calculate bufferSize correctly, we 
should have already seen in using that specific vector, thus we don't need to 
do it in mapVector code.

Actually my first thought was rewrite the logic about how to recursively 
doing bufferSize check. But after I read all getBufferSize on all vectors, I 
decided the check is not needed at all. Since any error in getBufferSize should 
immediately cause serious problem(serialize and deserialize) and can not pass 
the functional tests of that specific vector.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #709: DRILL-5156: BootStrapContext should close threads

[sudheeshkatkam]

Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/709
  
This change may not be that simple, but I could be wrong. I tried to do 
something similar as part of [PR 
429](https://github.com/apache/drill/pull/429/commits/0394f4ca5aed142bb2ba0b192f3588cfda7b).

The close happens elsewhere. The "loop" is actually closed as part of 

[BasicServer#close](https://github.com/apache/drill/blob/master/exec/rpc/src/main/java/org/apache/drill/exec/rpc/BasicServer.java#L218).
 But it will be closed multiple times (because there may be multiple instances 
of sub-classes of BasicServer, and all use the same loop), and looks like 
"loop2" is not closed anywhere. The changes in PR 429 close the loops exactly 
once, but I do not recollect why the PR is not merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #705: DRILL-5159: Make sure Drill's ProjectMergeRule operate on ...

[amansinha100]

Github user amansinha100 commented on the issue:

https://github.com/apache/drill/pull/705
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: IndexR, a new storage plugin for Drill

[Chunhui Shi]

Congratulations to IndexR team! What is your plan to add indexR storage
plugin to Drill? Cannot wait to try it out with Drill.



On Tue, Jan 3, 2017 at 9:02 AM, Jinfeng Ni  wrote:

> Forward to drill dev list. People on dev list might be interested in
> this as well.
>
>
> On Tue, Jan 3, 2017 at 8:22 AM, Jinfeng Ni  wrote:
> > Looks like IndexR is very interesting storage plugin. Although I have
> > not looked into the detail, I'm looking forward to seeing the PR and
> > hopefully getting this into Drill!
> >
> > Thanks,
> >
> > Jinfeng
> >
> >
> > On Tue, Jan 3, 2017 at 7:30 AM, WeiWan  wrote:
> >> Hi Charles,
> >>
> >> It would be great if IndexR plugin can be merged into official Drill
> project. I will do some more tests based on latest Drill version and submit
> a PR.
> >>
> >> Regards
> >> Flow Wei
> >>
> >>
> >>
> >>> On Jan 3, 2017, at 23:18, Charles Givre  wrote:
> >>>
> >>> This sounds really interesting.  Will you be submitting a PR to
> integrate this into the main Drill codebase?
> >>> — C
> >>>
>  On Jan 3, 2017, at 03:35, WeiWan  wrote:
> 
>  IndexR is a distributed, columnar storage system based on HDFS, which
> focus on fast analyse, both for massive static(historical) data and rapidly
> ingesting realtime data. IndexR is designed for OLAP.
> 
>  Fast analyze on large dataset
>  Realtime ingestion with zero delay for query
>  Deep integration with Hadoop ecosystem
>  Hardware efficiency
>  Highly avaliable, scalable, manageable and simple
>  Adapted with popular query engines like Apache Drill, Apache Hive,
> etc.
> 
>  And now it is open source.
> 
>  Project: https://github.com/shunfei/indexr <
> https://github.com/shunfei/indexr>
>  Wiki: https://github.com/shunfei/indexr/wiki <
> https://github.com/shunfei/indexr/wiki>
> 
>  IndexR is original developed by Sunteng Tech. This project started a
> year ago and now has been deployed to several productions in our company.
> The whole cluster consumes over 30 billions events each day in realtime
> from Kafka. The largest table contains over 10 billions rows (after rollup)
> and rapidly increasing. Most of the statistic/analyze queries’ latency is
> less than 3 seconds in real world production environment.
> 
>  Currently it is mainly used as Drill and Hive storage plugin. It
> should be quite easy to master.
> 
>  We hope IndexR be a favor to you and make it better.
> 
>  Regards
>  Flow Wei
> 
> 
> 
> >>>
> >>
>

[GitHub] drill pull request #705: DRILL-5159: Make sure Drill's ProjectMergeRule oper...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/705#discussion_r94443444
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillMergeProjectRule.java
 ---
@@ -48,6 +50,12 @@ public boolean matches(RelOptRuleCall call) {
 Project topProject = call.rel(0);
 Project bottomProject = call.rel(1);
 
+// Make sure both projects be LogicalProject.
+if (topProject.getTraitSet().getTrait(ConventionTraitDef.INSTANCE) != 
Convention.NONE ||
--- End diff --

You are right that the current Drill specific MergeProjectRule extends from 
Calcite's version and the only difference is to check complex output 
expression. Therefore, it make senses the currently Drill MergeProjectRule 
operates on Calcite logical convention only. 

Yes, in the future, we would consider allowing merging 2 DrillProjectRels. 
That should be another instance of rule, which operates on DRILL_LOGICAL 
convention only.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: IndexR, a new storage plugin for Drill

[Jinfeng Ni]

Forward to drill dev list. People on dev list might be interested in
this as well.


On Tue, Jan 3, 2017 at 8:22 AM, Jinfeng Ni  wrote:
> Looks like IndexR is very interesting storage plugin. Although I have
> not looked into the detail, I'm looking forward to seeing the PR and
> hopefully getting this into Drill!
>
> Thanks,
>
> Jinfeng
>
>
> On Tue, Jan 3, 2017 at 7:30 AM, WeiWan  wrote:
>> Hi Charles,
>>
>> It would be great if IndexR plugin can be merged into official Drill 
>> project. I will do some more tests based on latest Drill version and submit 
>> a PR.
>>
>> Regards
>> Flow Wei
>>
>>
>>
>>> On Jan 3, 2017, at 23:18, Charles Givre  wrote:
>>>
>>> This sounds really interesting.  Will you be submitting a PR to integrate 
>>> this into the main Drill codebase?
>>> — C
>>>
 On Jan 3, 2017, at 03:35, WeiWan  wrote:

 IndexR is a distributed, columnar storage system based on HDFS, which 
 focus on fast analyse, both for massive static(historical) data and 
 rapidly ingesting realtime data. IndexR is designed for OLAP.

 Fast analyze on large dataset
 Realtime ingestion with zero delay for query
 Deep integration with Hadoop ecosystem
 Hardware efficiency
 Highly avaliable, scalable, manageable and simple
 Adapted with popular query engines like Apache Drill, Apache Hive, etc.

 And now it is open source.

 Project: https://github.com/shunfei/indexr 
 
 Wiki: https://github.com/shunfei/indexr/wiki 
 

 IndexR is original developed by Sunteng Tech. This project started a year 
 ago and now has been deployed to several productions in our company. The 
 whole cluster consumes over 30 billions events each day in realtime from 
 Kafka. The largest table contains over 10 billions rows (after rollup) and 
 rapidly increasing. Most of the statistic/analyze queries’ latency is less 
 than 3 seconds in real world production environment.

 Currently it is mainly used as Drill and Hive storage plugin. It should be 
 quite easy to master.

 We hope IndexR be a favor to you and make it better.

 Regards
 Flow Wei



>>>
>>

[GitHub] drill pull request #705: DRILL-5159: Make sure Drill's ProjectMergeRule oper...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/705#discussion_r94437725
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillMergeProjectRule.java
 ---
@@ -48,6 +50,12 @@ public boolean matches(RelOptRuleCall call) {
 Project topProject = call.rel(0);
 Project bottomProject = call.rel(1);
 
+// Make sure both projects be LogicalProject.
+if (topProject.getTraitSet().getTrait(ConventionTraitDef.INSTANCE) != 
Convention.NONE ||
--- End diff --

At first glance, it is non-intuitive that a Drill specific MergeProjectRule 
returns False if a DRILL_LOGICAL convention is encountered.  However, 
currently, this rule only checks for the complex expression (e.g 
convert_to_json), so your change is fine in the context of the current rule.  
In future, would we consider allowing merging 2 DrillProjectRels ? (I think 
yes). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5175) Unable to Query a table has special characters

[Ravan (JIRA)]

Ravan created DRILL-5175:


 Summary: Unable to Query a table has special characters
 Key: DRILL-5175
 URL: https://issues.apache.org/jira/browse/DRILL-5175
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization
Affects Versions: 1.8.0
Reporter: Ravan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5174) when different Drillbit versions are up on a cluster - we need to report an error

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5174:
-

 Summary: when different Drillbit versions are up on a cluster - we 
need to report an error
 Key: DRILL-5174
 URL: https://issues.apache.org/jira/browse/DRILL-5174
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.10.0
Reporter: Khurram Faraaz


Drill 1.10.0 git commit : bbcf4b76

Note that one of the nodes in the 4 node cluster had a Drillbit from 1.9.0 and 
the other three Drillbits are of 1.10.0 version. We report an error on the Web 
UI (see DRILL-4604) for such a scenario, we need a similar message when 
querying sys.drillbits table from sqlline.

{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from sys.drillbits;
+---++---++--+--+
| hostname  | user_port  | control_port  | data_port  | current  | 
version  |
+---++---++--+--+
| centos-03.qa.lab  | 31010  | 31011 | 31012  | false| 
1.10.0-SNAPSHOT  |
| centos-02.qa.lab  | 31010  | 31011 | 31012  | false|  
|
| centos-04.qa.lab  | 31010  | 31011 | 31012  | false| 
1.10.0-SNAPSHOT  |
| centos-01.qa.lab  | 31010  | 31011 | 31012  | true | 
1.10.0-SNAPSHOT  |
+---++---++--+--+
4 rows selected (0.285 seconds)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)