[GitHub] weijietong commented on issue #1504: DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf…
weijietong commented on issue #1504: DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf… URL: https://github.com/apache/drill/pull/1504#issuecomment-435276846 @sohami have applied your review advice. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (DRILL-6826) Transfer RF ByteBuf owner causing the query hang
weijie.tong created DRILL-6826: -- Summary: Transfer RF ByteBuf owner causing the query hang Key: DRILL-6826 URL: https://issues.apache.org/jira/browse/DRILL-6826 Project: Apache Drill Issue Type: Improvement Reporter: weijie.tong To JPPD feature, when we transfer the received RF's ByteBuf owner at the WorkerBee's receiveRuntimeFilter method, the sent out aggregated RF will not receive any response, no Ack.OK , even exception information. This will eventually cause the query hanged at ForemanResult's close method which blocked at runtimeFilterRouter's waitForComplete method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Congratulations Hanu. Thanks Padma On Thu, Nov 1, 2018 at 7:44 PM weijie tong wrote: > Congratulations, Hanu! > > On Fri, Nov 2, 2018 at 8:22 AM Robert Hou wrote: > > > Congratulations, Hanu. Thanks for contributing to Drill. > > > > --Robert > > > > On Thu, Nov 1, 2018 at 4:06 PM Jyothsna Reddy > > wrote: > > > > > Congrats Hanu!! Well deserved :D > > > > > > Thank you, > > > Jyothsna > > > > > > On Thu, Nov 1, 2018 at 2:15 PM Sorabh Hamirwasia > > > > wrote: > > > > > > > Congratulations Hanu! > > > > > > > > Thanks, > > > > Sorabh > > > > > > > > On Thu, Nov 1, 2018 at 1:35 PM Hanumath Rao Maduri < > hanu@gmail.com > > > > > > > wrote: > > > > > > > > > Thank you all for the wishes! > > > > > > > > > > Thanks, > > > > > -Hanu > > > > > > > > > > On Thu, Nov 1, 2018 at 1:28 PM Chunhui Shi > > > > .invalid> > > > > > wrote: > > > > > > > > > > > Congratulations Hanu! > > > > > > > -- > > > > > > From:Arina Ielchiieva > > > > > > Send Time:2018 Nov 1 (Thu) 06:05 > > > > > > To:dev ; user > > > > > > Subject:[ANNOUNCE] New Committer: Hanumath Rao Maduri > > > > > > > > > > > > The Project Management Committee (PMC) for Apache Drill has > invited > > > > > > Hanumath > > > > > > Rao Maduri to become a committer, and we are pleased to announce > > that > > > > he > > > > > > has accepted. > > > > > > > > > > > > Hanumath became a contributor in 2017, making changes mostly in > the > > > > Drill > > > > > > planning side, including lateral / unnest support. He is also one > > of > > > > the > > > > > > contributors of index based planning and execution support. > > > > > > > > > > > > Welcome Hanumath, and thank you for your contributions! > > > > > > > > > > > > - Arina > > > > > > (on behalf of Drill PMC) > > > > > > > > > > > > > > > > > > > > >
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Congratulations, Hanu! On Fri, Nov 2, 2018 at 8:22 AM Robert Hou wrote: > Congratulations, Hanu. Thanks for contributing to Drill. > > --Robert > > On Thu, Nov 1, 2018 at 4:06 PM Jyothsna Reddy > wrote: > > > Congrats Hanu!! Well deserved :D > > > > Thank you, > > Jyothsna > > > > On Thu, Nov 1, 2018 at 2:15 PM Sorabh Hamirwasia > > wrote: > > > > > Congratulations Hanu! > > > > > > Thanks, > > > Sorabh > > > > > > On Thu, Nov 1, 2018 at 1:35 PM Hanumath Rao Maduri > > > > wrote: > > > > > > > Thank you all for the wishes! > > > > > > > > Thanks, > > > > -Hanu > > > > > > > > On Thu, Nov 1, 2018 at 1:28 PM Chunhui Shi > > > .invalid> > > > > wrote: > > > > > > > > > Congratulations Hanu! > > > > > -- > > > > > From:Arina Ielchiieva > > > > > Send Time:2018 Nov 1 (Thu) 06:05 > > > > > To:dev ; user > > > > > Subject:[ANNOUNCE] New Committer: Hanumath Rao Maduri > > > > > > > > > > The Project Management Committee (PMC) for Apache Drill has invited > > > > > Hanumath > > > > > Rao Maduri to become a committer, and we are pleased to announce > that > > > he > > > > > has accepted. > > > > > > > > > > Hanumath became a contributor in 2017, making changes mostly in the > > > Drill > > > > > planning side, including lateral / unnest support. He is also one > of > > > the > > > > > contributors of index based planning and execution support. > > > > > > > > > > Welcome Hanumath, and thank you for your contributions! > > > > > > > > > > - Arina > > > > > (on behalf of Drill PMC) > > > > > > > > > > > > > > >
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Congratulations, Hanu. Thanks for contributing to Drill. --Robert On Thu, Nov 1, 2018 at 4:06 PM Jyothsna Reddy wrote: > Congrats Hanu!! Well deserved :D > > Thank you, > Jyothsna > > On Thu, Nov 1, 2018 at 2:15 PM Sorabh Hamirwasia > wrote: > > > Congratulations Hanu! > > > > Thanks, > > Sorabh > > > > On Thu, Nov 1, 2018 at 1:35 PM Hanumath Rao Maduri > > wrote: > > > > > Thank you all for the wishes! > > > > > > Thanks, > > > -Hanu > > > > > > On Thu, Nov 1, 2018 at 1:28 PM Chunhui Shi > > .invalid> > > > wrote: > > > > > > > Congratulations Hanu! > > > > -- > > > > From:Arina Ielchiieva > > > > Send Time:2018 Nov 1 (Thu) 06:05 > > > > To:dev ; user > > > > Subject:[ANNOUNCE] New Committer: Hanumath Rao Maduri > > > > > > > > The Project Management Committee (PMC) for Apache Drill has invited > > > > Hanumath > > > > Rao Maduri to become a committer, and we are pleased to announce that > > he > > > > has accepted. > > > > > > > > Hanumath became a contributor in 2017, making changes mostly in the > > Drill > > > > planning side, including lateral / unnest support. He is also one of > > the > > > > contributors of index based planning and execution support. > > > > > > > > Welcome Hanumath, and thank you for your contributions! > > > > > > > > - Arina > > > > (on behalf of Drill PMC) > > > > > > > > > >
[GitHub] Ben-Zvi closed pull request #1514: DRILL-6798: Planner changes to support semi-join.
Ben-Zvi closed pull request #1514: DRILL-6798: Planner changes to support
semi-join.
URL: https://github.com/apache/drill/pull/1514
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/PlannerPhase.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/PlannerPhase.java
index ae55c9f21b5..17f8da52321 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/PlannerPhase.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/PlannerPhase.java
@@ -358,15 +358,18 @@ static RuleSet getDrillBasicRules(OptimizerRulesContext
optimizerRulesContext) {
* We have to create another copy of the ruleset with the context
dependent elements;
* this cannot be reused across queries.
*/
-final ImmutableSet basicRules =
ImmutableSet.builder()
+ImmutableSet.Builder basicRules =
ImmutableSet.builder()
.addAll(staticRuleSet)
.add(
DrillMergeProjectRule.getInstance(true,
RelFactories.DEFAULT_PROJECT_FACTORY,
optimizerRulesContext.getFunctionRegistry())
-)
-.build();
+);
+if (optimizerRulesContext.getPlannerSettings().isHashJoinEnabled() &&
+optimizerRulesContext.getPlannerSettings().isSemiJoinEnabled()) {
+ basicRules.add(RuleInstance.SEMI_JOIN_PROJECT_RULE);
+}
-return RuleSets.ofList(basicRules);
+return RuleSets.ofList(basicRules.build());
}
/**
@@ -474,7 +477,6 @@ static RuleSet getJoinPermRules(OptimizerRulesContext
optimizerRulesContext) {
static RuleSet getPhysicalRules(OptimizerRulesContext optimizerRulesContext)
{
final List ruleList = new ArrayList<>();
final PlannerSettings ps = optimizerRulesContext.getPlannerSettings();
-
ruleList.add(ConvertCountToDirectScan.AGG_ON_PROJ_ON_SCAN);
ruleList.add(ConvertCountToDirectScan.AGG_ON_SCAN);
ruleList.add(SortConvertPrule.INSTANCE);
@@ -509,9 +511,14 @@ static RuleSet getPhysicalRules(OptimizerRulesContext
optimizerRulesContext) {
if (ps.isHashJoinEnabled()) {
ruleList.add(HashJoinPrule.DIST_INSTANCE);
-
+ if (ps.isSemiJoinEnabled()) {
+ruleList.add(HashJoinPrule.SEMI_DIST_INSTANCE);
+ }
if(ps.isBroadcastJoinEnabled()){
ruleList.add(HashJoinPrule.BROADCAST_INSTANCE);
+if (ps.isSemiJoinEnabled()) {
+ ruleList.add(HashJoinPrule.SEMI_BROADCAST_INSTANCE);
+}
}
}
@@ -521,7 +528,6 @@ static RuleSet getPhysicalRules(OptimizerRulesContext
optimizerRulesContext) {
if(ps.isBroadcastJoinEnabled()){
ruleList.add(MergeJoinPrule.BROADCAST_INSTANCE);
}
-
}
// NLJ plans consist of broadcasting the right child, hence we need
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/RuleInstance.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/RuleInstance.java
index 8aec96c947f..b14488c9ca5 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/RuleInstance.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/RuleInstance.java
@@ -18,8 +18,11 @@
package org.apache.drill.exec.planner;
import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
import org.apache.calcite.plan.volcano.AbstractConverter;
+import org.apache.calcite.rel.core.Aggregate;
import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.Project;
import org.apache.calcite.rel.logical.LogicalAggregate;
import org.apache.calcite.rel.logical.LogicalCalc;
import org.apache.calcite.rel.logical.LogicalJoin;
@@ -39,12 +42,13 @@
import org.apache.calcite.rel.rules.ProjectToWindowRule;
import org.apache.calcite.rel.rules.ProjectWindowTransposeRule;
import org.apache.calcite.rel.rules.ReduceExpressionsRule;
+import org.apache.calcite.rel.rules.SemiJoinRule;
import org.apache.calcite.rel.rules.SortRemoveRule;
import org.apache.calcite.rel.rules.SubQueryRemoveRule;
import org.apache.calcite.rel.rules.UnionToDistinctRule;
import org.apache.drill.exec.planner.logical.DrillConditions;
import org.apache.drill.exec.planner.logical.DrillRelFactories;
-
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
/**
* Contains rule instances which use custom RelBuilder.
*/
@@ -58,6 +62,15 @@
new UnionToDistinctRule(LogicalUnion.class,
DrillRelFactories.LOGICAL_BUILDER);
+ SemiJoinRule SEMI_JOIN_PROJECT_RULE = new
SemiJoinRule.ProjectToSemiJoinRule(Project.class, Join.class, Aggregate.class,
+ DrillRelFactories.LOGICAL_BUILDER, "DrillSemiJoinRule:project") {
+public boolean matches(RelOptRuleCall call) {
+ Preconditions.che
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Congrats Hanu!! Well deserved :D Thank you, Jyothsna On Thu, Nov 1, 2018 at 2:15 PM Sorabh Hamirwasia wrote: > Congratulations Hanu! > > Thanks, > Sorabh > > On Thu, Nov 1, 2018 at 1:35 PM Hanumath Rao Maduri > wrote: > > > Thank you all for the wishes! > > > > Thanks, > > -Hanu > > > > On Thu, Nov 1, 2018 at 1:28 PM Chunhui Shi > .invalid> > > wrote: > > > > > Congratulations Hanu! > > > -- > > > From:Arina Ielchiieva > > > Send Time:2018 Nov 1 (Thu) 06:05 > > > To:dev ; user > > > Subject:[ANNOUNCE] New Committer: Hanumath Rao Maduri > > > > > > The Project Management Committee (PMC) for Apache Drill has invited > > > Hanumath > > > Rao Maduri to become a committer, and we are pleased to announce that > he > > > has accepted. > > > > > > Hanumath became a contributor in 2017, making changes mostly in the > Drill > > > planning side, including lateral / unnest support. He is also one of > the > > > contributors of index based planning and execution support. > > > > > > Welcome Hanumath, and thank you for your contributions! > > > > > > - Arina > > > (on behalf of Drill PMC) > > > > > >
Re: November Apache Drill board report
Hi Arina, Lateral/Unnest feature was part of 1.14 though it was disabled by default. Should we mention it as part of 1.14 enhancements in the report? Thanks, Sorabh On Thu, Nov 1, 2018 at 9:29 AM Arina Yelchiyeva wrote: > Thanks, Aman! Updated the report. > I went too far with 2019, luckily the meet up will be much earlier :) > > = > > ## Description: > - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud > Storage. > > ## Issues: > - There are no issues requiring board attention at this time. > > ## Activity: > - Since the last board report, Drill has released version 1.14.0, >including the following enhancements: > - Drill in a Docker container > - Image metadata format plugin > - Upgrade to Calcite 1.16.0 > - Kafka plugin push down support > - Phonetic and String functions > - Enhanced decimal data support > - Spill to disk for the Hash Join support > - CGROUPs resource management support > - There were active discussions about schema provision in Drill. >Based on these discussions two projects are currently evolving: >Drill metastore and schema provision in the file and in a query. > - Apache Drill book has been written by two PMC members (Charles and > Paul). > - Drill developer meet up will be held on November 14, 2018. >The following areas are going to be discussed: > - Storage plugins > - Schema discovery & Evolution > - Metadata Management > - Resource management > - Integration with Apache Arrow > > ## Health report: > - The project is healthy. Development activity >as reflected in the pull requests and JIRAs is good. > - Activity on the dev and user mailing lists are stable. > - Three committers and three new PMC member were added in the last period. > > ## PMC changes: > > - Currently 23 PMC members. > - New PMC members: > - Boaz Ben-Zvi was added to the PMC on Fri Aug 17 2018 > - Charles Givre was added to the PMC on Mon Sep 03 2018 > - Vova Vysotskyi was added to the PMC on Fri Aug 24 2018 > > ## Committer base changes: > > - Currently 48 committers. > - New commmitters: > - Chunhui Shi was added as a committer on Thu Sep 27 2018 > - Gautam Parai was added as a committer on Mon Oct 22 2018 > - Weijie Tong was added as a committer on Fri Aug 31 2018 > > ## Releases: > > - 1.14.0 was released on Sat Aug 04 2018 > > ## Mailing list activity: > > - dev@drill.apache.org: > - 427 subscribers (down -6 in the last 3 months): > - 2827 emails sent to list (2126 in previous quarter) > > - iss...@drill.apache.org: > - 18 subscribers (down -1 in the last 3 months): > - 3487 emails sent to list (4769 in previous quarter) > > - u...@drill.apache.org: > - 597 subscribers (down -6 in the last 3 months): > - 332 emails sent to list (346 in previous quarter) > > > ## JIRA activity: > > - 164 JIRA tickets created in the last 3 months > - 128 JIRA tickets closed/resolved in the last 3 months > > On Thu, Nov 1, 2018 at 6:20 PM Aman Sinha wrote: > > >Docket container ==> 'Docker' > >November 14, 2019 ==> 2018 :) (this is wrong in email that was > sent > > out) > > > > Rest LGTM. > > > > On Thu, Nov 1, 2018 at 6:42 AM Arina Ielchiieva > wrote: > > > > > Hi all, > > > > > > please take a look at the draft board report for the last quarter and > let > > > me know if you have any comments. > > > > > > Thanks, > > > Arina > > > > > > = > > > > > > ## Description: > > > - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud > > > Storage. > > > > > > ## Issues: > > > - There are no issues requiring board attention at this time. > > > > > > ## Activity: > > > - Since the last board report, Drill has released version 1.14.0, > > >including the following enhancements: > > > - Drill in a Docket container > > > - Image metadata format plugin > > > - Upgrade to Calcite 1.16.0 > > > - Kafka plugin push down support > > > - Phonetic and String functions > > > - Enhanced decimal data support > > > - Spill to disk for the Hash Join support > > > - CGROUPs resource management support > > > - There were active discussions about schema provision in Drill. > > >Based on these discussions two projects are currently evolving: > > >Drill metastore and schema provision in the file and in a query. > > > - Apache Drill book has been written by two PMC members (Charles and > > > Paul). > > > - Drill developer meet up will be held on November 14, 2019. > > >The following areas are going to be discussed: > > > - Storage plugins > > > - Schema discovery & Evolution > > > - Metadata Management > > > - Resource management > > > - Integration with Apache Arrow > > > > > > ## Health report: > > > - The project is healthy. Development activity > > >as reflected in the pull requests and JIRAs is good. > > > - Activity on the dev and user mailing lists are stable. > > > - Three committers and three new PMC member were added in the last > > period. > > > > > > ## PMC changes:
[GitHub] sohami closed pull request #1520: DRILL-6819: Remove invisible back link in Drill WebUI
sohami closed pull request #1520: DRILL-6819: Remove invisible back link in
Drill WebUI
URL: https://github.com/apache/drill/pull/1520
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/exec/java-exec/src/main/resources/rest/error.ftl
b/exec/java-exec/src/main/resources/rest/error.ftl
index aafdfc31f50..398ae078032 100644
--- a/exec/java-exec/src/main/resources/rest/error.ftl
+++ b/exec/java-exec/src/main/resources/rest/error.ftl
@@ -18,7 +18,6 @@
-->
-back
${model.printStackTrace()}
diff --git a/exec/java-exec/src/main/resources/rest/index.ftl
b/exec/java-exec/src/main/resources/rest/index.ftl
index 3b2be64ed07..10385c25587 100644
--- a/exec/java-exec/src/main/resources/rest/index.ftl
+++ b/exec/java-exec/src/main/resources/rest/index.ftl
@@ -37,7 +37,6 @@
<#macro page_body>
- back
diff --git a/exec/java-exec/src/main/resources/rest/login.ftl
b/exec/java-exec/src/main/resources/rest/login.ftl
index decf280b8d2..790c93496aa 100644
--- a/exec/java-exec/src/main/resources/rest/login.ftl
+++ b/exec/java-exec/src/main/resources/rest/login.ftl
@@ -22,7 +22,6 @@
<#macro page_body>
-back
diff --git a/exec/java-exec/src/main/resources/rest/logs/list.ftl
b/exec/java-exec/src/main/resources/rest/logs/list.ftl
index 7cde20b9e20..7482a4e7af7 100644
--- a/exec/java-exec/src/main/resources/rest/logs/list.ftl
+++ b/exec/java-exec/src/main/resources/rest/logs/list.ftl
@@ -22,7 +22,6 @@
<#macro page_body>
-back
diff --git a/exec/java-exec/src/main/resources/rest/logs/log.ftl
b/exec/java-exec/src/main/resources/rest/logs/log.ftl
index ae152cd4c86..0a65a92919d 100644
--- a/exec/java-exec/src/main/resources/rest/logs/log.ftl
+++ b/exec/java-exec/src/main/resources/rest/logs/log.ftl
@@ -22,7 +22,6 @@
<#macro page_body>
-back
${model.getName()} (last
${model.getMaxLines()} lines)
diff --git a/exec/java-exec/src/main/resources/rest/metrics/metrics.ftl
b/exec/java-exec/src/main/resources/rest/metrics/metrics.ftl
index d0263d3b240..629f8300a55 100644
--- a/exec/java-exec/src/main/resources/rest/metrics/metrics.ftl
+++ b/exec/java-exec/src/main/resources/rest/metrics/metrics.ftl
@@ -22,7 +22,6 @@
<#macro page_body>
- back
diff --git a/exec/java-exec/src/main/resources/rest/options.ftl
b/exec/java-exec/src/main/resources/rest/options.ftl
index 9d6a684eb44..085fb451b1d 100644
--- a/exec/java-exec/src/main/resources/rest/options.ftl
+++ b/exec/java-exec/src/main/resources/rest/options.ftl
@@ -41,7 +41,6 @@ table.sortable thead .sorting_desc { background-image:
url("/static/img/black-de
<#macro page_body>
- back
diff --git a/exec/java-exec/src/main/resources/rest/profile/list.ftl
b/exec/java-exec/src/main/resources/rest/profile/list.ftl
index 602a0e328c6..5b8d301a635 100644
--- a/exec/java-exec/src/main/resources/rest/profile/list.ftl
+++ b/exec/java-exec/src/main/resources/rest/profile/list.ftl
@@ -76,7 +76,6 @@
<#macro page_body>
- back
<#if (model.getErrors()?size > 0) >
diff --git a/exec/java-exec/src/main/resources/rest/profile/profile.ftl
b/exec/java-exec/src/main/resources/rest/profile/profile.ftl
index 13bd8c45d69..61f63fe0f23 100644
--- a/exec/java-exec/src/main/resources/rest/profile/profile.ftl
+++ b/exec/java-exec/src/main/resources/rest/profile/profile.ftl
@@ -65,7 +65,6 @@ table.sortable thead .sorting_desc { background-image:
url("/static/img/black-de
<#macro page_body>
- back
Query and Planning
diff --git a/exec/java-exec/src/main/resources/rest/query/errorMessage.ftl
b/exec/java-exec/src/main/resources/rest/query/errorMessage.ftl
index 00ad4c8469a..64662080efc 100644
--- a/exec/java-exec/src/main/resources/rest/query/errorMessage.ftl
+++ b/exec/java-exec/src/main/resources/rest/query/errorMessage.ftl
@@ -22,7 +22,6 @@
<#macro page_body>
- back
Query Failed: An Error Occurred
diff --git a/exec/java-exec/src/main/resources/rest/query/list.ftl
b/exec/java-exec/src/main/resources/rest/query/list.ftl
index cbde814832f..4242d3ffeb1 100644
--- a/exec/java-exec/src/main/resources/rest/query/list.ftl
+++ b/exec/java-exec/src/main/resources/rest/query/list.ftl
@@ -22,7 +22,6 @@
<#macro page_body>
- back
Under Construction.
diff --git a/exec/java-exec/src/main/resources/rest/query/query.ftl
b/exec/java-exec/src/main/resources/rest/query/query.ftl
index 078333e3f8e..f96d63940d2 100644
--- a/exec/java-exec/src/main/resources/rest/query/query.ftl
+++ b/exec/java-exec/src/main/resources/rest/query/query.ftl
@@ -34,7 +34,6 @@
<#macro page_body>
- back
diff --git a/exec/java-exec/src/main/resources/rest/query/result.ftl
b/exec/java-exec/src/main/resources/rest/query/result.ftl
index 817
[GitHub] sohami closed pull request #1517: DRILL-6822: Fixed imports for DRILL-6381
sohami closed pull request #1517: DRILL-6822: Fixed imports for DRILL-6381
URL: https://github.com/apache/drill/pull/1517
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDescriptor.java
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDescriptor.java
index 75e6bc23973..ae386ab93a0 100644
---
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDescriptor.java
+++
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDescriptor.java
@@ -22,8 +22,8 @@
import java.util.List;
import java.util.Set;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Sets;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+import org.apache.drill.shaded.guava.com.google.common.collect.Sets;
import org.apache.calcite.plan.RelOptCost;
import org.apache.calcite.plan.RelOptPlanner;
@@ -40,8 +40,8 @@
import org.apache.drill.exec.util.EncodedSchemaPathSet;
import org.apache.drill.common.expression.LogicalExpression;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableSet;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
public class MapRDBIndexDescriptor extends DrillIndexDescriptor {
diff --git
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDiscover.java
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDiscover.java
index aed3e045a02..f828ba02daf 100644
---
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDiscover.java
+++
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDiscover.java
@@ -18,7 +18,7 @@
package org.apache.drill.exec.planner.index;
-import com.google.common.collect.Maps;
+import org.apache.drill.shaded.guava.com.google.common.collect.Maps;
import com.mapr.db.Admin;
import com.mapr.db.MapRDB;
import com.mapr.db.exceptions.DBException;
diff --git
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBStatistics.java
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBStatistics.java
index e129b968bf7..6fedaffd092 100644
---
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBStatistics.java
+++
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBStatistics.java
@@ -17,11 +17,11 @@
*/
package org.apache.drill.exec.planner.index;
-import com.google.common.base.Charsets;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
+import org.apache.drill.shaded.guava.com.google.common.base.Charsets;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
+import org.apache.drill.shaded.guava.com.google.common.collect.Maps;
import org.apache.calcite.plan.RelOptUtil;
import org.apache.calcite.rel.RelNode;
import org.apache.calcite.rel.metadata.RelMdUtil;
diff --git
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBTableCache.java
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBTableCache.java
index f35a4c41668..924d9c0e8e7 100644
---
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBTableCache.java
+++
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBTableCache.java
@@ -17,11 +17,11 @@
*/
package org.apache.drill.exec.store.mapr.db;
-import com.google.common.cache.CacheBuilder;
-import com.google.common.cache.CacheLoader;
-import com.google.common.cache.LoadingCache;
-import com.google.common.cache.RemovalListener;
-import com.google.common.cache.RemovalNotification;
+import org.apache.drill.shaded.guava.com.google.common.cache.CacheBuilder;
+import org.apache.drill.shaded.guava.com.google.common.cache.CacheLoader;
+import org.apache.drill.shaded.guava.com.google.common.cache.LoadingCache;
+import org.apache.drill.shaded.guava.com.google.common.cache.RemovalListener;
+import
org.apache.drill.shaded.guava.com.google.common.cache.RemovalNotification;
import com.mapr.db.Table;
import com.mapr.db.impl.MapRDBImpl;
import com.mapr.db.index.IndexDesc;
diff --git
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/OjaiFunctionsProcessor.java
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/jso
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Congratulations Hanu! Thanks, Sorabh On Thu, Nov 1, 2018 at 1:35 PM Hanumath Rao Maduri wrote: > Thank you all for the wishes! > > Thanks, > -Hanu > > On Thu, Nov 1, 2018 at 1:28 PM Chunhui Shi .invalid> > wrote: > > > Congratulations Hanu! > > -- > > From:Arina Ielchiieva > > Send Time:2018 Nov 1 (Thu) 06:05 > > To:dev ; user > > Subject:[ANNOUNCE] New Committer: Hanumath Rao Maduri > > > > The Project Management Committee (PMC) for Apache Drill has invited > > Hanumath > > Rao Maduri to become a committer, and we are pleased to announce that he > > has accepted. > > > > Hanumath became a contributor in 2017, making changes mostly in the Drill > > planning side, including lateral / unnest support. He is also one of the > > contributors of index based planning and execution support. > > > > Welcome Hanumath, and thank you for your contributions! > > > > - Arina > > (on behalf of Drill PMC) > > >
Re: [Agenda] Drill developer meetup 2018
Hello All, There was typo for the year in the mail. It should be 2018 instead of 2019. Thanks Aman for correcting it. Regards, -Hanu On Thu, Nov 1, 2018 at 6:30 AM Charles Givre wrote: > Hi Hanumath, > This looks great!! Will you be streaming the event for those of us not in > the Bay Area? > Thx, > — C > > > On Nov 1, 2018, at 00:10, Hanumath Rao Maduri > wrote: > > > > Drill Developers, > > > > > > I am quite excited to announce the details of the Drill developers day > > 2018. I have consolidated the topics from our earlier discussions and > > prioritized them according to the votes. > > > > > > MapR has offered to host it on Nov 14th in Training room downstairs. > > > > > > Here is the exact location > > > > > > Training Room at > > > > 4555 Great America Pkwy, Suite 201, Santa Clara, CA, 95054. > > > > > > Please find the agenda for the meetup. > > > > > > > > *Lunch starts at 12:00PM.* > > > > > > *[12:25 - 12:40] Welcome * > > > > - Recap on last year's activities > > - Preview of this year's focus > > > > *[12:40 - 1:00] Storage plugins* > > > > > > > > - Adding new storage plugins for the following: > > - Netflix Iceberg, Kudu(some code already exists), Cassandra, > > Elasticsearch, Carbondata, ORC/XML file formats, Spark > > RDD/DataFrames/Datasets, Graph databases & more > > - Improving documentation related to Storage plugins > > > > > > *[1:00 - 1:45] Schema discovery & Evolution* > > > > > > > > - Creation, management of schema > > - Handling schema changes in certain common cases > > - Handling NULL values elegantly > > - Schema learning (similar to MSGpack plugin) > > - Query hints > > > > *[1:45 - 2:30] Metadata Management* > > > > > > > > - Defining an abstraction layer for various types of metadata: views, > > schema, statistics, security > > - Underlying storage for metadata: what are the options and their > > trade-offs? > > - Hive metastore > > - Parquet metadata cache (parquet specific for row group metadata) > > - Ease of using the parquet files generated by other engines (like > spark) > > > > > > *[2:30 - 2:45] Break* > > > > > > *[2:45 - 4:00] Resource management* > > > > > > > > - Resource limits per query > > - Optimal memory assignment for blocking operators based on stats > > - Enhancing the blocking and exchange operators to live within memory > > limits > > - Aligning with admission control/queueing (YARN concepts) > > - Query scheduling based on queues using tagging and costing > > - Drill on kubernetes > > > > > > *[4:00 - 4:20] Apache Arrow* > > > > - Benefits of integrating Apache Drill with Apache Arrow > > - Possible trade-offs & implementation hurdles > > > > *[4:20 - 4:40] **Performance Improvements* > > > > - Efficient handling of Broadcast/Semi/Anti Semi join > > - Drill Statistics handling > > - Optimizing complex Parquet reader > > > > Thanks, > > -Hanu > >
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Thank you all for the wishes! Thanks, -Hanu On Thu, Nov 1, 2018 at 1:28 PM Chunhui Shi wrote: > Congratulations Hanu! > -- > From:Arina Ielchiieva > Send Time:2018 Nov 1 (Thu) 06:05 > To:dev ; user > Subject:[ANNOUNCE] New Committer: Hanumath Rao Maduri > > The Project Management Committee (PMC) for Apache Drill has invited > Hanumath > Rao Maduri to become a committer, and we are pleased to announce that he > has accepted. > > Hanumath became a contributor in 2017, making changes mostly in the Drill > planning side, including lateral / unnest support. He is also one of the > contributors of index based planning and execution support. > > Welcome Hanumath, and thank you for your contributions! > > - Arina > (on behalf of Drill PMC) >
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Congratulations Hanu! -- From:Arina Ielchiieva Send Time:2018 Nov 1 (Thu) 06:05 To:dev ; user Subject:[ANNOUNCE] New Committer: Hanumath Rao Maduri The Project Management Committee (PMC) for Apache Drill has invited Hanumath Rao Maduri to become a committer, and we are pleased to announce that he has accepted. Hanumath became a contributor in 2017, making changes mostly in the Drill planning side, including lateral / unnest support. He is also one of the contributors of index based planning and execution support. Welcome Hanumath, and thank you for your contributions! - Arina (on behalf of Drill PMC)
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Congratulations Hanumath ! Aman On Thu, Nov 1, 2018 at 11:39 AM Paul Rogers wrote: > Congratulations Hanu! > > - Paul > > Sent from my iPhone > > > On Nov 1, 2018, at 11:09 AM, Kunal Khatua wrote: > > > > Congratulations, Hanu! > > On 11/1/2018 11:04:58 AM, Abhishek Girish wrote: > > Congratulations, Hanu! > > > >> On Thu, Nov 1, 2018 at 10:56 AM Khurram Faraaz wrote: > >> > >> Congratulations Hanu! > >> > >>> On Thu, Nov 1, 2018 at 10:14 AM Gautam Parai wrote: > >>> > >>> Congratulations Hanumath! Well deserved :) > >>> > >>> Gautam > >>> > >>> On Thu, Nov 1, 2018 at 9:44 AM AnilKumar B > >> wrote: > >>> > Congratulations Hanumath. > > Thanks & Regards, > B Anil Kumar. > > > On Thu, Nov 1, 2018 at 9:39 AM Vitalii Diravka > >>> wrote: > > > Congratulations! > > > > Kind regards > > Vitalii > > > > > > On Thu, Nov 1, 2018 at 5:43 PM salim achouche > > wrote: > > > >> Congrats Hanu! > >> > >> On Thu, Nov 1, 2018 at 6:05 AM Arina Ielchiieva > > wrote: > >> > >>> The Project Management Committee (PMC) for Apache Drill has > >> invited > >>> Hanumath > >>> Rao Maduri to become a committer, and we are pleased to announce > >>> that > > he > >>> has accepted. > >>> > >>> Hanumath became a contributor in 2017, making changes mostly in > >> the > > Drill > >>> planning side, including lateral / unnest support. He is also one > >>> of > > the > >>> contributors of index based planning and execution support. > >>> > >>> Welcome Hanumath, and thank you for your contributions! > >>> > >>> - Arina > >>> (on behalf of Drill PMC) > >>> > >> > >> > >> -- > >> Regards, > >> Salim > >> > > > > >>> > >> >
[GitHub] sohami commented on a change in pull request #1504: DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf…
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r230164571
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterRouter.java
##
@@ -125,33 +128,53 @@ public void waitForComplete() {
/**
* This method is passively invoked by receiving a runtime filter from the
network
- * @param runtimeFilterWritable
+ *
+ * @param srcRuntimeFilterWritable
*/
- public void registerRuntimeFilter(RuntimeFilterWritable
runtimeFilterWritable) {
-broadcastAggregatedRuntimeFilter(runtimeFilterWritable);
+ public void register(RuntimeFilterWritable srcRuntimeFilterWritable) {
+BitData.RuntimeFilterBDef runtimeFilterB =
srcRuntimeFilterWritable.getRuntimeFilterBDef();
+int joinMajorId = runtimeFilterB.getMajorFragmentId();
+int buildSideRfNumber;
+RuntimeFilterWritable toAggregated;
+synchronized (this) {
+ buildSideRfNumber = joinMjId2rfNumber.get(joinMajorId);
+ buildSideRfNumber--;
+ joinMjId2rfNumber.put(joinMajorId, buildSideRfNumber);
+ toAggregated = joinMjId2AggregatedRF.get(joinMajorId);
+ if (toAggregated == null) {
+toAggregated = srcRuntimeFilterWritable;
+toAggregated.retainBuffers(1);
Review comment:
makes sense. Thanks for explanation
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sohami commented on a change in pull request #1504: DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf…
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r230166642
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/filter/RuntimeFilterRecordBatch.java
##
@@ -263,4 +260,41 @@ public void dump() {
+ "originalRecordCount={}, batchSchema={}]",
container, sv2, toFilterFields, originalRecordCount,
incoming.getSchema());
}
+
+ public enum Metric implements MetricDef {
+FILTERED_ROWS, APPLIED_TIMES;
+
+@Override
+public int metricId() {
+ return ordinal();
+}
+ }
+
+ public void updateStats() {
+stats.setLongStat(Metric.FILTERED_ROWS, filteredRows);
+stats.setLongStat(Metric.APPLIED_TIMES, appliedTimes);
+ }
+
+ private void timedWaiting() {
+if (!enableRFWaiting || waited) {
+ return;
+}
+long startMs = System.currentTimeMillis();
+while (current == null && batchTimes > 0) {
Review comment:
ahh right. Thanks for explanation. Please add a comment for it that
downstream HashJoinBatch prefetches first batch from both sides in buildSchema
phase hence waiting is done post that phase.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sohami commented on a change in pull request #1504: DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf…
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r230165128
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterRouter.java
##
@@ -125,33 +128,53 @@ public void waitForComplete() {
/**
* This method is passively invoked by receiving a runtime filter from the
network
- * @param runtimeFilterWritable
+ *
+ * @param srcRuntimeFilterWritable
*/
- public void registerRuntimeFilter(RuntimeFilterWritable
runtimeFilterWritable) {
-broadcastAggregatedRuntimeFilter(runtimeFilterWritable);
+ public void register(RuntimeFilterWritable srcRuntimeFilterWritable) {
+BitData.RuntimeFilterBDef runtimeFilterB =
srcRuntimeFilterWritable.getRuntimeFilterBDef();
+int joinMajorId = runtimeFilterB.getMajorFragmentId();
+int buildSideRfNumber;
+RuntimeFilterWritable toAggregated;
+synchronized (this) {
+ buildSideRfNumber = joinMjId2rfNumber.get(joinMajorId);
+ buildSideRfNumber--;
+ joinMjId2rfNumber.put(joinMajorId, buildSideRfNumber);
+ toAggregated = joinMjId2AggregatedRF.get(joinMajorId);
+ if (toAggregated == null) {
+toAggregated = srcRuntimeFilterWritable;
+toAggregated.retainBuffers(1);
+ } else {
+toAggregated.aggregate(srcRuntimeFilterWritable);
+ }
+ joinMjId2AggregatedRF.put(joinMajorId, toAggregated);
+}
+if (buildSideRfNumber == 0) {
Review comment:
Yes makes sense. Somehow I missed that `buildSideRfNumber` is a local
variable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] denysord88 commented on issue #1520: DRILL-6819: Remove invisible back link in Drill WebUI
denysord88 commented on issue #1520: DRILL-6819: Remove invisible back link in Drill WebUI URL: https://github.com/apache/drill/pull/1520#issuecomment-435128508 @kkhatua please review. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] denysord88 opened a new pull request #1520: DRILL-6819: Remove invisible back link in Drill WebUI
denysord88 opened a new pull request #1520: DRILL-6819: Remove invisible back link in Drill WebUI URL: https://github.com/apache/drill/pull/1520 Details in [DRILL-6819](https://issues.apache.org/jira/browse/DRILL-6819). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] HanumathRao commented on issue #1514: DRILL-6798: Planner changes to support semi-join.
HanumathRao commented on issue #1514: DRILL-6798: Planner changes to support semi-join. URL: https://github.com/apache/drill/pull/1514#issuecomment-435120973 @amansinha100 Thank you for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Congratulations Hanumath! Well deserved :) Gautam On Thu, Nov 1, 2018 at 9:44 AM AnilKumar B wrote: > Congratulations Hanumath. > > Thanks & Regards, > B Anil Kumar. > > > On Thu, Nov 1, 2018 at 9:39 AM Vitalii Diravka wrote: > > > Congratulations! > > > > Kind regards > > Vitalii > > > > > > On Thu, Nov 1, 2018 at 5:43 PM salim achouche > > wrote: > > > > > Congrats Hanu! > > > > > > On Thu, Nov 1, 2018 at 6:05 AM Arina Ielchiieva > > wrote: > > > > > > > The Project Management Committee (PMC) for Apache Drill has invited > > > > Hanumath > > > > Rao Maduri to become a committer, and we are pleased to announce that > > he > > > > has accepted. > > > > > > > > Hanumath became a contributor in 2017, making changes mostly in the > > Drill > > > > planning side, including lateral / unnest support. He is also one of > > the > > > > contributors of index based planning and execution support. > > > > > > > > Welcome Hanumath, and thank you for your contributions! > > > > > > > > - Arina > > > > (on behalf of Drill PMC) > > > > > > > > > > > > > -- > > > Regards, > > > Salim > > > > > >
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Congratulations Hanumath. Thanks & Regards, B Anil Kumar. On Thu, Nov 1, 2018 at 9:39 AM Vitalii Diravka wrote: > Congratulations! > > Kind regards > Vitalii > > > On Thu, Nov 1, 2018 at 5:43 PM salim achouche > wrote: > > > Congrats Hanu! > > > > On Thu, Nov 1, 2018 at 6:05 AM Arina Ielchiieva > wrote: > > > > > The Project Management Committee (PMC) for Apache Drill has invited > > > Hanumath > > > Rao Maduri to become a committer, and we are pleased to announce that > he > > > has accepted. > > > > > > Hanumath became a contributor in 2017, making changes mostly in the > Drill > > > planning side, including lateral / unnest support. He is also one of > the > > > contributors of index based planning and execution support. > > > > > > Welcome Hanumath, and thank you for your contributions! > > > > > > - Arina > > > (on behalf of Drill PMC) > > > > > > > > > -- > > Regards, > > Salim > > >
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Congratulations! Kind regards Vitalii On Thu, Nov 1, 2018 at 5:43 PM salim achouche wrote: > Congrats Hanu! > > On Thu, Nov 1, 2018 at 6:05 AM Arina Ielchiieva wrote: > > > The Project Management Committee (PMC) for Apache Drill has invited > > Hanumath > > Rao Maduri to become a committer, and we are pleased to announce that he > > has accepted. > > > > Hanumath became a contributor in 2017, making changes mostly in the Drill > > planning side, including lateral / unnest support. He is also one of the > > contributors of index based planning and execution support. > > > > Welcome Hanumath, and thank you for your contributions! > > > > - Arina > > (on behalf of Drill PMC) > > > > > -- > Regards, > Salim >
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Congrats Hanu! On Thu, Nov 1, 2018 at 8:43 AM salim achouche wrote: > Congrats Hanu! > > On Thu, Nov 1, 2018 at 6:05 AM Arina Ielchiieva wrote: > > > The Project Management Committee (PMC) for Apache Drill has invited > > Hanumath > > Rao Maduri to become a committer, and we are pleased to announce that he > > has accepted. > > > > Hanumath became a contributor in 2017, making changes mostly in the Drill > > planning side, including lateral / unnest support. He is also one of the > > contributors of index based planning and execution support. > > > > Welcome Hanumath, and thank you for your contributions! > > > > - Arina > > (on behalf of Drill PMC) > > > > > -- > Regards, > Salim >
Re: November Apache Drill board report
Thanks, Aman! Updated the report. I went too far with 2019, luckily the meet up will be much earlier :) = ## Description: - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - Since the last board report, Drill has released version 1.14.0, including the following enhancements: - Drill in a Docker container - Image metadata format plugin - Upgrade to Calcite 1.16.0 - Kafka plugin push down support - Phonetic and String functions - Enhanced decimal data support - Spill to disk for the Hash Join support - CGROUPs resource management support - There were active discussions about schema provision in Drill. Based on these discussions two projects are currently evolving: Drill metastore and schema provision in the file and in a query. - Apache Drill book has been written by two PMC members (Charles and Paul). - Drill developer meet up will be held on November 14, 2018. The following areas are going to be discussed: - Storage plugins - Schema discovery & Evolution - Metadata Management - Resource management - Integration with Apache Arrow ## Health report: - The project is healthy. Development activity as reflected in the pull requests and JIRAs is good. - Activity on the dev and user mailing lists are stable. - Three committers and three new PMC member were added in the last period. ## PMC changes: - Currently 23 PMC members. - New PMC members: - Boaz Ben-Zvi was added to the PMC on Fri Aug 17 2018 - Charles Givre was added to the PMC on Mon Sep 03 2018 - Vova Vysotskyi was added to the PMC on Fri Aug 24 2018 ## Committer base changes: - Currently 48 committers. - New commmitters: - Chunhui Shi was added as a committer on Thu Sep 27 2018 - Gautam Parai was added as a committer on Mon Oct 22 2018 - Weijie Tong was added as a committer on Fri Aug 31 2018 ## Releases: - 1.14.0 was released on Sat Aug 04 2018 ## Mailing list activity: - dev@drill.apache.org: - 427 subscribers (down -6 in the last 3 months): - 2827 emails sent to list (2126 in previous quarter) - iss...@drill.apache.org: - 18 subscribers (down -1 in the last 3 months): - 3487 emails sent to list (4769 in previous quarter) - u...@drill.apache.org: - 597 subscribers (down -6 in the last 3 months): - 332 emails sent to list (346 in previous quarter) ## JIRA activity: - 164 JIRA tickets created in the last 3 months - 128 JIRA tickets closed/resolved in the last 3 months On Thu, Nov 1, 2018 at 6:20 PM Aman Sinha wrote: >Docket container ==> 'Docker' >November 14, 2019 ==> 2018 :) (this is wrong in email that was sent > out) > > Rest LGTM. > > On Thu, Nov 1, 2018 at 6:42 AM Arina Ielchiieva wrote: > > > Hi all, > > > > please take a look at the draft board report for the last quarter and let > > me know if you have any comments. > > > > Thanks, > > Arina > > > > = > > > > ## Description: > > - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud > > Storage. > > > > ## Issues: > > - There are no issues requiring board attention at this time. > > > > ## Activity: > > - Since the last board report, Drill has released version 1.14.0, > >including the following enhancements: > > - Drill in a Docket container > > - Image metadata format plugin > > - Upgrade to Calcite 1.16.0 > > - Kafka plugin push down support > > - Phonetic and String functions > > - Enhanced decimal data support > > - Spill to disk for the Hash Join support > > - CGROUPs resource management support > > - There were active discussions about schema provision in Drill. > >Based on these discussions two projects are currently evolving: > >Drill metastore and schema provision in the file and in a query. > > - Apache Drill book has been written by two PMC members (Charles and > > Paul). > > - Drill developer meet up will be held on November 14, 2019. > >The following areas are going to be discussed: > > - Storage plugins > > - Schema discovery & Evolution > > - Metadata Management > > - Resource management > > - Integration with Apache Arrow > > > > ## Health report: > > - The project is healthy. Development activity > >as reflected in the pull requests and JIRAs is good. > > - Activity on the dev and user mailing lists are stable. > > - Three committers and three new PMC member were added in the last > period. > > > > ## PMC changes: > > > > - Currently 23 PMC members. > > - New PMC members: > > - Boaz Ben-Zvi was added to the PMC on Fri Aug 17 2018 > > - Charles Givre was added to the PMC on Mon Sep 03 2018 > > - Vova Vysotskyi was added to the PMC on Fri Aug 24 2018 > > > > ## Committer base changes: > > > > - Currently 48 committers. > > - New commmitters: > > - Chunhui Shi was added as a committer on Thu Sep 27 2018 > > - Gautam Parai was added as a committer on Mon Oct 22 2018 > > - W
Re: November Apache Drill board report
Docket container ==> 'Docker' November 14, 2019 ==> 2018 :) (this is wrong in email that was sent out) Rest LGTM. On Thu, Nov 1, 2018 at 6:42 AM Arina Ielchiieva wrote: > Hi all, > > please take a look at the draft board report for the last quarter and let > me know if you have any comments. > > Thanks, > Arina > > = > > ## Description: > - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud > Storage. > > ## Issues: > - There are no issues requiring board attention at this time. > > ## Activity: > - Since the last board report, Drill has released version 1.14.0, >including the following enhancements: > - Drill in a Docket container > - Image metadata format plugin > - Upgrade to Calcite 1.16.0 > - Kafka plugin push down support > - Phonetic and String functions > - Enhanced decimal data support > - Spill to disk for the Hash Join support > - CGROUPs resource management support > - There were active discussions about schema provision in Drill. >Based on these discussions two projects are currently evolving: >Drill metastore and schema provision in the file and in a query. > - Apache Drill book has been written by two PMC members (Charles and > Paul). > - Drill developer meet up will be held on November 14, 2019. >The following areas are going to be discussed: > - Storage plugins > - Schema discovery & Evolution > - Metadata Management > - Resource management > - Integration with Apache Arrow > > ## Health report: > - The project is healthy. Development activity >as reflected in the pull requests and JIRAs is good. > - Activity on the dev and user mailing lists are stable. > - Three committers and three new PMC member were added in the last period. > > ## PMC changes: > > - Currently 23 PMC members. > - New PMC members: > - Boaz Ben-Zvi was added to the PMC on Fri Aug 17 2018 > - Charles Givre was added to the PMC on Mon Sep 03 2018 > - Vova Vysotskyi was added to the PMC on Fri Aug 24 2018 > > ## Committer base changes: > > - Currently 48 committers. > - New commmitters: > - Chunhui Shi was added as a committer on Thu Sep 27 2018 > - Gautam Parai was added as a committer on Mon Oct 22 2018 > - Weijie Tong was added as a committer on Fri Aug 31 2018 > > ## Releases: > > - 1.14.0 was released on Sat Aug 04 2018 > > ## Mailing list activity: > > - dev@drill.apache.org: > - 427 subscribers (down -6 in the last 3 months): > - 2827 emails sent to list (2126 in previous quarter) > > - iss...@drill.apache.org: > - 18 subscribers (down -1 in the last 3 months): > - 3487 emails sent to list (4769 in previous quarter) > > - u...@drill.apache.org: > - 597 subscribers (down -6 in the last 3 months): > - 332 emails sent to list (346 in previous quarter) > > > ## JIRA activity: > > - 164 JIRA tickets created in the last 3 months > - 128 JIRA tickets closed/resolved in the last 3 months >
Re: [ANNOUNCE] New Committer: Hanumath Rao Maduri
Congrats Hanu! On Thu, Nov 1, 2018 at 6:05 AM Arina Ielchiieva wrote: > The Project Management Committee (PMC) for Apache Drill has invited > Hanumath > Rao Maduri to become a committer, and we are pleased to announce that he > has accepted. > > Hanumath became a contributor in 2017, making changes mostly in the Drill > planning side, including lateral / unnest support. He is also one of the > contributors of index based planning and execution support. > > Welcome Hanumath, and thank you for your contributions! > > - Arina > (on behalf of Drill PMC) > -- Regards, Salim
November Apache Drill board report
Hi all, please take a look at the draft board report for the last quarter and let me know if you have any comments. Thanks, Arina = ## Description: - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - Since the last board report, Drill has released version 1.14.0, including the following enhancements: - Drill in a Docket container - Image metadata format plugin - Upgrade to Calcite 1.16.0 - Kafka plugin push down support - Phonetic and String functions - Enhanced decimal data support - Spill to disk for the Hash Join support - CGROUPs resource management support - There were active discussions about schema provision in Drill. Based on these discussions two projects are currently evolving: Drill metastore and schema provision in the file and in a query. - Apache Drill book has been written by two PMC members (Charles and Paul). - Drill developer meet up will be held on November 14, 2019. The following areas are going to be discussed: - Storage plugins - Schema discovery & Evolution - Metadata Management - Resource management - Integration with Apache Arrow ## Health report: - The project is healthy. Development activity as reflected in the pull requests and JIRAs is good. - Activity on the dev and user mailing lists are stable. - Three committers and three new PMC member were added in the last period. ## PMC changes: - Currently 23 PMC members. - New PMC members: - Boaz Ben-Zvi was added to the PMC on Fri Aug 17 2018 - Charles Givre was added to the PMC on Mon Sep 03 2018 - Vova Vysotskyi was added to the PMC on Fri Aug 24 2018 ## Committer base changes: - Currently 48 committers. - New commmitters: - Chunhui Shi was added as a committer on Thu Sep 27 2018 - Gautam Parai was added as a committer on Mon Oct 22 2018 - Weijie Tong was added as a committer on Fri Aug 31 2018 ## Releases: - 1.14.0 was released on Sat Aug 04 2018 ## Mailing list activity: - dev@drill.apache.org: - 427 subscribers (down -6 in the last 3 months): - 2827 emails sent to list (2126 in previous quarter) - iss...@drill.apache.org: - 18 subscribers (down -1 in the last 3 months): - 3487 emails sent to list (4769 in previous quarter) - u...@drill.apache.org: - 597 subscribers (down -6 in the last 3 months): - 332 emails sent to list (346 in previous quarter) ## JIRA activity: - 164 JIRA tickets created in the last 3 months - 128 JIRA tickets closed/resolved in the last 3 months
[GitHub] oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM URL: https://github.com/apache/drill/pull/1446#issuecomment-435043350 @vvysotskyi 1) Javassist with Java 11 support released today (https://github.com/jboss-javassist/javassist/releases/tag/rel_3_24_0_ga) and now missing in central maven repository. I will provide fixes when this version appears in maven. :) 2) According to http://www.javassist.org/html/javassist/util/proxy/DefineClassHelper.html#toClass(java.lang.String,java.lang.Class,java.lang.ClassLoader,java.security.ProtectionDomain,byte%5B%5D), --add-opens should be added for java 9+ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] oleg-zinovev commented on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
oleg-zinovev commented on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM URL: https://github.com/apache/drill/pull/1446#issuecomment-435043350 1) Javassist with Java 11 support released today (https://github.com/jboss-javassist/javassist/releases/tag/rel_3_24_0_ga) and now missing in central maven repository. I will provide fixes when this version appears in maven. :) 2) According to http://www.javassist.org/html/javassist/util/proxy/DefineClassHelper.html#toClass(java.lang.String,java.lang.Class,java.lang.ClassLoader,java.security.ProtectionDomain,byte%5B%5D), --add-opens should be added for java 9+ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: [Agenda] Drill developer meetup 2019
Hi Hanumath, This looks great!! Will you be streaming the event for those of us not in the Bay Area? Thx, — C > On Nov 1, 2018, at 00:10, Hanumath Rao Maduri wrote: > > Drill Developers, > > > I am quite excited to announce the details of the Drill developers day > 2019. I have consolidated the topics from our earlier discussions and > prioritized them according to the votes. > > > MapR has offered to host it on Nov 14th in Training room downstairs. > > > Here is the exact location > > > Training Room at > > 4555 Great America Pkwy, Suite 201, Santa Clara, CA, 95054. > > > Please find the agenda for the meetup. > > > > *Lunch starts at 12:00PM.* > > > *[12:25 - 12:40] Welcome * > > - Recap on last year's activities > - Preview of this year's focus > > *[12:40 - 1:00] Storage plugins* > > > > - Adding new storage plugins for the following: > - Netflix Iceberg, Kudu(some code already exists), Cassandra, > Elasticsearch, Carbondata, ORC/XML file formats, Spark > RDD/DataFrames/Datasets, Graph databases & more > - Improving documentation related to Storage plugins > > > *[1:00 - 1:45] Schema discovery & Evolution* > > > > - Creation, management of schema > - Handling schema changes in certain common cases > - Handling NULL values elegantly > - Schema learning (similar to MSGpack plugin) > - Query hints > > *[1:45 - 2:30] Metadata Management* > > > > - Defining an abstraction layer for various types of metadata: views, > schema, statistics, security > - Underlying storage for metadata: what are the options and their > trade-offs? > - Hive metastore > - Parquet metadata cache (parquet specific for row group metadata) > - Ease of using the parquet files generated by other engines (like spark) > > > *[2:30 - 2:45] Break* > > > *[2:45 - 4:00] Resource management* > > > > - Resource limits per query > - Optimal memory assignment for blocking operators based on stats > - Enhancing the blocking and exchange operators to live within memory > limits > - Aligning with admission control/queueing (YARN concepts) > - Query scheduling based on queues using tagging and costing > - Drill on kubernetes > > > *[4:00 - 4:20] Apache Arrow* > > - Benefits of integrating Apache Drill with Apache Arrow > - Possible trade-offs & implementation hurdles > > *[4:20 - 4:40] **Performance Improvements* > > - Efficient handling of Broadcast/Semi/Anti Semi join > - Drill Statistics handling > - Optimizing complex Parquet reader > > Thanks, > -Hanu
[ANNOUNCE] New Committer: Hanumath Rao Maduri
The Project Management Committee (PMC) for Apache Drill has invited Hanumath Rao Maduri to become a committer, and we are pleased to announce that he has accepted. Hanumath became a contributor in 2017, making changes mostly in the Drill planning side, including lateral / unnest support. He is also one of the contributors of index based planning and execution support. Welcome Hanumath, and thank you for your contributions! - Arina (on behalf of Drill PMC)
[GitHub] vvysotskyi commented on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
vvysotskyi commented on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM URL: https://github.com/apache/drill/pull/1446#issuecomment-435030809 @oleg-zinovev, also there is an issue with `GuavaPatcher`. When Drill is built under JDK 10+, after starting in embedded mode the next error is seen in the logs: ``` 2018-11-01 14:26:12,694 [main] WARN o.a.drill.exec.util.GuavaPatcher - Unable to patch Guava classes. java.lang.NullPointerException: null at javassist.util.proxy.SecurityActions.setAccessible(SecurityActions.java:103) ~[javassist-3.22.0-GA.jar:na] at javassist.util.proxy.DefineClassHelper.toClass3(DefineClassHelper.java:151) ~[javassist-3.22.0-GA.jar:na] at javassist.util.proxy.DefineClassHelper.toClass2(DefineClassHelper.java:134) ~[javassist-3.22.0-GA.jar:na] at javassist.util.proxy.DefineClassHelper.toClass(DefineClassHelper.java:95) ~[javassist-3.22.0-GA.jar:na] at javassist.ClassPool.toClass(ClassPool.java:1143) ~[javassist-3.22.0-GA.jar:na] at javassist.ClassPool.toClass(ClassPool.java:1106) ~[javassist-3.22.0-GA.jar:na] at javassist.ClassPool.toClass(ClassPool.java:1064) ~[javassist-3.22.0-GA.jar:na] at javassist.CtClass.toClass(CtClass.java:1275) ~[javassist-3.22.0-GA.jar:na] at org.apache.drill.exec.util.GuavaPatcher.patchStopwatch(GuavaPatcher.java:66) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT] at org.apache.drill.exec.util.GuavaPatcher.patch(GuavaPatcher.java:36) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT] at org.apache.drill.exec.server.Drillbit.(Drillbit.java:72) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT] at org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:129) ~[drill-jdbc-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT] at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67) ~[drill-jdbc-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT] at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:68) ~[drill-jdbc-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT] at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138) ~[avatica-1.12.0.jar:1.12.0] at org.apache.drill.jdbc.Driver.connect(Driver.java:72) ~[drill-jdbc-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT] at sqlline.DatabaseConnection.connect(DatabaseConnection.java:154) ~[sqlline-1.5.0.jar:1.5.0] at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:202) ~[sqlline-1.5.0.jar:1.5.0] at sqlline.Commands.connect(Commands.java:1165) ~[sqlline-1.5.0.jar:1.5.0] at sqlline.Commands.connect(Commands.java:1055) ~[sqlline-1.5.0.jar:1.5.0] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na] at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na] at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38) ~[sqlline-1.5.0.jar:1.5.0] at sqlline.SqlLine.dispatch(SqlLine.java:643) ~[sqlline-1.5.0.jar:1.5.0] at sqlline.SqlLine.initArgs(SqlLine.java:385) ~[sqlline-1.5.0.jar:1.5.0] at sqlline.SqlLine.begin(SqlLine.java:485) ~[sqlline-1.5.0.jar:1.5.0] at sqlline.SqlLine.start(SqlLine.java:233) ~[sqlline-1.5.0.jar:1.5.0] at sqlline.SqlLine.main(SqlLine.java:175) ~[sqlline-1.5.0.jar:1.5.0] ``` Could you please take a look at this? Looks loke it is connected with current `javassist` version. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] vvysotskyi commented on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
vvysotskyi commented on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM URL: https://github.com/apache/drill/pull/1446#issuecomment-435024088 @oleg-zinovev, thanks for fixing this. I think it is ok to exclude this library from the shade, since there are no other transitive dependencies on this, and as you pointed, the single class, where it is used was also excluded. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM URL: https://github.com/apache/drill/pull/1446#issuecomment-435013897 @vvysotskyi please, review my changes in jdbc-all pom.xml I had to add `com.jasonclawson` into relocation, because drill start fails with HoconFactory to JsonFactory cast error in StoragePluginsHandlerService P.S. Maybe com.jasonclawson can be excluded from shade? It is used only in StoragePluginsHandlerService located in org.apache.drill.exec.store package and this package excluded from shaded artifact. FYI. https://stackoverflow.com/questions/53010200/maven-surefire-could-not-find-forkedbooter-class/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM URL: https://github.com/apache/drill/pull/1446#issuecomment-435013897 @vvysotskyi please, review my changes in jdbc-all pom.xml I had to add `com.jasonclawson` into relocation, because drill start fails with HoconFactory to JsonFactory cast error in StoragePluginsHandlerService P.S. Maybe com.jasonclawson can be excluded from shade? It is used only in StoragePluginsHandlerService located in org.apache.drill.exec.store package and this package excluded from shaded artifact. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] oleg-zinovev commented on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
oleg-zinovev commented on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM URL: https://github.com/apache/drill/pull/1446#issuecomment-435013897 @vvysotskyi please, review my changes in jdbc-all pom.xml I had to add `com.jasonclawson` into relocation, because drill start fails with HoconFactory to JsonFactory cast error in StoragePluginsHandlerService This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (DRILL-6825) Applying different hash function according to data types and data size
weijie.tong created DRILL-6825: -- Summary: Applying different hash function according to data types and data size Key: DRILL-6825 URL: https://issues.apache.org/jira/browse/DRILL-6825 Project: Apache Drill Issue Type: Improvement Components: Execution - Codegen Reporter: weijie.tong Fix For: 1.16.0 Different hash functions have different performance according to different data types and data size. We should choose a right one to apply not just Murmurhash. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
