[jira] [Commented] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671093#comment-16671093
]
shuifeng lu commented on DRILL-6722:
May I ask again about saving class to file?
I can't find the right way to write bytes into file in Evaluate expression
pop-up, did I miss something?
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671065#comment-16671065 ] ASF GitHub Bot commented on DRILL-6792: --- weijietong commented on a change in pull request #1504: DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf… URL: https://github.com/apache/drill/pull/1504#discussion_r229929618 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java ## @@ -1222,7 +1222,7 @@ public HashJoinBatch(HashJoinPOP popConfig, FragmentContext context, RecordBatchStats.logRecordBatchStats(getRecordBatchStatsContext(), "configured output batch size: %d", configuredBatchSize); -enableRuntimeFilter = context.getOptions().getOption(ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER); +enableRuntimeFilter = context.getOptions().getOption(ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER) && popConfig.getRuntimeFilterDef() != null; Review comment: If the popConfig doesn’t hold a runtimeFilterDef,then this record batch should not do anything related to RF,like code generation. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Find the right probe side fragment to any storage plugin > > > Key: DRILL-6792 > URL: https://issues.apache.org/jira/browse/DRILL-6792 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Fix For: 1.15.0 > > > The current implementation of JPPD to find the probe side wrapper depends on > the GroupScan's digest. But there's no promise the GroupScan's digest will > not be changed since it is attached to the RuntimeFilterDef by different > storage plugin implementation logic.So here we assign a unique identifier to > the RuntimeFilter operator, and find the right probe side fragment wrapper by > the runtime filter identifier at the RuntimeFilterRouter class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671062#comment-16671062
]
ASF GitHub Bot commented on DRILL-6792:
---
weijietong commented on a change in pull request #1504: DRILL-6792: Find the
right probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229929162
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterRouter.java
##
@@ -125,33 +128,53 @@ public void waitForComplete() {
/**
* This method is passively invoked by receiving a runtime filter from the
network
- * @param runtimeFilterWritable
+ *
+ * @param srcRuntimeFilterWritable
*/
- public void registerRuntimeFilter(RuntimeFilterWritable
runtimeFilterWritable) {
-broadcastAggregatedRuntimeFilter(runtimeFilterWritable);
+ public void register(RuntimeFilterWritable srcRuntimeFilterWritable) {
+BitData.RuntimeFilterBDef runtimeFilterB =
srcRuntimeFilterWritable.getRuntimeFilterBDef();
+int joinMajorId = runtimeFilterB.getMajorFragmentId();
+int buildSideRfNumber;
+RuntimeFilterWritable toAggregated;
+synchronized (this) {
+ buildSideRfNumber = joinMjId2rfNumber.get(joinMajorId);
+ buildSideRfNumber--;
+ joinMjId2rfNumber.put(joinMajorId, buildSideRfNumber);
+ toAggregated = joinMjId2AggregatedRF.get(joinMajorId);
+ if (toAggregated == null) {
+toAggregated = srcRuntimeFilterWritable;
+toAggregated.retainBuffers(1);
Review comment:
The received RF will be held till a fixed number of RFs aggregating to send
out.If we don’t retain the first one,the caller will release it at the final
block, and the held to be used later RF ‘s reference number will be zero.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671060#comment-16671060
]
ASF GitHub Bot commented on DRILL-6792:
---
weijietong commented on a change in pull request #1504: DRILL-6792: Find the
right probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229928504
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterWritable.java
##
@@ -103,6 +103,27 @@ public RuntimeFilterWritable duplicate(BufferAllocator
bufferAllocator) {
return new RuntimeFilterWritable(runtimeFilterBDef, cloned);
}
+ public void retainBuffers(final int increment) {
+if (increment <= 0) {
+ return;
+}
+for (final DrillBuf buf : data) {
+ buf.retain(increment);
+}
+ }
+
+ public RuntimeFilterWritable newRuntimeFilterWritable(BufferAllocator
bufferAllocator) {
Review comment:
Ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671059#comment-16671059
]
ASF GitHub Bot commented on DRILL-6792:
---
weijietong commented on a change in pull request #1504: DRILL-6792: Find the
right probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229928054
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterRouter.java
##
@@ -125,33 +128,53 @@ public void waitForComplete() {
/**
* This method is passively invoked by receiving a runtime filter from the
network
- * @param runtimeFilterWritable
+ *
+ * @param srcRuntimeFilterWritable
*/
- public void registerRuntimeFilter(RuntimeFilterWritable
runtimeFilterWritable) {
-broadcastAggregatedRuntimeFilter(runtimeFilterWritable);
+ public void register(RuntimeFilterWritable srcRuntimeFilterWritable) {
+BitData.RuntimeFilterBDef runtimeFilterB =
srcRuntimeFilterWritable.getRuntimeFilterBDef();
+int joinMajorId = runtimeFilterB.getMajorFragmentId();
+int buildSideRfNumber;
+RuntimeFilterWritable toAggregated;
+synchronized (this) {
+ buildSideRfNumber = joinMjId2rfNumber.get(joinMajorId);
+ buildSideRfNumber--;
+ joinMjId2rfNumber.put(joinMajorId, buildSideRfNumber);
+ toAggregated = joinMjId2AggregatedRF.get(joinMajorId);
+ if (toAggregated == null) {
+toAggregated = srcRuntimeFilterWritable;
+toAggregated.retainBuffers(1);
+ } else {
+toAggregated.aggregate(srcRuntimeFilterWritable);
+ }
+ joinMjId2AggregatedRF.put(joinMajorId, toAggregated);
+}
+if (buildSideRfNumber == 0) {
Review comment:
I think this local stack variable doesn’t need to be in the blocking
nodes.It’s safe to check it. I don’t want the broadcasting work happened at
this blocking boundary.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671057#comment-16671057
]
ASF GitHub Bot commented on DRILL-6792:
---
weijietong commented on a change in pull request #1504: DRILL-6792: Find the
right probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229928054
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterRouter.java
##
@@ -125,33 +128,53 @@ public void waitForComplete() {
/**
* This method is passively invoked by receiving a runtime filter from the
network
- * @param runtimeFilterWritable
+ *
+ * @param srcRuntimeFilterWritable
*/
- public void registerRuntimeFilter(RuntimeFilterWritable
runtimeFilterWritable) {
-broadcastAggregatedRuntimeFilter(runtimeFilterWritable);
+ public void register(RuntimeFilterWritable srcRuntimeFilterWritable) {
+BitData.RuntimeFilterBDef runtimeFilterB =
srcRuntimeFilterWritable.getRuntimeFilterBDef();
+int joinMajorId = runtimeFilterB.getMajorFragmentId();
+int buildSideRfNumber;
+RuntimeFilterWritable toAggregated;
+synchronized (this) {
+ buildSideRfNumber = joinMjId2rfNumber.get(joinMajorId);
+ buildSideRfNumber--;
+ joinMjId2rfNumber.put(joinMajorId, buildSideRfNumber);
+ toAggregated = joinMjId2AggregatedRF.get(joinMajorId);
+ if (toAggregated == null) {
+toAggregated = srcRuntimeFilterWritable;
+toAggregated.retainBuffers(1);
+ } else {
+toAggregated.aggregate(srcRuntimeFilterWritable);
+ }
+ joinMjId2AggregatedRF.put(joinMajorId, toAggregated);
+}
+if (buildSideRfNumber == 0) {
Review comment:
I think the local stack
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671054#comment-16671054
]
ASF GitHub Bot commented on DRILL-6792:
---
weijietong commented on a change in pull request #1504: DRILL-6792: Find the
right probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229927582
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/filter/RuntimeFilterRecordBatch.java
##
@@ -263,4 +260,41 @@ public void dump() {
+ "originalRecordCount={}, batchSchema={}]",
container, sv2, toFilterFields, originalRecordCount,
incoming.getSchema());
}
+
+ public enum Metric implements MetricDef {
+FILTERED_ROWS, APPLIED_TIMES;
+
+@Override
+public int metricId() {
+ return ordinal();
+}
+ }
+
+ public void updateStats() {
+stats.setLongStat(Metric.FILTERED_ROWS, filteredRows);
+stats.setLongStat(Metric.APPLIED_TIMES, appliedTimes);
+ }
+
+ private void timedWaiting() {
+if (!enableRFWaiting || waited) {
+ return;
+}
+long startMs = System.currentTimeMillis();
+while (current == null && batchTimes > 0) {
Review comment:
Since the HashJoin will fetch the first batch from both sides first, we
should let the first probe side batch pass.
will refactor the waiting logic by the condition.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6760) The Profiles page does not retain the original exception if the query fails
[ https://issues.apache.org/jira/browse/DRILL-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671041#comment-16671041 ] ASF GitHub Bot commented on DRILL-6760: --- lushuifeng opened a new pull request #1519: DRILL-6760: Retain original exception in Verbose Error Message URL: https://github.com/apache/drill/pull/1519 please see [DRILL-6760](https://issues.apache.org/jira/browse/DRILL-6760) There are some cases that verbose error message in profile page contains no original exception stack trace. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > The Profiles page does not retain the original exception if the query fails > --- > > Key: DRILL-6760 > URL: https://issues.apache.org/jira/browse/DRILL-6760 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.14.0 >Reporter: Nitin Sharma >Priority: Major > Labels: errorhandling > > The profiles page (that shows query execution for the most recently issued > queries) on the foreman node does not retain the original exception if the > query failed. The error always says "Remote Drill Exception" and gives an id. > The actual exception could be anything from OOM, s3 access errors, out of > disk issue etc but they seem to get swallowed. > > It is hard to debug if there are long running queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6824) Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the only distinct Values are NULL and Text
[ https://issues.apache.org/jira/browse/DRILL-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670963#comment-16670963 ] ASF GitHub Bot commented on DRILL-6824: --- amansinha100 commented on issue #1518: DRILL-6824: Handle schema changes in MapRDBJsonRecordReader URL: https://github.com/apache/drill/pull/1518#issuecomment-434894435 Based on the description in the JIRA, the changes lgtm. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the > only distinct Values are NULL and Text > > > Key: DRILL-6824 > URL: https://issues.apache.org/jira/browse/DRILL-6824 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0, 1.15.0 >Reporter: Gautam Parai >Assignee: Gautam Parai >Priority: Major > Fix For: 1.15.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Drill Query on MapR DB JSON Table or it View fails with a > SchemaChangeException. The only distinct values is NULL and some Text. > The issue is that when Drill sees a NULL or does not see any values for a > column in the first batch it assumes the type as NULLABLE INT. Subsequently, > when the data shows up and it is different from NULLABLE INT there is a > schema change. Operators e.g. aggregators etc. cannot handle such a Schema > Change and throw a SchemaChangeException. > > One of the short-term solution implemented in this fix: Add a CAST expression > which will cast null values to the target type. Hence, we would never see a > SchemaChange due to NULLs. However, the MapRDB Reader code was written > differently than other reader and was hitting a SchemaChangeException. The > code was changed to make it similar to other scans i.e. emit a new batch > whenever a schema change is encountered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670947#comment-16670947
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229761029
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/filter/RuntimeFilterRecordBatch.java
##
@@ -263,4 +260,41 @@ public void dump() {
+ "originalRecordCount={}, batchSchema={}]",
container, sv2, toFilterFields, originalRecordCount,
incoming.getSchema());
}
+
+ public enum Metric implements MetricDef {
+FILTERED_ROWS, APPLIED_TIMES;
+
+@Override
+public int metricId() {
+ return ordinal();
+}
+ }
+
+ public void updateStats() {
+stats.setLongStat(Metric.FILTERED_ROWS, filteredRows);
+stats.setLongStat(Metric.APPLIED_TIMES, appliedTimes);
+ }
+
+ private void timedWaiting() {
+if (!enableRFWaiting || waited) {
+ return;
+}
+long startMs = System.currentTimeMillis();
+while (current == null && batchTimes > 0) {
Review comment:
What's the reason of waiting after processing first batch ? Why not wait
just on condition **(current==null)** ?
Also it looks more like the design of this wait should be done using a
conditional variable. Where FragmentContext will signal the conditional
variable once Filter is set and this minor fragment thread will wait for
timeout or signal on that conditional variable. See
[here](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/Condition.html#await-long-java.util.concurrent.TimeUnit-)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670949#comment-16670949
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229906462
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterRouter.java
##
@@ -125,33 +128,53 @@ public void waitForComplete() {
/**
* This method is passively invoked by receiving a runtime filter from the
network
- * @param runtimeFilterWritable
+ *
+ * @param srcRuntimeFilterWritable
*/
- public void registerRuntimeFilter(RuntimeFilterWritable
runtimeFilterWritable) {
-broadcastAggregatedRuntimeFilter(runtimeFilterWritable);
+ public void register(RuntimeFilterWritable srcRuntimeFilterWritable) {
+BitData.RuntimeFilterBDef runtimeFilterB =
srcRuntimeFilterWritable.getRuntimeFilterBDef();
+int joinMajorId = runtimeFilterB.getMajorFragmentId();
+int buildSideRfNumber;
+RuntimeFilterWritable toAggregated;
+synchronized (this) {
+ buildSideRfNumber = joinMjId2rfNumber.get(joinMajorId);
+ buildSideRfNumber--;
+ joinMjId2rfNumber.put(joinMajorId, buildSideRfNumber);
+ toAggregated = joinMjId2AggregatedRF.get(joinMajorId);
+ if (toAggregated == null) {
+toAggregated = srcRuntimeFilterWritable;
+toAggregated.retainBuffers(1);
+ } else {
+toAggregated.aggregate(srcRuntimeFilterWritable);
+ }
+ joinMjId2AggregatedRF.put(joinMajorId, toAggregated);
+}
+if (buildSideRfNumber == 0) {
Review comment:
this check needs to happen in `synchronized` block as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670942#comment-16670942
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229754301
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MurmurHash3.java
##
@@ -271,6 +270,5 @@ public static int hash32(double val, long seed) {
public static int hash32(int start, int end, DrillBuf buffer, int seed) {
return murmur3_32(start, end, buffer, seed);
}
-
}
Review comment:
please remove this file from your commits. there is only extra space added
here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670952#comment-16670952
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229909442
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterSink.java
##
@@ -115,7 +98,8 @@ public void aggregate(RuntimeFilterWritable
runtimeFilterWritable) {
public RuntimeFilterWritable fetchLatestDuplicatedAggregatedOne() {
try {
aggregatedRFLock.lock();
- return aggregated.duplicate(bufferAllocator);
+ RuntimeFilterWritable duplicated = aggregated.duplicate(bufferAllocator);
Review comment:
This class is not required anymore. It's only used in tests where we should
replace the reference with correct implementation. Basically update the test
implementation of [this
method](https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java#L172)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670946#comment-16670946
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229909546
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterWritable.java
##
@@ -103,6 +103,27 @@ public RuntimeFilterWritable duplicate(BufferAllocator
bufferAllocator) {
return new RuntimeFilterWritable(runtimeFilterBDef, cloned);
}
+ public void retainBuffers(final int increment) {
+if (increment <= 0) {
+ return;
+}
+for (final DrillBuf buf : data) {
+ buf.retain(increment);
+}
+ }
+
+ public RuntimeFilterWritable newRuntimeFilterWritable(BufferAllocator
bufferAllocator) {
Review comment:
Please add a TODO comment saying: Not used currently because of
reference the JIRA number which you will create with explanation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670950#comment-16670950
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229908466
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterRouter.java
##
@@ -125,33 +128,53 @@ public void waitForComplete() {
/**
* This method is passively invoked by receiving a runtime filter from the
network
- * @param runtimeFilterWritable
+ *
+ * @param srcRuntimeFilterWritable
*/
- public void registerRuntimeFilter(RuntimeFilterWritable
runtimeFilterWritable) {
-broadcastAggregatedRuntimeFilter(runtimeFilterWritable);
+ public void register(RuntimeFilterWritable srcRuntimeFilterWritable) {
+BitData.RuntimeFilterBDef runtimeFilterB =
srcRuntimeFilterWritable.getRuntimeFilterBDef();
+int joinMajorId = runtimeFilterB.getMajorFragmentId();
+int buildSideRfNumber;
+RuntimeFilterWritable toAggregated;
+synchronized (this) {
+ buildSideRfNumber = joinMjId2rfNumber.get(joinMajorId);
+ buildSideRfNumber--;
+ joinMjId2rfNumber.put(joinMajorId, buildSideRfNumber);
+ toAggregated = joinMjId2AggregatedRF.get(joinMajorId);
+ if (toAggregated == null) {
+toAggregated = srcRuntimeFilterWritable;
+toAggregated.retainBuffers(1);
Review comment:
why we need to `retainBuffers` here again ? Caller is already retaining this
buffer once.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670944#comment-16670944
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229908821
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java
##
@@ -379,11 +379,16 @@ public FragmentExecutor getFragmentRunner(final
FragmentHandle handle) {
return runningFragments.get(handle);
}
+/**
+ * receive the RuntimeFilter thorough the wire
+ * @param runtimeFilter
+ */
public void receiveRuntimeFilter(final RuntimeFilterWritable
runtimeFilter) {
BitData.RuntimeFilterBDef runtimeFilterDef =
runtimeFilter.getRuntimeFilterBDef();
boolean toForeman = runtimeFilterDef.getToForeman();
QueryId queryId = runtimeFilterDef.getQueryId();
String queryIdStr = QueryIdHelper.getQueryId(queryId);
+ runtimeFilter.retainBuffers(1);
Review comment:
Hmm initially I didn't looked from close() call perspective. But even with
respect to that it doesn't look like there will be any lost Ack case specially
just with the change for transfer the buffer. It looks like more of a race
condition somewhere to me. Can you open a JIRA for this issue - that we should
transfer the buffers and look into about the hang issue ? So that post this PR
we can look into that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670945#comment-16670945
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229765087
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/RuntimeFilterVisitor.java
##
@@ -134,8 +142,15 @@ private RuntimeFilterDef
generateRuntimeFilter(HashJoinPrel hashJoinPrel) {
List bloomFilterDefs = new ArrayList<>();
//find the possible left scan node of the left join key
-GroupScan groupScan = null;
+ScanPrel probeSideScanPrel = null;
RelNode left = hashJoinPrel.getLeft();
+RelNode right = hashJoinPrel.getRight();
+ExchangePrel exchangePrel = findRightExchangePrel(right);
+if (exchangePrel == null) {
+ //Does not support the single fragment mode ,that is the right build side
+ //can only be BroadcastExchangePrel or HashToRandomExchangePrel
+ return null;
+}
List leftFields = left.getRowType().getFieldNames();
List leftKeys = hashJoinPrel.getLeftKeys();
RelMetadataQuery metadataQuery = left.getCluster().getMetadataQuery();
Review comment:
Yes that was the point why ScanPrel is found for all left keys. Loop should
break once we found a ScanPrel for any left key.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670948#comment-16670948 ] ASF GitHub Bot commented on DRILL-6792: --- sohami commented on a change in pull request #1504: DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf… URL: https://github.com/apache/drill/pull/1504#discussion_r229763641 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java ## @@ -1222,7 +1222,7 @@ public HashJoinBatch(HashJoinPOP popConfig, FragmentContext context, RecordBatchStats.logRecordBatchStats(getRecordBatchStatsContext(), "configured output batch size: %d", configuredBatchSize); -enableRuntimeFilter = context.getOptions().getOption(ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER); +enableRuntimeFilter = context.getOptions().getOption(ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER) && popConfig.getRuntimeFilterDef() != null; Review comment: Is this an error condition when the feature is `enabled` but `runtimeFilterDef` is not present in popConfig ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Find the right probe side fragment to any storage plugin > > > Key: DRILL-6792 > URL: https://issues.apache.org/jira/browse/DRILL-6792 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Fix For: 1.15.0 > > > The current implementation of JPPD to find the probe side wrapper depends on > the GroupScan's digest. But there's no promise the GroupScan's digest will > not be changed since it is attached to the RuntimeFilterDef by different > storage plugin implementation logic.So here we assign a unique identifier to > the RuntimeFilter operator, and find the right probe side fragment wrapper by > the runtime filter identifier at the RuntimeFilterRouter class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670943#comment-16670943
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229909632
##
File path:
exec/java-exec/src/test/java/org/apache/drill/test/PhysicalOpUnitTestBase.java
##
@@ -310,6 +310,11 @@ public void addRuntimeFilter(RuntimeFilterWritable
runtimeFilter) {
this.runtimeFilterSink.aggregate(runtimeFilter);
Review comment:
Update this method as RuntimeFilter Sink is not used anymore.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670951#comment-16670951
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229908076
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterRouter.java
##
@@ -125,33 +128,53 @@ public void waitForComplete() {
/**
* This method is passively invoked by receiving a runtime filter from the
network
- * @param runtimeFilterWritable
+ *
+ * @param srcRuntimeFilterWritable
*/
- public void registerRuntimeFilter(RuntimeFilterWritable
runtimeFilterWritable) {
-broadcastAggregatedRuntimeFilter(runtimeFilterWritable);
+ public void register(RuntimeFilterWritable srcRuntimeFilterWritable) {
+BitData.RuntimeFilterBDef runtimeFilterB =
srcRuntimeFilterWritable.getRuntimeFilterBDef();
+int joinMajorId = runtimeFilterB.getMajorFragmentId();
+int buildSideRfNumber;
+RuntimeFilterWritable toAggregated;
+synchronized (this) {
+ buildSideRfNumber = joinMjId2rfNumber.get(joinMajorId);
+ buildSideRfNumber--;
+ joinMjId2rfNumber.put(joinMajorId, buildSideRfNumber);
+ toAggregated = joinMjId2AggregatedRF.get(joinMajorId);
+ if (toAggregated == null) {
+toAggregated = srcRuntimeFilterWritable;
+toAggregated.retainBuffers(1);
+ } else {
+toAggregated.aggregate(srcRuntimeFilterWritable);
+ }
+ joinMjId2AggregatedRF.put(joinMajorId, toAggregated);
+}
+if (buildSideRfNumber == 0) {
+ joinMjId2AggregatedRF.remove(joinMajorId);
+ route(toAggregated);
+}
}
-
- private void broadcastAggregatedRuntimeFilter(RuntimeFilterWritable
srcRuntimeFilterWritable) {
+ private void route(RuntimeFilterWritable srcRuntimeFilterWritable) {
BitData.RuntimeFilterBDef runtimeFilterB =
srcRuntimeFilterWritable.getRuntimeFilterBDef();
int joinMajorId = runtimeFilterB.getMajorFragmentId();
UserBitShared.QueryId queryId = runtimeFilterB.getQueryId();
List probeFields = runtimeFilterB.getProbeFieldsList();
+List sizeInBytes = runtimeFilterB.getBloomFilterSizeInBytesList();
DrillBuf[] data = srcRuntimeFilterWritable.getData();
-List scanNodeEps =
joinMjId2probdeScanEps.get(joinMajorId);
+List scanNodeEps =
joinMjId2probeScanEps.get(joinMajorId);
+int scanNodeSize = scanNodeEps.size();
+srcRuntimeFilterWritable.retainBuffers(scanNodeSize - 1);
int scanNodeMjId = joinMjId2ScanMjId.get(joinMajorId);
for (int minorId = 0; minorId < scanNodeEps.size(); minorId++) {
BitData.RuntimeFilterBDef.Builder builder =
BitData.RuntimeFilterBDef.newBuilder();
for (String probeField : probeFields) {
builder.addProbeFields(probeField);
}
- BitData.RuntimeFilterBDef runtimeFilterBDef = builder
-.setQueryId(queryId)
-.setMajorFragmentId(scanNodeMjId)
-.setMinorFragmentId(minorId)
-.build();
+ BitData.RuntimeFilterBDef runtimeFilterBDef =
builder.setQueryId(queryId).setMajorFragmentId(scanNodeMjId).setMinorFragmentId(minorId).setToForeman(false).addAllBloomFilterSizeInBytes(sizeInBytes).build();
Review comment:
please fix the indentation here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6824) Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the only distinct Values are NULL and Text
[ https://issues.apache.org/jira/browse/DRILL-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gautam Parai updated DRILL-6824: Description: Drill Query on MapR DB JSON Table or it View fails with a SchemaChangeException. The only distinct values is NULL and some Text. The issue is that when Drill sees a NULL or does not see any values for a column in the first batch it assumes the type as NULLABLE INT. Subsequently, when the data shows up and it is different from NULLABLE INT there is a schema change. Operators e.g. aggregators etc. cannot handle such a Schema Change and throw a SchemaChangeException. One of the short-term solution implemented in this fix: Add a CAST expression which will cast null values to the target type. Hence, we would never see a SchemaChange due to NULLs. However, the MapRDB Reader code was written differently than other reader and was hitting a SchemaChangeException. The code was changed to make it similar to other scans i.e. emit a new batch whenever a schema change is encountered. was:Drill Query on MapR DB JSON Table or it View fails with a SchemaChangeException. Th only distinct values is NULL and some Text. > Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the > only distinct Values are NULL and Text > > > Key: DRILL-6824 > URL: https://issues.apache.org/jira/browse/DRILL-6824 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0, 1.15.0 >Reporter: Gautam Parai >Assignee: Gautam Parai >Priority: Major > Fix For: 1.15.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Drill Query on MapR DB JSON Table or it View fails with a > SchemaChangeException. The only distinct values is NULL and some Text. > The issue is that when Drill sees a NULL or does not see any values for a > column in the first batch it assumes the type as NULLABLE INT. Subsequently, > when the data shows up and it is different from NULLABLE INT there is a > schema change. Operators e.g. aggregators etc. cannot handle such a Schema > Change and throw a SchemaChangeException. > > One of the short-term solution implemented in this fix: Add a CAST expression > which will cast null values to the target type. Hence, we would never see a > SchemaChange due to NULLs. However, the MapRDB Reader code was written > differently than other reader and was hitting a SchemaChangeException. The > code was changed to make it similar to other scans i.e. emit a new batch > whenever a schema change is encountered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6822) Fix imports for DRILL-6381
[ https://issues.apache.org/jira/browse/DRILL-6822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Jyothsna Donapati updated DRILL-6822: - Labels: ready-to-commit (was: ) > Fix imports for DRILL-6381 > -- > > Key: DRILL-6822 > URL: https://issues.apache.org/jira/browse/DRILL-6822 > Project: Apache Drill > Issue Type: Task >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Major > Labels: ready-to-commit > Fix For: 1.15.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6824) Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the only distinct Values are NULL and Text
[ https://issues.apache.org/jira/browse/DRILL-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670826#comment-16670826 ] ASF GitHub Bot commented on DRILL-6824: --- amansinha100 commented on issue #1518: DRILL-6824: Handle schema changes in MapRDBJsonRecordReader URL: https://github.com/apache/drill/pull/1518#issuecomment-434864102 @gparai can you describe the issue ? The JIRA does not have much details. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the > only distinct Values are NULL and Text > > > Key: DRILL-6824 > URL: https://issues.apache.org/jira/browse/DRILL-6824 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0, 1.15.0 >Reporter: Gautam Parai >Assignee: Gautam Parai >Priority: Major > Fix For: 1.15.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Drill Query on MapR DB JSON Table or it View fails with a > SchemaChangeException. Th only distinct values is NULL and some Text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
[
https://issues.apache.org/jira/browse/DRILL-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670756#comment-16670756
]
Khurram Faraaz commented on DRILL-6816:
---
Also reproducible from a single Thread
Stack trace for NPE
{noformat}
14:14:01.174 [pool-1-thread-1] INFO o.a.drill.exec.client.DrillClient - Foreman
drillbit is qa102-45
14:14:01.174 [pool-1-thread-1] INFO o.a.drill.exec.client.DrillClient -
Successfully connected to server qa102-45:31010
14:14:01.256 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@2e35015b
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.run(RunQuery.java:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
null
java.sql.SQLException
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:520)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.run(RunQuery.java:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
... 9 more
14:14:01.257 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Auto-closing (via open-statements registry):
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@2e35015b
14:14:01.257 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Removing from open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@2e35015b
14:14:01.258 [pool-1-thread-1] DEBUG o.apache.drill.exec.rpc.BasicClient -
Closing client
{noformat}
> NPE - Concurrent query execution using PreparedStatement
> -
>
> Key: DRILL-6816
> URL: https://issues.apache.org/jira/browse/DRILL-6816
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Priority: Major
>
> Concurrent query execution from JDBC program using PreparedStatement results
> in NPE.
> Queries that were executed concurrently are (part of a query file),
> {noformat}
> select id from `test_tbl.json`
> select count(id) from `test_tbl.json`
> select count(*) from `test_tbl.json`
> select * from `test_tbl.json`
> {noformat}
> Drill 1.14.0
> git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
> MapR version => 6.1.0.20180911143226.GA (secure cluster)
> JDBC driver used was org.apache.drill.jdbc.Driver
> Executing the above queries concurrently using a Statement object results in
> successful query execution.
> {noformat}
> Statement stmt = conn.createStatement();
> ResultSet rs = stmt.executeQuery(query);
> {noformat}
> However, when the same queries listed above are executed using a
>
[jira] [Commented] (DRILL-6610) Add support for Minimum TLS support
[
https://issues.apache.org/jira/browse/DRILL-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670659#comment-16670659
]
ASF GitHub Bot commented on DRILL-6610:
---
superbstreak commented on a change in pull request #1388: DRILL-6610: Add
support for Minimum TLS restriction.
URL: https://github.com/apache/drill/pull/1388#discussion_r229846007
##
File path: contrib/native/client/src/clientlib/channel.hpp
##
@@ -82,11 +82,31 @@ class UserProperties;
return boost::asio::ssl::context::tlsv11;
} else if (version == "tlsv1") {
return boost::asio::ssl::context::tlsv1;
+} else if ((version == "tlsv1+") || (version == "tlsv11+") ||
(version == "tlsv12+")) {
+// SSLv2 and SSLv3 are disabled, so this is the equivalent
of 'tls' only mode.
+// In boost version 1.64+, they've added support for
context::tls; method.
+return boost::asio::ssl::context::sslv23;
} else {
return boost::asio::ssl::context::tlsv12;
}
}
+/// @brief Applies Minimum TLS protocol restrictions.
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add support for Minimum TLS support
> ---
>
> Key: DRILL-6610
> URL: https://issues.apache.org/jira/browse/DRILL-6610
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - C++
>Affects Versions: 1.12.0
>Reporter: Rob Wu
>Assignee: Rob Wu
>Priority: Major
>
> Add support for minimum TLS support.
> Currently, the TLSProtocol parameter only supports a specific version of TLS
> to be used.
>
> Investigation:
> Setting the default SSL context method to be sslv23 with default sslv2 and
> sslv3 turned off would allow us to restrict the protocol to be TLS only.
> Additional flags can be applied to further restrict the minimum TLS version:
> For example:
> Minimum TLS 1.0 - Sets NO_SSLv2 and NO_SSLv3
> Minimum TLS 1.1 - Sets NO_SSLv2 and NO SSLv3 and NO_TLSv1
> Minimum TLS 1.2 - Sets NO_SSLv2 and NO SSLv3 and NO_TLSv1 and NO_TLSv1_1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6610) Add support for Minimum TLS support
[
https://issues.apache.org/jira/browse/DRILL-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670658#comment-16670658
]
ASF GitHub Bot commented on DRILL-6610:
---
superbstreak commented on a change in pull request #1388: DRILL-6610: Add
support for Minimum TLS restriction.
URL: https://github.com/apache/drill/pull/1388#discussion_r229845958
##
File path: contrib/native/client/src/clientlib/channel.hpp
##
@@ -82,11 +82,31 @@ class UserProperties;
return boost::asio::ssl::context::tlsv11;
} else if (version == "tlsv1") {
return boost::asio::ssl::context::tlsv1;
+} else if ((version == "tlsv1+") || (version == "tlsv11+") ||
(version == "tlsv12+")) {
+// SSLv2 and SSLv3 are disabled, so this is the equivalent
of 'tls' only mode.
+// In boost version 1.64+, they've added support for
context::tls; method.
+return boost::asio::ssl::context::sslv23;
} else {
return boost::asio::ssl::context::tlsv12;
}
}
+/// @brief Applies Minimum TLS protocol restrictions.
+///
+/// @param in_ver The protocol version.
+///
+/// @return The SSL context options.
+static long ApplyMinTLSRestriction(const std::string & in_ver){
+#if defined(IS_SSL_ENABLED)
+if (in_ver == "tlsv11+") {
+return SSL_OP_NO_TLSv1;
+} else if (in_ver == "tlsv12+") {
+return (SSL_OP_NO_TLSv1 | SSL_OP_NO_TLSv1_1);
+}
+#endif
+return 0;
Review comment:
Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add support for Minimum TLS support
> ---
>
> Key: DRILL-6610
> URL: https://issues.apache.org/jira/browse/DRILL-6610
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - C++
>Affects Versions: 1.12.0
>Reporter: Rob Wu
>Assignee: Rob Wu
>Priority: Major
>
> Add support for minimum TLS support.
> Currently, the TLSProtocol parameter only supports a specific version of TLS
> to be used.
>
> Investigation:
> Setting the default SSL context method to be sslv23 with default sslv2 and
> sslv3 turned off would allow us to restrict the protocol to be TLS only.
> Additional flags can be applied to further restrict the minimum TLS version:
> For example:
> Minimum TLS 1.0 - Sets NO_SSLv2 and NO_SSLv3
> Minimum TLS 1.1 - Sets NO_SSLv2 and NO SSLv3 and NO_TLSv1
> Minimum TLS 1.2 - Sets NO_SSLv2 and NO SSLv3 and NO_TLSv1 and NO_TLSv1_1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6610) Add support for Minimum TLS support
[
https://issues.apache.org/jira/browse/DRILL-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670657#comment-16670657
]
ASF GitHub Bot commented on DRILL-6610:
---
superbstreak commented on a change in pull request #1388: DRILL-6610: Add
support for Minimum TLS restriction.
URL: https://github.com/apache/drill/pull/1388#discussion_r229845756
##
File path: contrib/native/client/src/clientlib/channel.hpp
##
@@ -82,11 +82,31 @@ class UserProperties;
return boost::asio::ssl::context::tlsv11;
} else if (version == "tlsv1") {
return boost::asio::ssl::context::tlsv1;
+} else if ((version == "tlsv1+") || (version == "tlsv11+") ||
(version == "tlsv12+")) {
+// SSLv2 and SSLv3 are disabled, so this is the equivalent
of 'tls' only mode.
+// In boost version 1.64+, they've added support for
context::tls; method.
+return boost::asio::ssl::context::sslv23;
} else {
return boost::asio::ssl::context::tlsv12;
}
}
+/// @brief Applies Minimum TLS protocol restrictions.
+///
+/// @param in_ver The protocol version.
+///
+/// @return The SSL context options.
+static long ApplyMinTLSRestriction(const std::string & in_ver){
+#if defined(IS_SSL_ENABLED)
+if (in_ver == "tlsv11+") {
+return SSL_OP_NO_TLSv1;
+} else if (in_ver == "tlsv12+") {
+return (SSL_OP_NO_TLSv1 | SSL_OP_NO_TLSv1_1);
+}
Review comment:
I've added a note of that in the comments
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add support for Minimum TLS support
> ---
>
> Key: DRILL-6610
> URL: https://issues.apache.org/jira/browse/DRILL-6610
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - C++
>Affects Versions: 1.12.0
>Reporter: Rob Wu
>Assignee: Rob Wu
>Priority: Major
>
> Add support for minimum TLS support.
> Currently, the TLSProtocol parameter only supports a specific version of TLS
> to be used.
>
> Investigation:
> Setting the default SSL context method to be sslv23 with default sslv2 and
> sslv3 turned off would allow us to restrict the protocol to be TLS only.
> Additional flags can be applied to further restrict the minimum TLS version:
> For example:
> Minimum TLS 1.0 - Sets NO_SSLv2 and NO_SSLv3
> Minimum TLS 1.1 - Sets NO_SSLv2 and NO SSLv3 and NO_TLSv1
> Minimum TLS 1.2 - Sets NO_SSLv2 and NO SSLv3 and NO_TLSv1 and NO_TLSv1_1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6822) Fix imports for DRILL-6381
[ https://issues.apache.org/jira/browse/DRILL-6822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Sinha updated DRILL-6822: -- Fix Version/s: 1.15.0 > Fix imports for DRILL-6381 > -- > > Key: DRILL-6822 > URL: https://issues.apache.org/jira/browse/DRILL-6822 > Project: Apache Drill > Issue Type: Task >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.15.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6824) Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the only distinct Values are NULL and Text
[ https://issues.apache.org/jira/browse/DRILL-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670623#comment-16670623 ] ASF GitHub Bot commented on DRILL-6824: --- gparai opened a new pull request #1518: DRILL-6824: Handle schema changes in MapRDBJsonRecordReader URL: https://github.com/apache/drill/pull/1518 @amansinha100 can you please review the PR? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the > only distinct Values are NULL and Text > > > Key: DRILL-6824 > URL: https://issues.apache.org/jira/browse/DRILL-6824 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0, 1.15.0 >Reporter: Gautam Parai >Assignee: Gautam Parai >Priority: Major > Fix For: 1.15.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Drill Query on MapR DB JSON Table or it View fails with a > SchemaChangeException. Th only distinct values is NULL and some Text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6824) Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the only distinct Values are NULL and Text
Gautam Parai created DRILL-6824: --- Summary: Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the only distinct Values are NULL and Text Key: DRILL-6824 URL: https://issues.apache.org/jira/browse/DRILL-6824 Project: Apache Drill Issue Type: Bug Affects Versions: 1.14.0, 1.15.0 Reporter: Gautam Parai Assignee: Gautam Parai Fix For: 1.15.0 Drill Query on MapR DB JSON Table or it View fails with a SchemaChangeException. Th only distinct values is NULL and some Text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6823) Tests that require ZK client-specific configuration fail when they are not run standalone
[ https://issues.apache.org/jira/browse/DRILL-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthikeyan Manivannan updated DRILL-6823: -- Summary: Tests that require ZK client-specific configuration fail when they are not run standalone (was: Tests that require ZK client-specific configuration have to be run standalone) > Tests that require ZK client-specific configuration fail when they are not > run standalone > - > > Key: DRILL-6823 > URL: https://issues.apache.org/jira/browse/DRILL-6823 > Project: Apache Drill > Issue Type: Bug >Reporter: Karthikeyan Manivannan >Priority: Major > > ZK libraries only supports one client instance per-machine per-server and it > is cached. Tests that require client-specific configuration will fail when > run after other ZK tests that setup the client in a way that will cause this > test to fail. > Some investigation is necessary to see if the ZK ACL tests and any other such > tests, can be run standalone in our test framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6823) Tests that require ZK client-specific configuration have to be run standalone
[ https://issues.apache.org/jira/browse/DRILL-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthikeyan Manivannan updated DRILL-6823: -- Issue Type: Bug (was: Improvement) > Tests that require ZK client-specific configuration have to be run standalone > - > > Key: DRILL-6823 > URL: https://issues.apache.org/jira/browse/DRILL-6823 > Project: Apache Drill > Issue Type: Bug >Reporter: Karthikeyan Manivannan >Priority: Major > > ZK libraries only supports one client instance per-machine per-server and it > is cached. Tests that require client-specific configuration will fail when > run after other ZK tests that setup the client in a way that will cause this > test to fail. > Some investigation is necessary to see if the ZK ACL tests and any other such > tests, can be run standalone in our test framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6823) Tests that require ZK client-specific configuration have to be run standalone
Karthikeyan Manivannan created DRILL-6823: - Summary: Tests that require ZK client-specific configuration have to be run standalone Key: DRILL-6823 URL: https://issues.apache.org/jira/browse/DRILL-6823 Project: Apache Drill Issue Type: Improvement Reporter: Karthikeyan Manivannan ZK libraries only supports one client instance per-machine per-server and it is cached. Tests that require client-specific configuration will fail when run after other ZK tests that setup the client in a way that will cause this test to fail. Some investigation is necessary to see if the ZK ACL tests and any other such tests, can be run standalone in our test framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670400#comment-16670400
]
ASF GitHub Bot commented on DRILL-6349:
---
vvysotskyi commented on issue #1446: DRILL-6349: Drill JDBC driver fails on
Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434769786
These tests will be run after triggering install phase: `mvn clean install
-pl exec/jdbc-all`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)}}
> {{ at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)}}
> {{... 20 more}}
[jira] [Commented] (DRILL-6822) Fix imports for DRILL-6381
[ https://issues.apache.org/jira/browse/DRILL-6822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670399#comment-16670399 ] ASF GitHub Bot commented on DRILL-6822: --- dvjyothsna commented on issue #1517: DRILL-6822: Fixed imports for DRILL-6381 URL: https://github.com/apache/drill/pull/1517#issuecomment-434768757 @amansinha100 Please review. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix imports for DRILL-6381 > -- > > Key: DRILL-6822 > URL: https://issues.apache.org/jira/browse/DRILL-6822 > Project: Apache Drill > Issue Type: Task >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6822) Fix imports for DRILL-6381
[ https://issues.apache.org/jira/browse/DRILL-6822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670398#comment-16670398 ] ASF GitHub Bot commented on DRILL-6822: --- dvjyothsna opened a new pull request #1517: DRILL-6822: Fixed imports for DRILL-6381 URL: https://github.com/apache/drill/pull/1517 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix imports for DRILL-6381 > -- > > Key: DRILL-6822 > URL: https://issues.apache.org/jira/browse/DRILL-6822 > Project: Apache Drill > Issue Type: Task >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6381) Add capability to do index based planning and execution
[
https://issues.apache.org/jira/browse/DRILL-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670397#comment-16670397
]
ASF GitHub Bot commented on DRILL-6381:
---
dvjyothsna closed pull request #1516: Fixed imports for DRILL-6381
URL: https://github.com/apache/drill/pull/1516
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDescriptor.java
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDescriptor.java
index 75e6bc23973..ae386ab93a0 100644
---
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDescriptor.java
+++
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDescriptor.java
@@ -22,8 +22,8 @@
import java.util.List;
import java.util.Set;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Sets;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+import org.apache.drill.shaded.guava.com.google.common.collect.Sets;
import org.apache.calcite.plan.RelOptCost;
import org.apache.calcite.plan.RelOptPlanner;
@@ -40,8 +40,8 @@
import org.apache.drill.exec.util.EncodedSchemaPathSet;
import org.apache.drill.common.expression.LogicalExpression;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableSet;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
public class MapRDBIndexDescriptor extends DrillIndexDescriptor {
diff --git
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDiscover.java
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDiscover.java
index aed3e045a02..f828ba02daf 100644
---
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDiscover.java
+++
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBIndexDiscover.java
@@ -18,7 +18,7 @@
package org.apache.drill.exec.planner.index;
-import com.google.common.collect.Maps;
+import org.apache.drill.shaded.guava.com.google.common.collect.Maps;
import com.mapr.db.Admin;
import com.mapr.db.MapRDB;
import com.mapr.db.exceptions.DBException;
diff --git
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBStatistics.java
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBStatistics.java
index e129b968bf7..6fedaffd092 100644
---
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBStatistics.java
+++
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/planner/index/MapRDBStatistics.java
@@ -17,11 +17,11 @@
*/
package org.apache.drill.exec.planner.index;
-import com.google.common.base.Charsets;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
+import org.apache.drill.shaded.guava.com.google.common.base.Charsets;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
+import org.apache.drill.shaded.guava.com.google.common.collect.Maps;
import org.apache.calcite.plan.RelOptUtil;
import org.apache.calcite.rel.RelNode;
import org.apache.calcite.rel.metadata.RelMdUtil;
diff --git
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBTableCache.java
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBTableCache.java
index f35a4c41668..924d9c0e8e7 100644
---
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBTableCache.java
+++
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBTableCache.java
@@ -17,11 +17,11 @@
*/
package org.apache.drill.exec.store.mapr.db;
-import com.google.common.cache.CacheBuilder;
-import com.google.common.cache.CacheLoader;
-import com.google.common.cache.LoadingCache;
-import com.google.common.cache.RemovalListener;
-import com.google.common.cache.RemovalNotification;
+import org.apache.drill.shaded.guava.com.google.common.cache.CacheBuilder;
+import org.apache.drill.shaded.guava.com.google.common.cache.CacheLoader;
+import org.apache.drill.shaded.guava.com.google.common.cache.LoadingCache;
+import org.apache.drill.shaded.guava.com.google.common.cache.RemovalListener;
+import
org.apache.drill.shaded.guava.com.google.common.cache.RemovalNotification;
import com.mapr.db.Table;
import com.mapr.db.impl.MapRDBImpl;
[jira] [Created] (DRILL-6822) Fix imports for DRILL-6381
Venkata Jyothsna Donapati created DRILL-6822: Summary: Fix imports for DRILL-6381 Key: DRILL-6822 URL: https://issues.apache.org/jira/browse/DRILL-6822 Project: Apache Drill Issue Type: Task Reporter: Venkata Jyothsna Donapati Assignee: Venkata Jyothsna Donapati -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6381) Add capability to do index based planning and execution
[ https://issues.apache.org/jira/browse/DRILL-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670391#comment-16670391 ] ASF GitHub Bot commented on DRILL-6381: --- dvjyothsna opened a new pull request #1516: Fixed imports for DRILL-6381 URL: https://github.com/apache/drill/pull/1516 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add capability to do index based planning and execution > --- > > Key: DRILL-6381 > URL: https://issues.apache.org/jira/browse/DRILL-6381 > Project: Apache Drill > Issue Type: New Feature > Components: Execution - Relational Operators, Query Planning > Optimization >Reporter: Aman Sinha >Assignee: Aman Sinha >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.15.0 > > > If the underlying data source supports indexes (primary and secondary > indexes), Drill should leverage those during planning and execution in order > to improve query performance. > On the planning side, Drill planner should be enhanced to provide an > abstraction layer which express the index metadata and statistics. Further, > a cost-based index selection is needed to decide which index(es) are > suitable. > On the execution side, appropriate operator enhancements would be needed to > handle different categories of indexes such as covering, non-covering > indexes, taking into consideration the index data may not be co-located with > the primary table, i.e a global index. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670390#comment-16670390
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev commented on issue #1446: DRILL-6349: Drill JDBC driver fails on
Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434767030
Could you provide information, how I can run this tests on my local machine?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)}}
> {{ at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)}}
> {{... 20 more}}
> apache
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[
https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670333#comment-16670333
]
ASF GitHub Bot commented on DRILL-6798:
---
amansinha100 commented on a change in pull request #1514: DRILL-6798: Planner
changes to support semi-join.
URL: https://github.com/apache/drill/pull/1514#discussion_r229765364
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashJoinPrel.java
##
@@ -50,14 +51,21 @@
private int joinControl;
public HashJoinPrel(RelOptCluster cluster, RelTraitSet traits, RelNode left,
RelNode right, RexNode condition,
- JoinRelType joinType) throws InvalidRelException {
-this(cluster, traits, left, right, condition, joinType, false, null,
false, JoinControl.DEFAULT);
+ JoinRelType joinType, boolean semiJoin) throws
InvalidRelException {
+this(cluster, traits, left, right, condition, joinType, false, null,
false, JoinControl.DEFAULT, semiJoin);
+ }
+
+ public HashJoinPrel(RelOptCluster cluster, RelTraitSet traits, RelNode left,
RelNode right, RexNode condition,
+ JoinRelType joinType, boolean swapped, RuntimeFilterDef
runtimeFilterDef,
+ boolean isRowKeyJoin, int joinControl) throws
InvalidRelException {
+this(cluster, traits, left, right, condition, joinType, swapped,
runtimeFilterDef, isRowKeyJoin, joinControl, false);
}
public HashJoinPrel(RelOptCluster cluster, RelTraitSet traits, RelNode left,
RelNode right, RexNode condition,
JoinRelType joinType, boolean swapped, RuntimeFilterDef runtimeFilterDef,
- boolean isRowKeyJoin, int joinControl) throws InvalidRelException {
-super(cluster, traits, left, right, condition, joinType);
+ boolean isRowKeyJoin, int joinControl, boolean semiJoin) throws
InvalidRelException {
+super(cluster, traits, left, right, condition, joinType, semiJoin);
+Preconditions.checkArgument(isSemiJoin && !swapped || swapped &&
!isSemiJoin || (!swapped && !isSemiJoin));
Review comment:
Why not simplify to :
`if (isSemiJoin) {`
`Preconditions.checkArgument(!swapped : 'swapping of inputs is not
allowed for semi-joins') `
`}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner changes to support semi-join
>
>
> Key: DRILL-6798
> URL: https://issues.apache.org/jira/browse/DRILL-6798
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Boaz Ben-Zvi
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[
https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670334#comment-16670334
]
ASF GitHub Bot commented on DRILL-6798:
---
amansinha100 commented on a change in pull request #1514: DRILL-6798: Planner
changes to support semi-join.
URL: https://github.com/apache/drill/pull/1514#discussion_r229762981
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillSemiJoinRel.java
##
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTrait;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinInfo;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.SemiJoin;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.ImmutableIntList;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.FieldReference;
+import org.apache.drill.common.logical.data.Join;
+import org.apache.drill.common.logical.data.JoinCondition;
+import org.apache.drill.common.logical.data.LogicalOperator;
+import org.apache.drill.common.logical.data.LogicalSemiJoin;
+import org.apache.drill.exec.planner.physical.DrillDistributionTrait;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class DrillSemiJoinRel extends SemiJoin implements DrillJoin, DrillRel {
+
+ public DrillSemiJoinRel(
+ RelOptCluster cluster,
+ RelTraitSet traitSet,
+ RelNode left,
+ RelNode right,
+ RexNode condition,
+ ImmutableIntList leftKeys,
+ ImmutableIntList rightKeys) {
+super(cluster,
+ traitSet,
+ left,
+ right,
+ condition,
+ leftKeys,
+ rightKeys);
+Preconditions.checkArgument(validateTraits(traitSet, left, right));
Review comment:
The intent of my previous question was to see if this traits validation is
already happening in Calcite since this functionality ideally belongs there.
I suppose you checked that ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner changes to support semi-join
>
>
> Key: DRILL-6798
> URL: https://issues.apache.org/jira/browse/DRILL-6798
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Boaz Ben-Zvi
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[
https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670332#comment-16670332
]
ASF GitHub Bot commented on DRILL-6798:
---
amansinha100 commented on a change in pull request #1514: DRILL-6798: Planner
changes to support semi-join.
URL: https://github.com/apache/drill/pull/1514#discussion_r229763866
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillSemiJoinRel.java
##
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTrait;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinInfo;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.SemiJoin;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.ImmutableIntList;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.FieldReference;
+import org.apache.drill.common.logical.data.Join;
+import org.apache.drill.common.logical.data.JoinCondition;
+import org.apache.drill.common.logical.data.LogicalOperator;
+import org.apache.drill.common.logical.data.LogicalSemiJoin;
+import org.apache.drill.exec.planner.physical.DrillDistributionTrait;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class DrillSemiJoinRel extends SemiJoin implements DrillJoin, DrillRel {
+
+ public DrillSemiJoinRel(
+ RelOptCluster cluster,
+ RelTraitSet traitSet,
+ RelNode left,
+ RelNode right,
+ RexNode condition,
+ ImmutableIntList leftKeys,
+ ImmutableIntList rightKeys) {
+super(cluster,
+ traitSet,
+ left,
+ right,
+ condition,
+ leftKeys,
+ rightKeys);
+Preconditions.checkArgument(validateTraits(traitSet, left, right));
+ }
+
+ private static boolean validateTraits(RelTraitSet traitSet, RelNode left,
RelNode right) {
+ImmutableBitSet bitSet =
ImmutableBitSet.range(left.getRowType().getFieldCount(),
+left.getRowType().getFieldCount() +
right.getRowType().getFieldCount());
+for (RelTrait trait: traitSet) {
+ if (trait.getTraitDef().getTraitClass().equals(RelCollation.class)) {
+RelCollation collationTrait = (RelCollation)trait;
+for (RelFieldCollation field : collationTrait.getFieldCollations()) {
+ if (bitSet.indexOf(field.getFieldIndex()) < 0) {
Review comment:
Should this be ` > 0` ? since we want to check presence of the right side's
field in the collation traits.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner changes to support semi-join
>
>
> Key: DRILL-6798
> URL: https://issues.apache.org/jira/browse/DRILL-6798
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Boaz Ben-Zvi
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670178#comment-16670178
]
ASF GitHub Bot commented on DRILL-6349:
---
vvysotskyi commented on issue #1446: DRILL-6349: Drill JDBC driver fails on
Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434710546
The problem was connected with `maven-failsafe-plugin`, it should be updated
to 2.22.0. Also, please fix issues with `URLClassLoader` in `ITTestShadedJar`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)}}
> {{ at
[jira] [Updated] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-6349:
---
Labels: (was: ready-to-commit)
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)}}
> {{ at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)}}
> {{... 20 more}}
> apache drill 1.13.0
> "just drill it"
> 0: jdbc:drill:drillbit=localhost>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670165#comment-16670165
]
ASF GitHub Bot commented on DRILL-6349:
---
vvysotskyi commented on issue #1446: DRILL-6349: Drill JDBC driver fails on
Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434705079
@oleg-zinovev, on JDK 10 there is a tests failures for `drill-jdbc-all`
module. Could you please take a look?
```
Execution default of goal
org.apache.maven.plugins:maven-failsafe-plugin:2.18.1:integration-test failed:
java.lang.NoClassDefFoundError: java/sql/Connection: java.sql.Connection
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
>
[jira] [Commented] (DRILL-786) Implement CROSS JOIN
[
https://issues.apache.org/jira/browse/DRILL-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670157#comment-16670157
]
ASF GitHub Bot commented on DRILL-786:
--
ihuzenko commented on issue #1488: DRILL-786: Allow CROSS JOIN syntax
URL: https://github.com/apache/drill/pull/1488#issuecomment-434703183
Hi @amansinha100 . Thanks for your active involvement, I investigated the
issue and found that:
when option `planner.slice_target` is set to low value, then StreamAggPrule
at the first match will try to create 2 phase plan, but method
call.transformTo(newRel) won't be called (means that rule results are ignored),
because the rule matches at the very early stage of physical planning, there
aren't rels matched by new traitSet with physical convention. Currently for
simple cross join query with aggregate function, physical rules are applied in
order, which is wrong and results into PlanningException:
Prel.ScreenPrule
Prel.ScreenPrule
Prel.ScanPrule
Prel.ScanPrule
ProjectPrule
ProjectPrule
Prel.NestedLoopJoinPrule
ExpandConversionRule
ExpandConversionRule
The problem may be fixed either by returning false from
StreamAggPrule.create2PhasePlan(call, aggregate) for cross join queries, so
then StreamAggPrule will be applied successfully at early stage. Or another
option is to change NestedLoopJoinPrule left and join node's distribution trait
from ANY to SINGLETON, which then results in activation of ProjectPrule ->
StreamAggPrule etc. So then StreamAggPrule is activated again, but now at the
right stage when it can be applied successfully.
@vvysotskyi suggests to proceed with second option, but in order to do this
I need to investigate more about how changing of the distribution traits for
NestedLoopJoinPrule left and join node impacts physical planning. And then, if
possible, try to activate necessary rules without distribution trait change.
@amansinha100 I'd appreciate if you could share your thoughts about the
options.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement CROSS JOIN
>
>
> Key: DRILL-786
> URL: https://issues.apache.org/jira/browse/DRILL-786
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Krystal
>Assignee: Igor Guzenko
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> git.commit.id.abbrev=5d7e3d3
> 0: jdbc:drill:schema=dfs> select student.name, student.age,
> student.studentnum from student cross join voter where student.age = 20 and
> voter.age = 20;
> Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while
> running query.[error_id: "af90e65a-c4d7-4635-a436-bbc1444c8db2"
> Root: rel#318:Subset#28.PHYSICAL.SINGLETON([]).[]
> Original rel:
> AbstractConverter(subset=[rel#318:Subset#28.PHYSICAL.SINGLETON([]).[]],
> convention=[PHYSICAL], DrillDistributionTraitDef=[SINGLETON([])], sort=[[]]):
> rowcount = 22500.0, cumulative cost = {inf}, id = 320
> DrillScreenRel(subset=[rel#317:Subset#28.LOGICAL.ANY([]).[]]): rowcount =
> 22500.0, cumulative cost = {2250.0 rows, 2250.0 cpu, 0.0 io, 0.0 network}, id
> = 316
> DrillProjectRel(subset=[rel#315:Subset#27.LOGICAL.ANY([]).[]], name=[$2],
> age=[$1], studentnum=[$3]): rowcount = 22500.0, cumulative cost = {22500.0
> rows, 12.0 cpu, 0.0 io, 0.0 network}, id = 314
> DrillJoinRel(subset=[rel#313:Subset#26.LOGICAL.ANY([]).[]],
> condition=[true], joinType=[inner]): rowcount = 22500.0, cumulative cost =
> {22500.0 rows, 0.0 cpu, 0.0 io, 0.0 network}, id = 312
> DrillFilterRel(subset=[rel#308:Subset#23.LOGICAL.ANY([]).[]],
> condition=[=(CAST($1):INTEGER, 20)]): rowcount = 150.0, cumulative cost =
> {1000.0 rows, 4000.0 cpu, 0.0 io, 0.0 network}, id = 307
> DrillScanRel(subset=[rel#306:Subset#22.LOGICAL.ANY([]).[]],
> table=[[dfs, student]]): rowcount = 1000.0, cumulative cost = {1000.0 rows,
> 4000.0 cpu, 0.0 io, 0.0 network}, id = 129
> DrillFilterRel(subset=[rel#311:Subset#25.LOGICAL.ANY([]).[]],
> condition=[=(CAST($1):INTEGER, 20)]): rowcount = 150.0, cumulative cost =
> {1000.0 rows, 4000.0 cpu, 0.0 io, 0.0 network}, id = 310
> DrillScanRel(subset=[rel#309:Subset#24.LOGICAL.ANY([]).[]],
> table=[[dfs, voter]]): rowcount = 1000.0, cumulative cost = {1000.0 rows,
> 2000.0 cpu, 0.0 io, 0.0 network}, id = 140
> Stack trace:
> org.eigenbase.relopt.RelOptPlanner$CannotPlanException: Node
>
[jira] [Updated] (DRILL-3610) TimestampAdd/Diff (SQL_TSI_) functions
[
https://issues.apache.org/jira/browse/DRILL-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-3610:
Labels: doc-impacting (was: )
> TimestampAdd/Diff (SQL_TSI_) functions
> --
>
> Key: DRILL-3610
> URL: https://issues.apache.org/jira/browse/DRILL-3610
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Drill
>Reporter: Andries Engelbrecht
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Add TimestampAdd and TimestampDiff (SQL_TSI) functions for year, quarter,
> month, week, day, hour, minute, second.
> Examples
> SELECT CAST(TIMESTAMPADD(SQL_TSI_QUARTER,1,Date('2013-03-31'), SQL_DATE) AS
> `column_quarter`
> FROM `table_in`
> HAVING (COUNT(1) > 0)
> SELECT `table_in`.`datetime` AS `column1`,
> `table`.`Key` AS `column_Key`,
> TIMESTAMPDIFF(SQL_TSI_MINUTE,to_timestamp('2004-07-04',
> '-MM-dd'),`table_in`.`datetime`) AS `sum_datediff_minute`
> FROM `calcs`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-3610) TimestampAdd/Diff (SQL_TSI_) functions
[
https://issues.apache.org/jira/browse/DRILL-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva reassigned DRILL-3610:
---
Assignee: Volodymyr Vysotskyi (was: Arina Ielchiieva)
> TimestampAdd/Diff (SQL_TSI_) functions
> --
>
> Key: DRILL-3610
> URL: https://issues.apache.org/jira/browse/DRILL-3610
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Drill
>Reporter: Andries Engelbrecht
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> Add TimestampAdd and TimestampDiff (SQL_TSI) functions for year, quarter,
> month, week, day, hour, minute, second.
> Examples
> SELECT CAST(TIMESTAMPADD(SQL_TSI_QUARTER,1,Date('2013-03-31'), SQL_DATE) AS
> `column_quarter`
> FROM `table_in`
> HAVING (COUNT(1) > 0)
> SELECT `table_in`.`datetime` AS `column1`,
> `table`.`Key` AS `column_Key`,
> TIMESTAMPDIFF(SQL_TSI_MINUTE,to_timestamp('2004-07-04',
> '-MM-dd'),`table_in`.`datetime`) AS `sum_datediff_minute`
> FROM `calcs`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-3610) TimestampAdd/Diff (SQL_TSI_) functions
[
https://issues.apache.org/jira/browse/DRILL-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-3610:
Fix Version/s: (was: Future)
1.15.0
> TimestampAdd/Diff (SQL_TSI_) functions
> --
>
> Key: DRILL-3610
> URL: https://issues.apache.org/jira/browse/DRILL-3610
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Drill
>Reporter: Andries Engelbrecht
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> Add TimestampAdd and TimestampDiff (SQL_TSI) functions for year, quarter,
> month, week, day, hour, minute, second.
> Examples
> SELECT CAST(TIMESTAMPADD(SQL_TSI_QUARTER,1,Date('2013-03-31'), SQL_DATE) AS
> `column_quarter`
> FROM `table_in`
> HAVING (COUNT(1) > 0)
> SELECT `table_in`.`datetime` AS `column1`,
> `table`.`Key` AS `column_Key`,
> TIMESTAMPDIFF(SQL_TSI_MINUTE,to_timestamp('2004-07-04',
> '-MM-dd'),`table_in`.`datetime`) AS `sum_datediff_minute`
> FROM `calcs`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-3610) TimestampAdd/Diff (SQL_TSI_) functions
[
https://issues.apache.org/jira/browse/DRILL-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-3610:
Reviewer: Arina Ielchiieva
> TimestampAdd/Diff (SQL_TSI_) functions
> --
>
> Key: DRILL-3610
> URL: https://issues.apache.org/jira/browse/DRILL-3610
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Drill
>Reporter: Andries Engelbrecht
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> Add TimestampAdd and TimestampDiff (SQL_TSI) functions for year, quarter,
> month, week, day, hour, minute, second.
> Examples
> SELECT CAST(TIMESTAMPADD(SQL_TSI_QUARTER,1,Date('2013-03-31'), SQL_DATE) AS
> `column_quarter`
> FROM `table_in`
> HAVING (COUNT(1) > 0)
> SELECT `table_in`.`datetime` AS `column1`,
> `table`.`Key` AS `column_Key`,
> TIMESTAMPDIFF(SQL_TSI_MINUTE,to_timestamp('2004-07-04',
> '-MM-dd'),`table_in`.`datetime`) AS `sum_datediff_minute`
> FROM `calcs`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-4456:
Reviewer: Arina Ielchiieva
> Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6792) Find the right probe side fragment to any storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Diravka updated DRILL-6792: --- Reviewer: Sorabh Hamirwasia > Find the right probe side fragment to any storage plugin > > > Key: DRILL-6792 > URL: https://issues.apache.org/jira/browse/DRILL-6792 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Fix For: 1.15.0 > > > The current implementation of JPPD to find the probe side wrapper depends on > the GroupScan's digest. But there's no promise the GroupScan's digest will > not be changed since it is attached to the RuntimeFilterDef by different > storage plugin implementation logic.So here we assign a unique identifier to > the RuntimeFilter operator, and find the right probe side fragment wrapper by > the runtime filter identifier at the RuntimeFilterRouter class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6768) Improve to_date, to_time and to_timestamp and corresponding cast functions to handle empty string when `drill.exec.functions.cast_empty_string_to_null` option is enable
[ https://issues.apache.org/jira/browse/DRILL-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670007#comment-16670007 ] Bohdan Kazydub commented on DRILL-6768: --- [~vitalii], yes, it is possible. One way is to introduce `NullHandling.NULL_IF_NULL_OR_EMPTY_STRING` strategy, which handles empty string input as if it was NULL. The other way is to enhance `NullHandling.NULL_IF_NULL` (there is an open issue for this) to allow setting null values to output from function body methods. While there will still be a need to handle empty strings yourself there could be less function definitions because handling framework should take care of this cases. > Improve to_date, to_time and to_timestamp and corresponding cast functions to > handle empty string when `drill.exec.functions.cast_empty_string_to_null` > option is enabled > - > > Key: DRILL-6768 > URL: https://issues.apache.org/jira/browse/DRILL-6768 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.14.0 >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.15.0 > > > When `drill.exec.functions.cast_empty_string_to_null` option is enabled > `to_date`, `to_time` and `to_timestamp` functions while converting string to > according type in case if null or empty string values are passed will return > NULL (to avoid CASE clauses which are littering a query and will work in > accordance with their respective CAST counterparts) for both cases. > > > > CASTs will be handled in a similar way (uniformly with numeric types): > > ||Value to cast||Now||Will be|| > |NULL|NULL|NULL| > |'' (empty string)|Error in many cases (except numerical types)|NULL| > CAST empty string to null (in case of enabled option) will be supported by > DATE, TIME, TIMESTAMP, INTERVAL YEAR, INTERVAL MONTH and INTERVAL DAY > functions in addition to numeric types. > > *For documentation* > TBA -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6815) Improve code generation to handle functions with NullHandling.NULL_IF_NULL better
[
https://issues.apache.org/jira/browse/DRILL-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated DRILL-6815:
---
Component/s: Execution - Codegen
> Improve code generation to handle functions with NullHandling.NULL_IF_NULL
> better
> -
>
> Key: DRILL-6815
> URL: https://issues.apache.org/jira/browse/DRILL-6815
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Codegen
>Reporter: Bohdan Kazydub
>Priority: Minor
> Fix For: Future
>
>
> If a (simple) function is declared with NULL_IF_NULL null handling strategy
> (`nulls = NullHandling.NULL_IF_NULL`) there is a additional code generated
> which checks if any of the inputs is NULL (not set). In case if there is,
> output is set to be null otherwise function's code is executed and at the end
> output value is marked as set in case if ANY of the inputs is OPTIONAL (see
> https://github.com/apache/drill/blob/8edeb49873d1a1710cfe28e0b49364d07eb1aef4/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillSimpleFuncHolder.java#L143).
> The problem is, this behavior makes it impossible to make output value NULL
> from within function's evaluation body. Which may prove useful in certain
> situations, e.g. when input is an empty string and output should be NULL in
> the case etc. Sometimes it may result in creation of two separate functions
> with NullHanling.INTERNAL (one for OPTIONAL and one for REQUIRED inputs)
> instead of one with NULL_IF_NULL. It does not follow a Principle of Least
> Astonishment as effectively it behaves more like "null if and only if null"
> and documentation for NULL_IF_NULL is as follows:
> {code}
> enum NullHandling {
> ...
> /**
> * Null output if any null input:
> * Indicates that a method's associated logical operation returns NULL if
> * either input is NULL, and therefore that the method must not be called
> * with null inputs. (The calling framework must handle NULLs.)
> */
> NULL_IF_NULL
> }
> {code}
> It looks as if this behavior was not intended.
> Intent of this improvement is to allow output NULL values based on function's
> eval() method when NULL_IF_NULL null handling strategy is chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6815) Improve code generation to handle functions with NullHandling.NULL_IF_NULL better
[
https://issues.apache.org/jira/browse/DRILL-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated DRILL-6815:
---
Fix Version/s: Future
> Improve code generation to handle functions with NullHandling.NULL_IF_NULL
> better
> -
>
> Key: DRILL-6815
> URL: https://issues.apache.org/jira/browse/DRILL-6815
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Codegen
>Reporter: Bohdan Kazydub
>Priority: Minor
> Fix For: Future
>
>
> If a (simple) function is declared with NULL_IF_NULL null handling strategy
> (`nulls = NullHandling.NULL_IF_NULL`) there is a additional code generated
> which checks if any of the inputs is NULL (not set). In case if there is,
> output is set to be null otherwise function's code is executed and at the end
> output value is marked as set in case if ANY of the inputs is OPTIONAL (see
> https://github.com/apache/drill/blob/8edeb49873d1a1710cfe28e0b49364d07eb1aef4/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillSimpleFuncHolder.java#L143).
> The problem is, this behavior makes it impossible to make output value NULL
> from within function's evaluation body. Which may prove useful in certain
> situations, e.g. when input is an empty string and output should be NULL in
> the case etc. Sometimes it may result in creation of two separate functions
> with NullHanling.INTERNAL (one for OPTIONAL and one for REQUIRED inputs)
> instead of one with NULL_IF_NULL. It does not follow a Principle of Least
> Astonishment as effectively it behaves more like "null if and only if null"
> and documentation for NULL_IF_NULL is as follows:
> {code}
> enum NullHandling {
> ...
> /**
> * Null output if any null input:
> * Indicates that a method's associated logical operation returns NULL if
> * either input is NULL, and therefore that the method must not be called
> * with null inputs. (The calling framework must handle NULLs.)
> */
> NULL_IF_NULL
> }
> {code}
> It looks as if this behavior was not intended.
> Intent of this improvement is to allow output NULL values based on function's
> eval() method when NULL_IF_NULL null handling strategy is chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6768) Improve to_date, to_time and to_timestamp and corresponding cast functions to handle empty string when `drill.exec.functions.cast_empty_string_to_null` option is e
[
https://issues.apache.org/jira/browse/DRILL-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669995#comment-16669995
]
Vitalii Diravka edited comment on DRILL-6768 at 10/31/18 11:50 AM:
---
[~KazydubB] Is it possible to improve the solution in the future?
Current approach is to have the separate functions for the cases with Nullable
output and possible empty input String.
But the only one example of function can exist and the following code can be
added in the process of the code generation:
{code:java}
if (left.start == left.end) {
out.isSet = 0;
return;
}
{code}
was (Author: vitalii):
Please describe the possibility of improving it in the future?
Current approach is to have the separate functions for the cases with Nullable
output and possible empty input String.
But the only one example of function can exist and the following code can be
added in the process of the code generation:
{code:java}
if (left.start == left.end) {
out.isSet = 0;
return;
}
{code}
> Improve to_date, to_time and to_timestamp and corresponding cast functions to
> handle empty string when `drill.exec.functions.cast_empty_string_to_null`
> option is enabled
> -
>
> Key: DRILL-6768
> URL: https://issues.apache.org/jira/browse/DRILL-6768
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.15.0
>
>
> When `drill.exec.functions.cast_empty_string_to_null` option is enabled
> `to_date`, `to_time` and `to_timestamp` functions while converting string to
> according type in case if null or empty string values are passed will return
> NULL (to avoid CASE clauses which are littering a query and will work in
> accordance with their respective CAST counterparts) for both cases.
>
>
>
> CASTs will be handled in a similar way (uniformly with numeric types):
>
> ||Value to cast||Now||Will be||
> |NULL|NULL|NULL|
> |'' (empty string)|Error in many cases (except numerical types)|NULL|
> CAST empty string to null (in case of enabled option) will be supported by
> DATE, TIME, TIMESTAMP, INTERVAL YEAR, INTERVAL MONTH and INTERVAL DAY
> functions in addition to numeric types.
>
> *For documentation*
> TBA
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6768) Improve to_date, to_time and to_timestamp and corresponding cast functions to handle empty string when `drill.exec.functions.cast_empty_string_to_null` option is enable
[
https://issues.apache.org/jira/browse/DRILL-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669995#comment-16669995
]
Vitalii Diravka commented on DRILL-6768:
Please describe the possibility of improving it in the future?
Current approach is to have the separate functions for the cases with Nullable
output and possible empty input String.
But the only one example of function can exist and the following code can be
added in the process of the code generation:
{code:java}
if (left.start == left.end) {
out.isSet = 0;
return;
}
{code}
> Improve to_date, to_time and to_timestamp and corresponding cast functions to
> handle empty string when `drill.exec.functions.cast_empty_string_to_null`
> option is enabled
> -
>
> Key: DRILL-6768
> URL: https://issues.apache.org/jira/browse/DRILL-6768
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.15.0
>
>
> When `drill.exec.functions.cast_empty_string_to_null` option is enabled
> `to_date`, `to_time` and `to_timestamp` functions while converting string to
> according type in case if null or empty string values are passed will return
> NULL (to avoid CASE clauses which are littering a query and will work in
> accordance with their respective CAST counterparts) for both cases.
>
>
>
> CASTs will be handled in a similar way (uniformly with numeric types):
>
> ||Value to cast||Now||Will be||
> |NULL|NULL|NULL|
> |'' (empty string)|Error in many cases (except numerical types)|NULL|
> CAST empty string to null (in case of enabled option) will be supported by
> DATE, TIME, TIMESTAMP, INTERVAL YEAR, INTERVAL MONTH and INTERVAL DAY
> functions in addition to numeric types.
>
> *For documentation*
> TBA
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6768) Improve to_date, to_time and to_timestamp and corresponding cast functions to handle empty string when `drill.exec.functions.cast_empty_string_to_null` option is enabled
[ https://issues.apache.org/jira/browse/DRILL-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Diravka updated DRILL-6768: --- Labels: doc-impacting ready-to-commit (was: doc-impacting) > Improve to_date, to_time and to_timestamp and corresponding cast functions to > handle empty string when `drill.exec.functions.cast_empty_string_to_null` > option is enabled > - > > Key: DRILL-6768 > URL: https://issues.apache.org/jira/browse/DRILL-6768 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.14.0 >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.15.0 > > > When `drill.exec.functions.cast_empty_string_to_null` option is enabled > `to_date`, `to_time` and `to_timestamp` functions while converting string to > according type in case if null or empty string values are passed will return > NULL (to avoid CASE clauses which are littering a query and will work in > accordance with their respective CAST counterparts) for both cases. > > > > CASTs will be handled in a similar way (uniformly with numeric types): > > ||Value to cast||Now||Will be|| > |NULL|NULL|NULL| > |'' (empty string)|Error in many cases (except numerical types)|NULL| > CAST empty string to null (in case of enabled option) will be supported by > DATE, TIME, TIMESTAMP, INTERVAL YEAR, INTERVAL MONTH and INTERVAL DAY > functions in addition to numeric types. > > *For documentation* > TBA -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6768) Improve to_date, to_time and to_timestamp and corresponding cast functions to handle empty string when `drill.exec.functions.cast_empty_string_to_null` option is enable
[ https://issues.apache.org/jira/browse/DRILL-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669994#comment-16669994 ] ASF GitHub Bot commented on DRILL-6768: --- vdiravka commented on issue #1494: DRILL-6768: Improve to_date, to_time and to_timestamp and correspondi… URL: https://github.com/apache/drill/pull/1494#issuecomment-434657378 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve to_date, to_time and to_timestamp and corresponding cast functions to > handle empty string when `drill.exec.functions.cast_empty_string_to_null` > option is enabled > - > > Key: DRILL-6768 > URL: https://issues.apache.org/jira/browse/DRILL-6768 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.14.0 >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > When `drill.exec.functions.cast_empty_string_to_null` option is enabled > `to_date`, `to_time` and `to_timestamp` functions while converting string to > according type in case if null or empty string values are passed will return > NULL (to avoid CASE clauses which are littering a query and will work in > accordance with their respective CAST counterparts) for both cases. > > > > CASTs will be handled in a similar way (uniformly with numeric types): > > ||Value to cast||Now||Will be|| > |NULL|NULL|NULL| > |'' (empty string)|Error in many cases (except numerical types)|NULL| > CAST empty string to null (in case of enabled option) will be supported by > DATE, TIME, TIMESTAMP, INTERVAL YEAR, INTERVAL MONTH and INTERVAL DAY > functions in addition to numeric types. > > *For documentation* > TBA -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669986#comment-16669986
]
shuifeng lu commented on DRILL-6722:
Oh, sorry, saveCodeForDebugging is setting to true for debugging.:D
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669989#comment-16669989
]
Oleg Zinoviev commented on DRILL-6722:
--
I'll try to provide PR tomorrow :)
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6821) Update Hive to the next version which supports JDK 9+
Volodymyr Vysotskyi created DRILL-6821:
--
Summary: Update Hive to the next version which supports JDK 9+
Key: DRILL-6821
URL: https://issues.apache.org/jira/browse/DRILL-6821
Project: Apache Drill
Issue Type: Task
Affects Versions: 1.15.0
Reporter: Volodymyr Vysotskyi
In DRILL-6349 was allowed building Drill with JDK 9-11 versions and fixed unit
tests for all modules except {{storage-hive}} module. The problem is that the
current Hive version (2.3.2) does not support JDK 9+ (HIVE-17632).
The aim of this Jira is to update Hive to the next version which supports JDK
9-11.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669976#comment-16669976
]
Oleg Zinoviev commented on DRILL-6722:
--
I have a latest master revision, and bug is still here... Could it be that your
drill.exec.compile.prefer_plain_java value is true?
{noformat}
By the way, how to generate this class without setting saveCodeForDebugging to
true?{noformat}
Breakpoint -> Evaluate expression -> Write a byte array with class into file ->
Use any decompiler
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669968#comment-16669968
]
shuifeng lu commented on DRILL-6722:
[~le.louch] Thanks for your reply.
This test works fine with DRILL-6763 on my laptop.
By the way, how to generate this class without setting saveCodeForDebugging to
true?
I've been struggling with it for a time.
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6791) Merge scan projection framework into master
[
https://issues.apache.org/jira/browse/DRILL-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669949#comment-16669949
]
ASF GitHub Bot commented on DRILL-6791:
---
arina-ielchiieva commented on a change in pull request #1501: DRILL-6791: Scan
projection framework
URL: https://github.com/apache/drill/pull/1501#discussion_r226964830
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/ScanLevelProjection.java
##
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.scan.project;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.rowSet.project.RequestedTuple;
+import
org.apache.drill.exec.physical.rowSet.project.RequestedTuple.RequestedColumn;
+import org.apache.drill.exec.physical.rowSet.project.RequestedTupleImpl;
+
+/**
+ * Parses and analyzes the projection list passed to the scanner. The
+ * projection list is per scan, independent of any tables that the
+ * scanner might scan. The projection list is then used as input to the
+ * per-table projection planning.
+ *
+ * Accepts the inputs needed to
+ * plan a projection, builds the mappings, and constructs the projection
+ * mapping object.
+ *
+ * Builds the per-scan projection plan given a set of projected columns.
+ * Determines the output schema, which columns to project from the data
+ * source, which are metadata, and so on.
+ *
+ * An annoying aspect of SQL is that the projection list (the list of
+ * columns to appear in the output) is specified after the SELECT keyword.
+ * In Relational theory, projection is about columns, selection is about
+ * rows...
+ *
+ * Mappings can be based on three primary use cases:
+ *
+ * SELECT *: Project all data source columns, whatever they happen
+ * to be. Create columns using names from the data source. The data source
+ * also determines the order of columns within the row.
+ * SELECT columns: Similar to SELECT * in that it projects all
columns
+ * from the data source, in data source order. But, rather than creating
+ * individual output columns for each data source column, creates a single
+ * column which is an array of Varchars which holds the (text form) of
+ * each column as an array element.
+ * SELECT a, b, c, ...: Project a specific set of columns,
identified by
+ * case-insensitive name. The output row uses the names from the SELECT list,
+ * but types from the data source. Columns appear in the row in the order
+ * specified by the SELECT.
+ * SELECT ...: SELECT nothing, occurs in SELECT COUNT(*)
+ * type queries. The provided projection list contains no (table) columns,
though
+ * it may contain metadata columns.
+ *
+ * Names in the SELECT list can reference any of five distinct types of output
+ * columns:
+ *
+ * Wildcard ("*") column: indicates the place in the projection list to
insert
+ * the table columns once found in the table projection plan.
+ * Data source columns: columns from the underlying table. The table
+ * projection planner will determine if the column exists, or must be filled
+ * in with a null column.
+ * The generic data source columns array: columns, or optionally
+ * specific members of the columns array such as
columns[1].
+ * Implicit columns: fqn, filename, filepath
+ * and suffix. These reference
+ * parts of the name of the file being scanned.
+ * Partition columns: dir0, dir1, ...: These reference
+ * parts of the path name of the file.
+ *
+ *
+ * @see {@link ImplicitColumnExplorer}, the class from which this class
+ * evolved
+ */
+
+public class ScanLevelProjection {
+
+ static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ScanLevelProjection.class);
+
+ /**
+ * Interface for add-on parsers, avoids the need to create
+ * a single, tightly-coupled parser for all types of columns.
+ * The main parser handles wildcards and assumes the rest of
+ * the columns are table columns. The add-on parser can tag
+ * columns as special, such as to hold metadata.
+ */
+
+
[jira] [Commented] (DRILL-6791) Merge scan projection framework into master
[
https://issues.apache.org/jira/browse/DRILL-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669943#comment-16669943
]
ASF GitHub Bot commented on DRILL-6791:
---
arina-ielchiieva commented on a change in pull request #1501: DRILL-6791: Scan
projection framework
URL: https://github.com/apache/drill/pull/1501#discussion_r225643112
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/NoOpMetadataManager.java
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.scan.project;
+
+import
org.apache.drill.exec.physical.impl.scan.project.ScanLevelProjection.ScanProjectionParser;
+import
org.apache.drill.exec.physical.impl.scan.project.SchemaLevelProjection.SchemaProjectionResolver;
+import org.apache.drill.exec.physical.rowSet.ResultVectorCache;
+
+/**
+ * Do-nothing implementation of the metadata manager. Allows the
+ * metadata manager to be optional without needing an if-statement
+ * on every access.
+ */
+
+public class NoOpMetadataManager implements MetadataManager {
+
+ @Override
+ public void bind(ResultVectorCache vectorCache) { }
+
+ @Override
+ public ScanProjectionParser projectionParser() { return null; }
+
+ @Override
+ public SchemaProjectionResolver resolver() {
+throw new UnsupportedOperationException();
Review comment:
Please provide error message.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Merge scan projection framework into master
> ---
>
> Key: DRILL-6791
> URL: https://issues.apache.org/jira/browse/DRILL-6791
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.15.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.15.0
>
>
> Merge the next set of "result set loader" code into master via a PR. This one
> covers the "schema projection" mechanism which:
> * Handles none (SELECT COUNT\(*)), some (SELECT a, b, x) and all (SELECT *)
> projection.
> * Handles null columns (for projection a column "x" that does not exist in
> the base table.)
> * Handles constant columns as used for file metadata (AKA "implicit" columns).
> * Handle schema persistence: the need to reuse the same vectors across
> different scanners
> * Provides a framework for consuming externally-supplied metadata
> * Since we don't yet have a way to provide "real" metadata, obtains metadata
> hints from previous batches and from the projection list (a.b implies that
> "a" is a map, c[0] implies that "c" is an array, etc.)
> * Handles merging the set of data source columns and null columns to create
> the final output batch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6791) Merge scan projection framework into master
[ https://issues.apache.org/jira/browse/DRILL-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669944#comment-16669944 ] ASF GitHub Bot commented on DRILL-6791: --- arina-ielchiieva commented on a change in pull request #1501: DRILL-6791: Scan projection framework URL: https://github.com/apache/drill/pull/1501#discussion_r226276316 ## File path: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/scan/project/TestScanBatchWriters.java ## @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + **/ +package org.apache.drill.exec.physical.impl.scan.project; Review comment: Please remove. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Merge scan projection framework into master > --- > > Key: DRILL-6791 > URL: https://issues.apache.org/jira/browse/DRILL-6791 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.15.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Fix For: 1.15.0 > > > Merge the next set of "result set loader" code into master via a PR. This one > covers the "schema projection" mechanism which: > * Handles none (SELECT COUNT\(*)), some (SELECT a, b, x) and all (SELECT *) > projection. > * Handles null columns (for projection a column "x" that does not exist in > the base table.) > * Handles constant columns as used for file metadata (AKA "implicit" columns). > * Handle schema persistence: the need to reuse the same vectors across > different scanners > * Provides a framework for consuming externally-supplied metadata > * Since we don't yet have a way to provide "real" metadata, obtains metadata > hints from previous batches and from the projection list (a.b implies that > "a" is a map, c[0] implies that "c" is an array, etc.) > * Handles merging the set of data source columns and null columns to create > the final output batch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6791) Merge scan projection framework into master
[
https://issues.apache.org/jira/browse/DRILL-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669947#comment-16669947
]
ASF GitHub Bot commented on DRILL-6791:
---
arina-ielchiieva commented on a change in pull request #1501: DRILL-6791: Scan
projection framework
URL: https://github.com/apache/drill/pull/1501#discussion_r226967308
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/scan/ScanTestUtils.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.scan;
+
+import java.util.List;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.scan.project.ResolvedColumn;
+import org.apache.drill.exec.physical.impl.scan.project.ResolvedTuple;
+import
org.apache.drill.exec.physical.impl.scan.project.ScanLevelProjection.ScanProjectionParser;
+import
org.apache.drill.exec.physical.impl.scan.project.SchemaLevelProjection.SchemaProjectionResolver;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.record.metadata.TupleSchema;
+import com.google.common.collect.Lists;
+
+public class ScanTestUtils {
+
+ // Default file metadata column names; primarily for testing.
+
+ public static final String FILE_NAME_COL = "filename";
Review comment:
Can be taken from `ImplicitColumnExplorer`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Merge scan projection framework into master
> ---
>
> Key: DRILL-6791
> URL: https://issues.apache.org/jira/browse/DRILL-6791
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.15.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.15.0
>
>
> Merge the next set of "result set loader" code into master via a PR. This one
> covers the "schema projection" mechanism which:
> * Handles none (SELECT COUNT\(*)), some (SELECT a, b, x) and all (SELECT *)
> projection.
> * Handles null columns (for projection a column "x" that does not exist in
> the base table.)
> * Handles constant columns as used for file metadata (AKA "implicit" columns).
> * Handle schema persistence: the need to reuse the same vectors across
> different scanners
> * Provides a framework for consuming externally-supplied metadata
> * Since we don't yet have a way to provide "real" metadata, obtains metadata
> hints from previous batches and from the projection list (a.b implies that
> "a" is a map, c[0] implies that "c" is an array, etc.)
> * Handles merging the set of data source columns and null columns to create
> the final output batch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6791) Merge scan projection framework into master
[
https://issues.apache.org/jira/browse/DRILL-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669945#comment-16669945
]
ASF GitHub Bot commented on DRILL-6791:
---
arina-ielchiieva commented on a change in pull request #1501: DRILL-6791: Scan
projection framework
URL: https://github.com/apache/drill/pull/1501#discussion_r226964830
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/ScanLevelProjection.java
##
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.scan.project;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.rowSet.project.RequestedTuple;
+import
org.apache.drill.exec.physical.rowSet.project.RequestedTuple.RequestedColumn;
+import org.apache.drill.exec.physical.rowSet.project.RequestedTupleImpl;
+
+/**
+ * Parses and analyzes the projection list passed to the scanner. The
+ * projection list is per scan, independent of any tables that the
+ * scanner might scan. The projection list is then used as input to the
+ * per-table projection planning.
+ *
+ * Accepts the inputs needed to
+ * plan a projection, builds the mappings, and constructs the projection
+ * mapping object.
+ *
+ * Builds the per-scan projection plan given a set of projected columns.
+ * Determines the output schema, which columns to project from the data
+ * source, which are metadata, and so on.
+ *
+ * An annoying aspect of SQL is that the projection list (the list of
+ * columns to appear in the output) is specified after the SELECT keyword.
+ * In Relational theory, projection is about columns, selection is about
+ * rows...
+ *
+ * Mappings can be based on three primary use cases:
+ *
+ * SELECT *: Project all data source columns, whatever they happen
+ * to be. Create columns using names from the data source. The data source
+ * also determines the order of columns within the row.
+ * SELECT columns: Similar to SELECT * in that it projects all
columns
+ * from the data source, in data source order. But, rather than creating
+ * individual output columns for each data source column, creates a single
+ * column which is an array of Varchars which holds the (text form) of
+ * each column as an array element.
+ * SELECT a, b, c, ...: Project a specific set of columns,
identified by
+ * case-insensitive name. The output row uses the names from the SELECT list,
+ * but types from the data source. Columns appear in the row in the order
+ * specified by the SELECT.
+ * SELECT ...: SELECT nothing, occurs in SELECT COUNT(*)
+ * type queries. The provided projection list contains no (table) columns,
though
+ * it may contain metadata columns.
+ *
+ * Names in the SELECT list can reference any of five distinct types of output
+ * columns:
+ *
+ * Wildcard ("*") column: indicates the place in the projection list to
insert
+ * the table columns once found in the table projection plan.
+ * Data source columns: columns from the underlying table. The table
+ * projection planner will determine if the column exists, or must be filled
+ * in with a null column.
+ * The generic data source columns array: columns, or optionally
+ * specific members of the columns array such as
columns[1].
+ * Implicit columns: fqn, filename, filepath
+ * and suffix. These reference
+ * parts of the name of the file being scanned.
+ * Partition columns: dir0, dir1, ...: These reference
+ * parts of the path name of the file.
+ *
+ *
+ * @see {@link ImplicitColumnExplorer}, the class from which this class
+ * evolved
+ */
+
+public class ScanLevelProjection {
+
+ static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ScanLevelProjection.class);
+
+ /**
+ * Interface for add-on parsers, avoids the need to create
+ * a single, tightly-coupled parser for all types of columns.
+ * The main parser handles wildcards and assumes the rest of
+ * the columns are table columns. The add-on parser can tag
+ * columns as special, such as to hold metadata.
+ */
+
+
[jira] [Commented] (DRILL-6791) Merge scan projection framework into master
[
https://issues.apache.org/jira/browse/DRILL-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669946#comment-16669946
]
ASF GitHub Bot commented on DRILL-6791:
---
arina-ielchiieva commented on a change in pull request #1501: DRILL-6791: Scan
projection framework
URL: https://github.com/apache/drill/pull/1501#discussion_r226967308
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/scan/ScanTestUtils.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.scan;
+
+import java.util.List;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.scan.project.ResolvedColumn;
+import org.apache.drill.exec.physical.impl.scan.project.ResolvedTuple;
+import
org.apache.drill.exec.physical.impl.scan.project.ScanLevelProjection.ScanProjectionParser;
+import
org.apache.drill.exec.physical.impl.scan.project.SchemaLevelProjection.SchemaProjectionResolver;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.record.metadata.TupleSchema;
+import com.google.common.collect.Lists;
+
+public class ScanTestUtils {
+
+ // Default file metadata column names; primarily for testing.
+
+ public static final String FILE_NAME_COL = "filename";
Review comment:
Can betaken from `ImplicitColumnExplorer`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Merge scan projection framework into master
> ---
>
> Key: DRILL-6791
> URL: https://issues.apache.org/jira/browse/DRILL-6791
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.15.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.15.0
>
>
> Merge the next set of "result set loader" code into master via a PR. This one
> covers the "schema projection" mechanism which:
> * Handles none (SELECT COUNT\(*)), some (SELECT a, b, x) and all (SELECT *)
> projection.
> * Handles null columns (for projection a column "x" that does not exist in
> the base table.)
> * Handles constant columns as used for file metadata (AKA "implicit" columns).
> * Handle schema persistence: the need to reuse the same vectors across
> different scanners
> * Provides a framework for consuming externally-supplied metadata
> * Since we don't yet have a way to provide "real" metadata, obtains metadata
> hints from previous batches and from the projection list (a.b implies that
> "a" is a map, c[0] implies that "c" is an array, etc.)
> * Handles merging the set of data source columns and null columns to create
> the final output batch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669937#comment-16669937
]
Oleg Zinoviev commented on DRILL-6722:
--
[~shuifeng], thanks for your comment.
This is generated class file before ValueHolderReplacementVisitor call:
[^JaininoJava.class]
This is generated class file after ValueHolderReplacementVisitor call:
[^JaininoJava2_merged.class]
Decompiled code of doEval method:
{code:java}
boolean var3 = false;
boolean var4 = false;
int var27 = this.vv1.getAccessor().get(inIndex);
boolean var5 = false;
int var28 = this.vv5.getAccessor().get(inIndex);
boolean var6 = false;
boolean var7 = false;
if (!Double.isNaN((double)var28) && (!Double.isNaN((double)var27) ||
!Double.isNaN((double)var28))) {
if (Double.isNaN((double)var27) && !Double.isNaN((double)var28)) {
var7 = true;
} else {
var7 = var27 > var28;
}
} else {
var7 = false;
}
int var26;
if (var7) {
var26 = var27;
} else {
var26 = var28;
}
this.vv10.getMutator().set(outIndex, var26);
boolean var8 = false;
boolean var9 = false;
int var29 = var26 - var26; // << Error reason. Bug somewhere in
ValueHolderReplacementVisitor.
boolean var10 = false;
boolean var11 = false;
int var30 = Math.abs(var29);
this.vv15.getMutator().set(outIndex, var30);
{code}
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zinoviev updated DRILL-6722:
-
Attachment: JaininoJava.class
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zinoviev updated DRILL-6722:
-
Attachment: JaininoJava2_merged.class
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6611) Add [meta]-[Enter] js handler for query form submission
[
https://issues.apache.org/jira/browse/DRILL-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669926#comment-16669926
]
ASF GitHub Bot commented on DRILL-6611:
---
arina-ielchiieva commented on issue #1392: DRILL-6611 : Enable meta-enter query
submission in web query interface
URL: https://github.com/apache/drill/pull/1392#issuecomment-434642295
@kkhatua since there is no reply, please go ahead and open alternative PR
but keep the commit with original author name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add [meta]-[Enter] js handler for query form submission
> ---
>
> Key: DRILL-6611
> URL: https://issues.apache.org/jira/browse/DRILL-6611
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.14.0
>Reporter: Bob Rudis
>Assignee: Bob Rudis
>Priority: Minor
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> The new ACE-based SQL query editor is great. Being able to submit the form
> without using a mouse would be even better.
> Adding:
>
> {noformat}
> document.getElementById('queryForm')
> .addEventListener('keydown', function(e) {
> if (!(e.keyCode == 13 && e.metaKey)) return;
> if (e.target.form) doSubmitQueryWithUserName();
> });
> {noformat}
> {{to ./exec/java-exec/src/main/resources/rest/query/query.ftl adds such
> support.}}
> I can file a PR with the code if desired.
> --
> Functionality (for the documentation):
> This JIRA's commit introduces the following to Drill:
> When composing queries in the web query editor it is now possible to submit
> the query text by using the {{Meta+Enter}} key combination. This will trigger
> the same action as pressing the {{Submit}} button. On Mac keyboards
> {{Meta+Enter}} is {{Cmd+Enter}}. On Windows or Linux is {{Ctrl+Enter}} though
> Linux users may have keymapped the {{Meta}} key to another physical keyboard
> key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-6611) Add [meta]-[Enter] js handler for query form submission
[
https://issues.apache.org/jira/browse/DRILL-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva reassigned DRILL-6611:
---
Assignee: Kunal Khatua (was: Bob Rudis)
> Add [meta]-[Enter] js handler for query form submission
> ---
>
> Key: DRILL-6611
> URL: https://issues.apache.org/jira/browse/DRILL-6611
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.14.0
>Reporter: Bob Rudis
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> The new ACE-based SQL query editor is great. Being able to submit the form
> without using a mouse would be even better.
> Adding:
>
> {noformat}
> document.getElementById('queryForm')
> .addEventListener('keydown', function(e) {
> if (!(e.keyCode == 13 && e.metaKey)) return;
> if (e.target.form) doSubmitQueryWithUserName();
> });
> {noformat}
> {{to ./exec/java-exec/src/main/resources/rest/query/query.ftl adds such
> support.}}
> I can file a PR with the code if desired.
> --
> Functionality (for the documentation):
> This JIRA's commit introduces the following to Drill:
> When composing queries in the web query editor it is now possible to submit
> the query text by using the {{Meta+Enter}} key combination. This will trigger
> the same action as pressing the {{Submit}} button. On Mac keyboards
> {{Meta+Enter}} is {{Cmd+Enter}}. On Windows or Linux is {{Ctrl+Enter}} though
> Linux users may have keymapped the {{Meta}} key to another physical keyboard
> key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669920#comment-16669920
]
ASF GitHub Bot commented on DRILL-5671:
---
arina-ielchiieva commented on a change in pull request #1467: DRILL-5671: Set
secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229643735
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderTemplate.java
##
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * Annotation for {@link ZKACLProviderDelegate} implementation to identify the
+ * implementation type. Implementation type is set in BOOT option
drill.exec.zk.acl_provider.
+ */
+@Retention(RetentionPolicy.RUNTIME)
+@Target({ElementType.TYPE})
+public @interface ZKACLProviderTemplate {
+ /**
+ * {@link ZKACLProviderDelegate} implementation type.
+ * @return
Review comment:
add description of the return, otherwise IDE shows warning.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669919#comment-16669919
]
ASF GitHub Bot commented on DRILL-5671:
---
arina-ielchiieva commented on a change in pull request #1467: DRILL-5671: Set
secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229642941
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderFactory.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.drill.common.config.DrillConfig;
+import static org.apache.drill.exec.ExecConstants.ZK_ACL_PROVIDER;
+import static org.apache.drill.exec.ExecConstants.ZK_APPLY_SECURE_ACL;
+
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.apache.drill.exec.exception.DrillbitStartupException;
+import org.apache.drill.exec.server.BootStrapContext;
+import org.apache.drill.shaded.guava.com.google.common.base.Strings;
+import java.lang.reflect.InvocationTargetException;
+
+import java.lang.reflect.Constructor;
+import java.util.Collection;
+
+/**
+ * This function returns a {@link ZKACLProviderDelegate} which will be used to
set ACLs on Drill ZK nodes
+ * If secure ACLs are required, the {@link ZKACLProviderFactory} looks up and
instantiates a {@link ZKACLProviderDelegate}
+ * specified in the config file. Else it returns the {@link
ZKDefaultACLProvider}
+ */
+public class ZKACLProviderFactory {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKACLProviderFactory.class);
+
+public static ZKACLProviderDelegate getACLProvider(DrillConfig config,
String drillClusterPath, BootStrapContext context)
+throws DrillbitStartupException {
+ZKACLContextProvider stateProvider = new
ZKACLContextProviderImpl(drillClusterPath);
+
+if (config.getBoolean(ZK_APPLY_SECURE_ACL)) {
+logger.trace("Using secure ZK ACL. Drill cluster path " +
drillClusterPath);
+ZKACLProviderDelegate aclProvider = findACLProvider(config,
stateProvider, context);
+return aclProvider;
+}
+logger.trace("Using un-secure default ZK ACL");
+final ZKDefaultACLProvider defaultAclProvider = new
ZKDefaultACLProvider(stateProvider);
+return new ZKACLProviderDelegate(defaultAclProvider);
+}
+
+public static ZKACLProviderDelegate findACLProvider(DrillConfig config,
ZKACLContextProvider contextProvider,
+BootStrapContext
context) throws DrillbitStartupException {
+if (!config.hasPath(ZK_ACL_PROVIDER)) {
+throw new DrillbitStartupException(String.format("BOOT option '%s'
is missing in config.", ZK_ACL_PROVIDER));
+}
+
+final String aclProviderName = config.getString(ZK_ACL_PROVIDER);
+
+if (Strings.isNullOrEmpty(aclProviderName)) {
+throw new DrillbitStartupException(String.format("Invalid value
'%s' for BOOT option '%s'", aclProviderName,
+ZK_ACL_PROVIDER));
+}
+
+ScanResult scan = context.getClasspathScan();
+final Collection> aclProviderImpls =
+scan.getImplementations(ZKACLProvider.class);
+logger.debug("Found ZkACLProvider implementations: {}",
aclProviderImpls);
+
+for (Class clazz : aclProviderImpls) {
+ final ZKACLProviderTemplate template =
clazz.getAnnotation(ZKACLProviderTemplate.class);
+ if (template == null) {
+logger.warn("{} doesn't have {} annotation. Skipping.",
clazz.getCanonicalName(),
+ZKACLProviderTemplate.class);
+continue;
+ }
+
+ if (template.type().equalsIgnoreCase(aclProviderName)) {
+Constructor validConstructor = null;
+Class constructorArgumentClass = ZKACLContextProvider.class;
+for (Constructor c : clazz.getConstructors()) {
+ Class[] params = c.getParameterTypes();
+ if (params.length
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669918#comment-16669918
]
ASF GitHub Bot commented on DRILL-5671:
---
arina-ielchiieva commented on a change in pull request #1467: DRILL-5671: Set
secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229642255
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/coord/zk/TestZKACL.java
##
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import com.typesafe.config.ConfigValueFactory;
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.api.ACLProvider;
+import org.apache.curator.retry.RetryNTimes;
+import org.apache.curator.test.TestingServer;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.scanner.ClassPathScanner;
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.server.BootStrapContext;
+import org.apache.drill.exec.server.options.SystemOptionManager;
+import org.apache.zookeeper.data.ACL;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import java.util.List;
+
+public class TestZKACL {
+
+ private TestingServer server;
+ private final static String cluster_config_znode =
"test-cluster_config_znode";
+ private final static byte[] cluster_config_data = "drill-node-1".getBytes();
+ private final static String drill_zk_root = "drill-test-drill_zk_root";
+ private final static String drill_cluster_name = "test-drillbits";
+ private static final String drillClusterPath = "/" + drill_zk_root + "/" +
drill_cluster_name;
+ private static final RetryPolicy retryPolicy = new RetryNTimes(1, 1000);
+
+ private static final String drillUDFName = "test-udfs";
+ private final static byte[] udf_data = "test-udf-1".getBytes();
+ private static final String drillUDFPath = "/" + drill_zk_root + "/" +
drillUDFName;
+ private static ACLProvider aclProviderDelegate;
+
+ private static CuratorFramework client;
+
+ @Before
+ public void setUp() throws Exception {
+System.setProperty("zookeeper.authProvider.1",
"org.apache.zookeeper.server.auth.SASLAuthenticationProvider");
+String configPath =
(ClassLoader.getSystemResource("zkacltest.conf")).getPath();
+System.setProperty("java.security.auth.login.config", configPath);
+server = new TestingServer();
+
+
+final DrillConfig config = new
DrillConfig(DrillConfig.create().withValue(ExecConstants.ZK_ACL_PROVIDER,
+ConfigValueFactory.fromAnyRef("creator-all")
+).withValue(ExecConstants.ZK_APPLY_SECURE_ACL,
ConfigValueFactory.fromAnyRef(true)));
+
+final ScanResult result = ClassPathScanner.fromPrescan(config);
+final BootStrapContext bootStrapContext =
+new BootStrapContext(config,
SystemOptionManager.createDefaultOptionDefinitions(), result);
+aclProviderDelegate = ZKACLProviderFactory.getACLProvider(config,
drillClusterPath, bootStrapContext);
+
+server.start();
+
+client = CuratorFrameworkFactory.builder().
+retryPolicy(retryPolicy).
+connectString(server.getConnectString()).
+aclProvider(aclProviderDelegate).
+build();
+client.start();
+ }
+
+ /**
+ * Test ACLs on znodes required to discover the cluster
+ */
+ // ZK libraries only supports one client instance per-machine per-server and
it is cached.
+ // This test will fail when run after other ZK tests that setup the client
in a way that will cause this test to fail.
+ @Ignore
+ @Test
+ public void testClusterDiscoveryPaths() {
+try {
+ String path = PathUtils.join(drillClusterPath, cluster_config_znode);
+ client.create().creatingParentsIfNeeded().forPath(path,
cluster_config_data);
+ List remoteACLs =
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669923#comment-16669923
]
ASF GitHub Bot commented on DRILL-5671:
---
arina-ielchiieva commented on a change in pull request #1467: DRILL-5671: Set
secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229643348
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderFactory.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.drill.common.config.DrillConfig;
+import static org.apache.drill.exec.ExecConstants.ZK_ACL_PROVIDER;
+import static org.apache.drill.exec.ExecConstants.ZK_APPLY_SECURE_ACL;
+
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.apache.drill.exec.exception.DrillbitStartupException;
+import org.apache.drill.exec.server.BootStrapContext;
+import org.apache.drill.shaded.guava.com.google.common.base.Strings;
+import java.lang.reflect.InvocationTargetException;
+
+import java.lang.reflect.Constructor;
+import java.util.Collection;
+
+/**
+ * This function returns a {@link ZKACLProviderDelegate} which will be used to
set ACLs on Drill ZK nodes
+ * If secure ACLs are required, the {@link ZKACLProviderFactory} looks up and
instantiates a {@link ZKACLProviderDelegate}
+ * specified in the config file. Else it returns the {@link
ZKDefaultACLProvider}
+ */
+public class ZKACLProviderFactory {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKACLProviderFactory.class);
Review comment:
private
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669921#comment-16669921
]
ASF GitHub Bot commented on DRILL-5671:
---
arina-ielchiieva commented on a change in pull request #1467: DRILL-5671: Set
secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229643252
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderDelegate.java
##
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.curator.framework.api.ACLProvider;
+import
org.apache.drill.shaded.guava.com.google.common.annotations.VisibleForTesting;
+import org.apache.zookeeper.data.ACL;
+
+import java.util.List;
+
+/**
+ * This class hides the {@link ZKACLProvider} from Curator-specific functions
Review comment:
Please add the reason why we need such hiding.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669922#comment-16669922
]
ASF GitHub Bot commented on DRILL-5671:
---
arina-ielchiieva commented on a change in pull request #1467: DRILL-5671: Set
secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229642632
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKSecureACLProvider.java
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.data.ACL;
+
+import java.util.List;
+
+/**
+ * ZKSecureACLProvider restricts access to znodes created by Drill in a secure
installation.
+ *
+ * The cluster discovery znode i.e. the znode containing the list of Drillbits
is
+ * readable by anyone.
+ *
+ * For all other znodes, only the creator of the znode, i.e the Drillbit user,
has full access.
+ *
+ */
+@ZKACLProviderTemplate(type = "creator-all")
+public class ZKSecureACLProvider implements ZKACLProvider {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKSecureACLProvider.class);
+
+/**
+ * DEFAULT_ACL gives the creator of a znode full-access to it
+ */
+static ImmutableList DEFAULT_ACL = new ImmutableList.Builder()
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+ .build();
+/**
+ * DRILL_CLUSTER_ACL gives the creator full access and everyone else only
read access.
+ * Used on the Drillbit discovery znode (znode path
//)
+ * i.e. the node that contains the list of Drillbits in the cluster.
+ */
+ static ImmutableList DRILL_CLUSTER_ACL = new
ImmutableList.Builder()
+
.addAll(Ids.READ_ACL_UNSAFE.iterator())
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+.build();
+final String drillClusterPath;
+
+public ZKSecureACLProvider(ZKACLContextProvider state) {
+this.drillClusterPath = state.getClusterPath();
+}
+
+@Override
+public List getDrillDefaultAcl() {
+return DEFAULT_ACL;
+}
+
+@Override
+public List getDrillAclForPath(String path) {
+logger.trace("getAclForPath " + path);
+if(path.startsWith(drillClusterPath)) {
Review comment:
space: `if (`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an
[jira] [Commented] (DRILL-6464) Disallow System.out, System.err, and Exception.printStackTrace
[
https://issues.apache.org/jira/browse/DRILL-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669924#comment-16669924
]
Volodymyr Vysotskyi commented on DRILL-6464:
It may be done by using
[{{forbiddenapis}}|https://github.com/policeman-tools/forbidden-apis] plugin.
We should define {{signaturesFile}} config, where will be specified classes and
methods which shouldn't be used.
As an example, may be used [this
commit|https://github.com/apache/drill/pull/1397/commits/dfe303956d73d9bb1703b9e497585079fd54949b].
> Disallow System.out, System.err, and Exception.printStackTrace
> --
>
> Key: DRILL-6464
> URL: https://issues.apache.org/jira/browse/DRILL-6464
> Project: Apache Drill
> Issue Type: Task
>Reporter: Timothy Farkas
>Priority: Major
>
> Add checkstyle rules to disallow using these print methods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6717) lower and upper functions not works with national charactes
[
https://issues.apache.org/jira/browse/DRILL-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669903#comment-16669903
]
ASF GitHub Bot commented on DRILL-6717:
---
arina-ielchiieva commented on issue #1450: DRILL-6717: lower and upper
functions not works with national characters
URL: https://github.com/apache/drill/pull/1450#issuecomment-434638067
@Ben-Zvi looks like Oleg changed the code. Is it ok now? Could you please
take a look.
@oleg-zinovev please do not forget to tag reviewers when you have made the
changes, otherwise nobody will know that you have addressed code review
comments.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> lower and upper functions not works with national charactes
> ---
>
> Key: DRILL-6717
> URL: https://issues.apache.org/jira/browse/DRILL-6717
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: result.csv, sample.json
>
>
> upper() and lower() functions works only with a ascii characters.
> sample dataset: [^sample.json]
> query:
> {code:sql}
> select
> upper(ascii_l) as upper_ascii,
> upper(national_l) as upper_national,
> lower(ascii_u) as lower_ascii,
> lower(national_u) as lower_national
> from dfs.`sample.json`
> {code}
> drill response: [^result.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-6349:
---
Labels: ready-to-commit (was: )
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)}}
> {{ at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)}}
> {{... 20 more}}
> apache drill 1.13.0
> "just drill it"
> 0: jdbc:drill:drillbit=localhost>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6817) Update to_number function to be consistent with CAST function
[
https://issues.apache.org/jira/browse/DRILL-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6817:
Labels: doc-impacting ready-to-commit (was: ready-to-commit)
> Update to_number function to be consistent with CAST function
> -
>
> Key: DRILL-6817
> URL: https://issues.apache.org/jira/browse/DRILL-6817
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.15.0
>
>
> {{In case when `drill.exec.functions.cast_empty_string_to_null` is enabled
> casting empty string ('') to numeric types will return NULL. If `to_number`
> is used to convert empty string to a number, UnsupportedOperationException
> will be thrown.}}
> The aim is to make these functions (CASTs and `to_number`) work consistently
> as is done for date/time functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6817) Update to_number function to be consistent with CAST function
[
https://issues.apache.org/jira/browse/DRILL-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6817:
Fix Version/s: 1.15.0
> Update to_number function to be consistent with CAST function
> -
>
> Key: DRILL-6817
> URL: https://issues.apache.org/jira/browse/DRILL-6817
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Minor
> Labels: ready-to-commit
> Fix For: 1.15.0
>
>
> {{In case when `drill.exec.functions.cast_empty_string_to_null` is enabled
> casting empty string ('') to numeric types will return NULL. If `to_number`
> is used to convert empty string to a number, UnsupportedOperationException
> will be thrown.}}
> The aim is to make these functions (CASTs and `to_number`) work consistently
> as is done for date/time functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6817) Update to_number function to be consistent with CAST function
[
https://issues.apache.org/jira/browse/DRILL-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6817:
Labels: ready-to-commit (was: )
> Update to_number function to be consistent with CAST function
> -
>
> Key: DRILL-6817
> URL: https://issues.apache.org/jira/browse/DRILL-6817
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.15.0
>
>
> {{In case when `drill.exec.functions.cast_empty_string_to_null` is enabled
> casting empty string ('') to numeric types will return NULL. If `to_number`
> is used to convert empty string to a number, UnsupportedOperationException
> will be thrown.}}
> The aim is to make these functions (CASTs and `to_number`) work consistently
> as is done for date/time functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6817) Update to_number function to be consistent with CAST function
[
https://issues.apache.org/jira/browse/DRILL-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6817:
Priority: Major (was: Minor)
> Update to_number function to be consistent with CAST function
> -
>
> Key: DRILL-6817
> URL: https://issues.apache.org/jira/browse/DRILL-6817
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.15.0
>
>
> {{In case when `drill.exec.functions.cast_empty_string_to_null` is enabled
> casting empty string ('') to numeric types will return NULL. If `to_number`
> is used to convert empty string to a number, UnsupportedOperationException
> will be thrown.}}
> The aim is to make these functions (CASTs and `to_number`) work consistently
> as is done for date/time functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6817) Update to_number function to be consistent with CAST function
[
https://issues.apache.org/jira/browse/DRILL-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6817:
Reviewer: Vitalii Diravka
> Update to_number function to be consistent with CAST function
> -
>
> Key: DRILL-6817
> URL: https://issues.apache.org/jira/browse/DRILL-6817
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.15.0
>
>
> {{In case when `drill.exec.functions.cast_empty_string_to_null` is enabled
> casting empty string ('') to numeric types will return NULL. If `to_number`
> is used to convert empty string to a number, UnsupportedOperationException
> will be thrown.}}
> The aim is to make these functions (CASTs and `to_number`) work consistently
> as is done for date/time functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6768) Improve to_date, to_time and to_timestamp and corresponding cast functions to handle empty string when `drill.exec.functions.cast_empty_string_to_null` option is enable
[
https://issues.apache.org/jira/browse/DRILL-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669848#comment-16669848
]
ASF GitHub Bot commented on DRILL-6768:
---
KazydubB commented on a change in pull request #1494: DRILL-6768: Improve
to_date, to_time and to_timestamp and correspondi…
URL: https://github.com/apache/drill/pull/1494#discussion_r229286194
##
File path:
logical/src/main/java/org/apache/drill/common/expression/fn/ToFunctions.java
##
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.common.expression.fn;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+
+public class ToFunctions {
+
+ private static Map TO_FUNC = new HashMap<>();
+ /** The to functions that need to be replaced (if
+ * "drill.exec.functions.cast_empty_string_to_null" is set to true). */
+ private static Set TO_FUNC_REPLACEMENT_NEEDED = new HashSet<>();
+ /** Map from the replaced functions to the new ones (for non-nullable
VARCHAR). */
+ private static Map FUNC_REPLACEMENT_FROM_NONNULLABLE_VARCHAR
= new HashMap<>();
+ /** Map from the replaced functions to the new ones (for non-nullable
VAR16CHAR). */
+ private static Map
FUNC_REPLACEMENT_FROM_NONNULLABLE_VAR16CHAR = new HashMap<>();
+ /** Map from the replaced functions to the new ones (for non-nullable
VARBINARY). */
+ private static Map
FUNC_REPLACEMENT_FROM_NONNULLABLE_VARBINARY = new HashMap<>();
+ /** Map from the replaced functions to the new ones (for nullable VARCHAR).
*/
+ private static Map FUNC_REPLACEMENT_FROM_NULLABLE_VARCHAR =
new HashMap<>();
+ /** Map from the replaced functions to the new ones (for nullable
VAR16CHAR). */
+ private static Map FUNC_REPLACEMENT_FROM_NULLABLE_VAR16CHAR
= new HashMap<>();
+ /** Map from the replaced functions to the new ones (for nullable
VARBINARY). */
+ private static Map FUNC_REPLACEMENT_FROM_NULLABLE_VARBINARY
= new HashMap<>();
+
+ static {
+TO_FUNC.put(MinorType.DATE, "to_date");
Review comment:
Filed issue: https://issues.apache.org/jira/browse/DRILL-6817. And resolved
it here subsequently.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Improve to_date, to_time and to_timestamp and corresponding cast functions to
> handle empty string when `drill.exec.functions.cast_empty_string_to_null`
> option is enabled
> -
>
> Key: DRILL-6768
> URL: https://issues.apache.org/jira/browse/DRILL-6768
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> When `drill.exec.functions.cast_empty_string_to_null` option is enabled
> `to_date`, `to_time` and `to_timestamp` functions while converting string to
> according type in case if null or empty string values are passed will return
> NULL (to avoid CASE clauses which are littering a query and will work in
> accordance with their respective CAST counterparts) for both cases.
>
>
>
> CASTs will be handled in a similar way (uniformly with numeric types):
>
> ||Value to cast||Now||Will be||
> |NULL|NULL|NULL|
> |'' (empty string)|Error in many cases (except numerical types)|NULL|
> CAST empty string to null (in case of enabled option) will be supported by
> DATE, TIME, TIMESTAMP, INTERVAL YEAR, INTERVAL MONTH and INTERVAL DAY
> functions in addition to numeric types.
>
> *For
