[jira] [Commented] (DRILL-6814) Query performance on S3 files
[
https://issues.apache.org/jira/browse/DRILL-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669597#comment-16669597
]
Kunal Khatua commented on DRILL-6814:
-
10K records every batch might not be too much, depending on what the average
row size is. If you can save the profile even after a 10-15 minute run, that
helps (though, a complete 30 min run will help most).
When the query completes (or is running after a reasonable amount of time), you
can add a {{.json}} to the URL to get the profile's JSON and save that to your
computer to share on this jira.
e.g.
from http://ashish.drill.com:8047/profiles/24270d0f-249a-128d-b007-1b4602cba4f4
save the link
http://ashish.drill.com:8047/profiles/24270d0f-249a-128d-b007-1b4602cba4f4.json
Also, it might be worth taking a stack trace of the Drillbits occassionally.
Luckily, 1.14 has an option to copy to clipboard on the /threads page.
e.g.
http://ashish.drill.com:8047/threads
Paste the contents to a file. You can do this every 30-60 secs... to crudely
sample what the Drillbit threads are doing. Sharing that would help narrow down
the issue as well.
> Query performance on S3 files
> -
>
> Key: DRILL-6814
> URL: https://issues.apache.org/jira/browse/DRILL-6814
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Other
>Affects Versions: 1.14.0
> Environment: Amazon EC2 instances-
> 4 Linux Redhat machines -version 7.5
> RAM- 32GB
>Reporter: Ashish Shukla
>Assignee: Robert Hou
>Priority: Major
>
> I have installed 4 Node drill cluster on Amazon EC2 and trying to execute a
> simple count on one Amazon S3 file. File type is CSV and size is approx- 14GB.
> The query returns expected count after the execution of approx 30 minutes.
> If we keep the same file in hdfs or create a table in postgres, execution
> time is relatively very less (approx 2-3 minutes).
> Is it normal behavior or something can be done for S3 files to make
> execution time comparable ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6814) Query performance on S3 files
[ https://issues.apache.org/jira/browse/DRILL-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669518#comment-16669518 ] Ashish Shukla edited comment on DRILL-6814 at 10/31/18 2:46 AM: [~kkhatua] I had to terminate the EC2 instances and don't have the stats right now.But one thing I observed while going though the profile that it had created too many batches(approx . 9000) and approx 10k record in every batch while reading in case of S3 files. was (Author: ashishkshukladb): [~kkhatua] I had to terminate the EC2 instances and don't have the stats right now > Query performance on S3 files > - > > Key: DRILL-6814 > URL: https://issues.apache.org/jira/browse/DRILL-6814 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.14.0 > Environment: Amazon EC2 instances- > 4 Linux Redhat machines -version 7.5 > RAM- 32GB >Reporter: Ashish Shukla >Assignee: Robert Hou >Priority: Major > > I have installed 4 Node drill cluster on Amazon EC2 and trying to execute a > simple count on one Amazon S3 file. File type is CSV and size is approx- 14GB. > The query returns expected count after the execution of approx 30 minutes. > If we keep the same file in hdfs or create a table in postgres, execution > time is relatively very less (approx 2-3 minutes). > Is it normal behavior or something can be done for S3 files to make > execution time comparable ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6814) Query performance on S3 files
[ https://issues.apache.org/jira/browse/DRILL-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669518#comment-16669518 ] Ashish Shukla commented on DRILL-6814: -- [~kkhatua] I had to terminate the EC2 instances and don't have the stats right now > Query performance on S3 files > - > > Key: DRILL-6814 > URL: https://issues.apache.org/jira/browse/DRILL-6814 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.14.0 > Environment: Amazon EC2 instances- > 4 Linux Redhat machines -version 7.5 > RAM- 32GB >Reporter: Ashish Shukla >Assignee: Robert Hou >Priority: Major > > I have installed 4 Node drill cluster on Amazon EC2 and trying to execute a > simple count on one Amazon S3 file. File type is CSV and size is approx- 14GB. > The query returns expected count after the execution of approx 30 minutes. > If we keep the same file in hdfs or create a table in postgres, execution > time is relatively very less (approx 2-3 minutes). > Is it normal behavior or something can be done for S3 files to make > execution time comparable ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[ https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669456#comment-16669456 ] ASF GitHub Bot commented on DRILL-6798: --- HanumathRao commented on issue #1514: DRILL-6798: Planner changes to support semi-join. URL: https://github.com/apache/drill/pull/1514#issuecomment-434519626 @amansinha100 Thank you for the review. I have addressed the review comments. @Ben-Zvi Thank you for the comment. I have made changes accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Planner changes to support semi-join > > > Key: DRILL-6798 > URL: https://issues.apache.org/jira/browse/DRILL-6798 > Project: Apache Drill > Issue Type: Sub-task > Components: Query Planning Optimization >Affects Versions: 1.14.0 >Reporter: Boaz Ben-Zvi >Assignee: Hanumath Rao Maduri >Priority: Major > Fix For: 1.15.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[
https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669455#comment-16669455
]
ASF GitHub Bot commented on DRILL-6798:
---
HanumathRao commented on a change in pull request #1514: DRILL-6798: Planner
changes to support semi-join.
URL: https://github.com/apache/drill/pull/1514#discussion_r229532642
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestSemiJoin.java
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.join;
+
+import org.apache.drill.categories.OperatorTest;
+import org.apache.drill.categories.SlowTest;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.test.BaseTestQuery;
+import org.apache.drill.test.ClientFixture;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.junit.Test;
+import static org.junit.Assert.assertTrue;
+import org.junit.experimental.categories.Category;
+
+@Category({SlowTest.class, OperatorTest.class})
+public class TestSemiJoin extends BaseTestQuery {
Review comment:
@amansinha100 Boaz tried out cases with in subqueries and normal joins.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner changes to support semi-join
>
>
> Key: DRILL-6798
> URL: https://issues.apache.org/jira/browse/DRILL-6798
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Boaz Ben-Zvi
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669422#comment-16669422
]
ASF GitHub Bot commented on DRILL-6792:
---
weijietong commented on a change in pull request #1504: DRILL-6792: Find the
right probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229519355
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java
##
@@ -379,11 +379,16 @@ public FragmentExecutor getFragmentRunner(final
FragmentHandle handle) {
return runningFragments.get(handle);
}
+/**
+ * receive the RuntimeFilter thorough the wire
+ * @param runtimeFilter
+ */
public void receiveRuntimeFilter(final RuntimeFilterWritable
runtimeFilter) {
BitData.RuntimeFilterBDef runtimeFilterDef =
runtimeFilter.getRuntimeFilterBDef();
boolean toForeman = runtimeFilterDef.getToForeman();
QueryId queryId = runtimeFilterDef.getQueryId();
String queryIdStr = QueryIdHelper.getQueryId(queryId);
+ runtimeFilter.retainBuffers(1);
Review comment:
The router uses AccountingDataTunel which holds a SendingAccounter to send
out the RF. The SendingAccounter acts as a blocking counter to book sending
status.The ForemanResult’s close method will call router’s waiForComplete
method which calls SendingAccounter’s synchronized method
waitForSendComplete.Since without receiving Ack.Ok and failing exception ,the
SendingAccounter’s batchSent counter only incremented while sending RF,no
chance to decrease it ,its waiForSendComplete method will be blocked,then
foreman hanged. The tricky problem here is no error message to see what
happened by making this change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6758) Hash Join should not return the join columns when they are not needed downstream
[ https://issues.apache.org/jira/browse/DRILL-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanumath Rao Maduri updated DRILL-6758: --- Fix Version/s: (was: 1.15.0) 1.16.0 > Hash Join should not return the join columns when they are not needed > downstream > > > Key: DRILL-6758 > URL: https://issues.apache.org/jira/browse/DRILL-6758 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators, Query Planning > Optimization >Affects Versions: 1.14.0 >Reporter: Boaz Ben-Zvi >Assignee: Hanumath Rao Maduri >Priority: Minor > Fix For: 1.16.0 > > > Currently the Hash-Join operator returns all its (both sides) incoming > columns. In cases where the join columns are not used further downstream, > this is a waste (allocating vectors, copying each value, etc). > Suggestion: Have the planner pass this information to the Hash-Join > operator, to enable skipping the return of these columns. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669381#comment-16669381
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r229501155
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java
##
@@ -379,11 +379,16 @@ public FragmentExecutor getFragmentRunner(final
FragmentHandle handle) {
return runningFragments.get(handle);
}
+/**
+ * receive the RuntimeFilter thorough the wire
+ * @param runtimeFilter
+ */
public void receiveRuntimeFilter(final RuntimeFilterWritable
runtimeFilter) {
BitData.RuntimeFilterBDef runtimeFilterDef =
runtimeFilter.getRuntimeFilterBDef();
boolean toForeman = runtimeFilterDef.getToForeman();
QueryId queryId = runtimeFilterDef.getQueryId();
String queryIdStr = QueryIdHelper.getQueryId(queryId);
+ runtimeFilter.retainBuffers(1);
Review comment:
I looked into the code. It's weird that you are seeing data tunnel being
hanged while sending runtimeFilter since for runtime filter it doesn't look
like we are using throttling listener based on semaphore count. If ACK is not
received the DataTunnel shouldn't hang while sending RuntimeFilter. It will
only hang while sending the DataBatches.
DataTunnel is created here for sending RuntimeFilter:
https://github.com/apache/drill/pull/1504/files#diff-86eb65fdf93f77f95838f885752e660cR191
In `tunnel.sendRuntimeFilter` I don't see anywhere `semaphore.acquire` is
called which will result in hang if Ack is not received.
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataTunnel.java#L94
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669316#comment-16669316
]
ASF GitHub Bot commented on DRILL-5671:
---
sohami commented on a change in pull request #1467: DRILL-5671: Set secure ACLs
(Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229471787
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKSecureACLProvider.java
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.data.ACL;
+
+import java.util.List;
+
+/**
+ * ZKSecureACLProvider restricts access to znodes created by Drill in a secure
installation.
+ *
+ * The cluster discovery znode i.e. the znode containing the list of Drillbits
is
+ * readable by anyone.
+ *
+ * For all other znodes, only the creator of the znode, i.e the Drillbit user,
has full access.
+ *
+ */
+@ZKACLProviderTemplate(type = "creator-all")
+public class ZKSecureACLProvider implements ZKACLProvider {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKSecureACLProvider.class);
+
+/**
+ * DEFAULT_ACL gives the creator of a znode full-access to it
+ */
+static ImmutableList DEFAULT_ACL = new ImmutableList.Builder()
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+ .build();
+/**
+ * DRILL_CLUSTER_ACL gives the creator full access and everyone else only
read access.
+ * Used on the Drillbit discovery znode (znode path
//)
+ * i.e. the node that contains the list of Drillbits in the cluster.
+ */
+ static ImmutableList DRILL_CLUSTER_ACL = new
ImmutableList.Builder()
+
.addAll(Ids.READ_ACL_UNSAFE.iterator())
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+.build();
+final String drillClusterPath;
+
+public ZKSecureACLProvider(ZKACLContextProvider state) {
Review comment:
rename **state** to **context**.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669312#comment-16669312
]
ASF GitHub Bot commented on DRILL-5671:
---
sohami commented on a change in pull request #1467: DRILL-5671: Set secure ACLs
(Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229476798
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderFactory.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.drill.common.config.DrillConfig;
+import static org.apache.drill.exec.ExecConstants.ZK_ACL_PROVIDER;
+import static org.apache.drill.exec.ExecConstants.ZK_APPLY_SECURE_ACL;
+
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.apache.drill.exec.exception.DrillbitStartupException;
+import org.apache.drill.exec.server.BootStrapContext;
+import org.apache.drill.shaded.guava.com.google.common.base.Strings;
+import java.lang.reflect.InvocationTargetException;
+
+import java.lang.reflect.Constructor;
+import java.util.Collection;
+
+/**
+ * This function returns a {@link ZKACLProviderDelegate} which will be used to
set ACLs on Drill ZK nodes
Review comment:
**function** to **factory class**
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669315#comment-16669315
]
ASF GitHub Bot commented on DRILL-5671:
---
sohami commented on a change in pull request #1467: DRILL-5671: Set secure ACLs
(Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229478256
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKSecureACLProvider.java
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.data.ACL;
+
+import java.util.List;
+
+/**
+ * ZKSecureACLProvider restricts access to znodes created by Drill in a secure
installation.
+ *
+ * The cluster discovery znode i.e. the znode containing the list of Drillbits
is
+ * readable by anyone.
+ *
+ * For all other znodes, only the creator of the znode, i.e the Drillbit user,
has full access.
+ *
+ */
+@ZKACLProviderTemplate(type = "creator-all")
+public class ZKSecureACLProvider implements ZKACLProvider {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKSecureACLProvider.class);
+
+/**
+ * DEFAULT_ACL gives the creator of a znode full-access to it
+ */
+static ImmutableList DEFAULT_ACL = new ImmutableList.Builder()
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+ .build();
+/**
+ * DRILL_CLUSTER_ACL gives the creator full access and everyone else only
read access.
+ * Used on the Drillbit discovery znode (znode path
//)
+ * i.e. the node that contains the list of Drillbits in the cluster.
+ */
+ static ImmutableList DRILL_CLUSTER_ACL = new
ImmutableList.Builder()
Review comment:
this and above - private static final ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669314#comment-16669314
]
ASF GitHub Bot commented on DRILL-5671:
---
sohami commented on a change in pull request #1467: DRILL-5671: Set secure ACLs
(Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229473228
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/coord/zk/TestZKACL.java
##
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import com.typesafe.config.ConfigValueFactory;
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.api.ACLProvider;
+import org.apache.curator.retry.RetryNTimes;
+import org.apache.curator.test.TestingServer;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.scanner.ClassPathScanner;
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.server.BootStrapContext;
+import org.apache.drill.exec.server.options.SystemOptionManager;
+import org.apache.zookeeper.data.ACL;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import java.util.List;
+
+public class TestZKACL {
+
+ private TestingServer server;
+ private final static String cluster_config_znode =
"test-cluster_config_znode";
+ private final static byte[] cluster_config_data = "drill-node-1".getBytes();
+ private final static String drill_zk_root = "drill-test-drill_zk_root";
+ private final static String drill_cluster_name = "test-drillbits";
+ private static final String drillClusterPath = "/" + drill_zk_root + "/" +
drill_cluster_name;
+ private static final RetryPolicy retryPolicy = new RetryNTimes(1, 1000);
+
+ private static final String drillUDFName = "test-udfs";
+ private final static byte[] udf_data = "test-udf-1".getBytes();
+ private static final String drillUDFPath = "/" + drill_zk_root + "/" +
drillUDFName;
+ private static ACLProvider aclProviderDelegate;
+
+ private static CuratorFramework client;
+
+ @Before
+ public void setUp() throws Exception {
+System.setProperty("zookeeper.authProvider.1",
"org.apache.zookeeper.server.auth.SASLAuthenticationProvider");
+String configPath =
(ClassLoader.getSystemResource("zkacltest.conf")).getPath();
+System.setProperty("java.security.auth.login.config", configPath);
+server = new TestingServer();
+
+
+final DrillConfig config = new
DrillConfig(DrillConfig.create().withValue(ExecConstants.ZK_ACL_PROVIDER,
+ConfigValueFactory.fromAnyRef("creator-all")
+).withValue(ExecConstants.ZK_APPLY_SECURE_ACL,
ConfigValueFactory.fromAnyRef(true)));
+
+final ScanResult result = ClassPathScanner.fromPrescan(config);
+final BootStrapContext bootStrapContext =
+new BootStrapContext(config,
SystemOptionManager.createDefaultOptionDefinitions(), result);
+aclProviderDelegate = ZKACLProviderFactory.getACLProvider(config,
drillClusterPath, bootStrapContext);
+
+server.start();
+
+client = CuratorFrameworkFactory.builder().
+retryPolicy(retryPolicy).
+connectString(server.getConnectString()).
+aclProvider(aclProviderDelegate).
+build();
+client.start();
+ }
+
+ /**
+ * Test ACLs on znodes required to discover the cluster
+ */
+ // ZK libraries only supports one client instance per-machine per-server and
it is cached.
+ // This test will fail when run after other ZK tests that setup the client
in a way that will cause this test to fail.
+ @Ignore
+ @Test
Review comment:
Please _Ignore_ at class level. Also provide category for this test as
_SecurityTest.class_. It would be great if you can open a JIRA listing the
reason why this tests are ignored and refer that JIRA# in the Ignore tag.
Please see
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[ https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669317#comment-16669317 ] ASF GitHub Bot commented on DRILL-6798: --- Ben-Zvi commented on a change in pull request #1514: DRILL-6798: Planner changes to support semi-join. URL: https://github.com/apache/drill/pull/1514#discussion_r229480249 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/planner/RuleInstance.java ## @@ -17,7 +17,9 @@ */ package org.apache.drill.exec.planner; +import com.google.common.base.Preconditions; Review comment: Need to add the "shaded guava" prefix: `org.apache.drill.shaded.guava.` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Planner changes to support semi-join > > > Key: DRILL-6798 > URL: https://issues.apache.org/jira/browse/DRILL-6798 > Project: Apache Drill > Issue Type: Sub-task > Components: Query Planning Optimization >Affects Versions: 1.14.0 >Reporter: Boaz Ben-Zvi >Assignee: Hanumath Rao Maduri >Priority: Major > Fix For: 1.15.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[ https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669313#comment-16669313 ] ASF GitHub Bot commented on DRILL-5671: --- sohami commented on a change in pull request #1467: DRILL-5671: Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster URL: https://github.com/apache/drill/pull/1467#discussion_r229477955 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKSecureACLProvider.java ## @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.coord.zk; + +import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList; +import org.apache.zookeeper.ZooDefs.Ids; +import org.apache.zookeeper.data.ACL; + +import java.util.List; + +/** + * ZKSecureACLProvider restricts access to znodes created by Drill in a secure installation. + * Review comment: Below is an example of what different nodes ACL's will be. But in general we should mention here that all znodes under cluster path with have read-all permission whereas other znodes will have creator-only access. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster > > > Key: DRILL-5671 > URL: https://issues.apache.org/jira/browse/DRILL-5671 > Project: Apache Drill > Issue Type: New Feature > Components: Server >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e. > anyone gets to do CDRWA(create, delete, read, write, admin access). This > means that even on a secure cluster anyone can perform all CRDWA actions on > the znodes. > This should be changed such that: > - In a non-secure cluster, Drill will continue using the current default > [world:all] ACL > - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the > authenticated user that created the znode gets full access. The discovery > znodes i.e. the znodes with the list of Drillbits will have an additional > [world:read] ACL, i.e. the list of Drillbits will be readable by anyone. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669311#comment-16669311
]
ASF GitHub Bot commented on DRILL-5671:
---
sohami commented on a change in pull request #1467: DRILL-5671: Set secure ACLs
(Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229478870
##
File path: exec/java-exec/src/main/resources/drill-module.conf
##
@@ -123,7 +125,9 @@ drill.exec: {
retry: {
count: 7200,
delay: 500
-}
+},
+apply_secure_acl: false,
+acl_provider: "creator-all"
Review comment:
Please add these new configuration in **drill-override-example.conf** with a
comment defining supported values of both configuration and what it means.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (DRILL-4897) NumberFormatException in Drill SQL while casting to BIGINT when its actually a number
[ https://issues.apache.org/jira/browse/DRILL-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthikeyan Manivannan resolved DRILL-4897. --- Resolution: Not A Problem > NumberFormatException in Drill SQL while casting to BIGINT when its actually > a number > - > > Key: DRILL-4897 > URL: https://issues.apache.org/jira/browse/DRILL-4897 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Reporter: Srihari Karanth >Assignee: Karthikeyan Manivannan >Priority: Blocker > Fix For: 1.15.0 > > > In the following SQL, drill cribs when trying to convert a number which is in > varchar >select cast (case IsNumeric(Delta_Radio_Delay) > when 0 then 0 else Delta_Radio_Delay end as BIGINT) > from datasource.`./sometable` > where Delta_Radio_Delay='4294967294'; > BIGINT should be able to take very large number. I dont understand how it > throws the below error: > 0: jdbc:drill:> select cast (case IsNumeric(Delta_Radio_Delay) > when 0 then 0 else Delta_Radio_Delay end as BIGINT) > from datasource.`./sometable` > where Delta_Radio_Delay='4294967294'; > Error: SYSTEM ERROR: NumberFormatException: 4294967294 > Fragment 1:29 > [Error Id: a63bb113-271f-4d8b-8194-2c9728543200 on cluster-3:31010] > (state=,code=0) > How can i modify SQL to fix this? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-4897) NumberFormatException in Drill SQL while casting to BIGINT when its actually a number
[ https://issues.apache.org/jira/browse/DRILL-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669292#comment-16669292 ] Karthikeyan Manivannan commented on DRILL-4897: --- This works as expected. When Codegen is being done for the IfExpression corresponding to the Case, then the Codegen realizes that the type of IfExpression (Int) and ElseExpression(Varchar) are different. ResolverTypePrecedence.precedenceMap specifies the precedence order for type conversions and it clearly indicates that Varchar -> Int is a valid conversion but not the other way round. So as part of the If-Else type resolution, Codegen consults the precedenceMap and casts the Else-type(Varchar) to the If-type (Int). The user can avoid this error by explicitly casting the ElseExpression to be BigInt. _SELECT CAST(case isnumeric(columns[0]) WHEN 0 THEN 0 ELSE CAST(columns[0] as BigInt) END AS BIGINT) from ..._ > NumberFormatException in Drill SQL while casting to BIGINT when its actually > a number > - > > Key: DRILL-4897 > URL: https://issues.apache.org/jira/browse/DRILL-4897 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Reporter: Srihari Karanth >Assignee: Karthikeyan Manivannan >Priority: Blocker > Fix For: 1.15.0 > > > In the following SQL, drill cribs when trying to convert a number which is in > varchar >select cast (case IsNumeric(Delta_Radio_Delay) > when 0 then 0 else Delta_Radio_Delay end as BIGINT) > from datasource.`./sometable` > where Delta_Radio_Delay='4294967294'; > BIGINT should be able to take very large number. I dont understand how it > throws the below error: > 0: jdbc:drill:> select cast (case IsNumeric(Delta_Radio_Delay) > when 0 then 0 else Delta_Radio_Delay end as BIGINT) > from datasource.`./sometable` > where Delta_Radio_Delay='4294967294'; > Error: SYSTEM ERROR: NumberFormatException: 4294967294 > Fragment 1:29 > [Error Id: a63bb113-271f-4d8b-8194-2c9728543200 on cluster-3:31010] > (state=,code=0) > How can i modify SQL to fix this? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6814) Query performance on S3 files
[ https://issues.apache.org/jira/browse/DRILL-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669218#comment-16669218 ] Kunal Khatua commented on DRILL-6814: - [~ashishkshukladb] can you share the query profiles? What do the profiles show the bottlenecks for two systems? > Query performance on S3 files > - > > Key: DRILL-6814 > URL: https://issues.apache.org/jira/browse/DRILL-6814 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.14.0 > Environment: Amazon EC2 instances- > 4 Linux Redhat machines -version 7.5 > RAM- 32GB >Reporter: Ashish Shukla >Assignee: Robert Hou >Priority: Major > > I have installed 4 Node drill cluster on Amazon EC2 and trying to execute a > simple count on one Amazon S3 file. File type is CSV and size is approx- 14GB. > The query returns expected count after the execution of approx 30 minutes. > If we keep the same file in hdfs or create a table in postgres, execution > time is relatively very less (approx 2-3 minutes). > Is it normal behavior or something can be done for S3 files to make > execution time comparable ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-6814) Query performance on S3 files
[ https://issues.apache.org/jira/browse/DRILL-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua reassigned DRILL-6814: --- Assignee: Robert Hou (was: Arina Ielchiieva) > Query performance on S3 files > - > > Key: DRILL-6814 > URL: https://issues.apache.org/jira/browse/DRILL-6814 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.14.0 > Environment: Amazon EC2 instances- > 4 Linux Redhat machines -version 7.5 > RAM- 32GB >Reporter: Ashish Shukla >Assignee: Robert Hou >Priority: Major > > I have installed 4 Node drill cluster on Amazon EC2 and trying to execute a > simple count on one Amazon S3 file. File type is CSV and size is approx- 14GB. > The query returns expected count after the execution of approx 30 minutes. > If we keep the same file in hdfs or create a table in postgres, execution > time is relatively very less (approx 2-3 minutes). > Is it normal behavior or something can be done for S3 files to make > execution time comparable ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6819) Remove invisible "back" link on query results page in Drill WebUI
[ https://issues.apache.org/jira/browse/DRILL-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-6819: Fix Version/s: 1.15.0 > Remove invisible "back" link on query results page in Drill WebUI > - > > Key: DRILL-6819 > URL: https://issues.apache.org/jira/browse/DRILL-6819 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.15.0 >Reporter: Denys Ordynskiy >Assignee: Denys Ordynskiy >Priority: Minor > Fix For: 1.15.0 > > Attachments: image.png > > > In Drill WebUI on page with query results and some other pages we have > invisible link with text - "back" and URL - "/queries". Since we have visible > link "Query" on navigation tab with the same URL, we don't need "back" link > on query results page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
[
https://issues.apache.org/jira/browse/DRILL-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6816:
--
Description:
Concurrent query execution from JDBC program using PreparedStatement results in
NPE.
Queries that were executed concurrently are (part of a query file),
{noformat}
select id from `test_tbl.json`
select count(id) from `test_tbl.json`
select count(*) from `test_tbl.json`
select * from `test_tbl.json`
{noformat}
Drill 1.14.0
git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
MapR version => 6.1.0.20180911143226.GA (secure cluster)
JDBC driver used was org.apache.drill.jdbc.Driver
Executing the above queries concurrently using a Statement object results in
successful query execution.
{noformat}
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
{noformat}
However, when the same queries listed above are executed using a
PreparedStatement object we see an NPE
{noformat}
PreparedStatement prdstmnt = conn.prepareStatement(query);
prdstmnt.executeUpdate();
{noformat}
Stack trace from drillbit.log
{noformat}
17:04:32.941 [pool-1-thread-3] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@35757005
17:04:32.941 [pool-1-thread-2] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d4413b8
17:04:32.956 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@5eb3b9ab
17:04:32.956 [pool-1-thread-4] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d9367d0
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.executeQuery(RunQuery.java:61)
at RunQuery.run(RunQuery.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:
Concurrent query execution from JDBC program using PreparedStatement results in
NPE.
Queries that were executed concurrently are (part of a query file),
{noformat}
select id from `test_tbl.json`
select count(id) from `test_tbl.json`
select count(*) from `test_tbl.json`
select * from `test_tbl.json`
{noformat}
Drill 1.14.0
git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
MapR version => 6.1.0.20180911143226.GA (secure cluster)
Executing the above queries concurrently using a Statement object results in
successful query execution.
{noformat}
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
{noformat}
However, when the same queries listed above are executed using a
PreparedStatement object we see an NPE
{noformat}
PreparedStatement prdstmnt = conn.prepareStatement(query);
prdstmnt.executeUpdate();
{noformat}
Stack trace from drillbit.log
{noformat}
17:04:32.941 [pool-1-thread-3] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@35757005
17:04:32.941 [pool-1-thread-2] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d4413b8
17:04:32.956 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@5eb3b9ab
17:04:32.956 [pool-1-thread-4] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d9367d0
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
[jira] [Updated] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
[
https://issues.apache.org/jira/browse/DRILL-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6816:
--
Description:
Concurrent query execution from JDBC program using PreparedStatement results in
NPE.
Queries that were executed concurrently are (part of a query file),
{noformat}
select id from `test_tbl.json`
select count(id) from `test_tbl.json`
select count(*) from `test_tbl.json`
select * from `test_tbl.json`
{noformat}
Drill 1.14.0
git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
MapR version => 6.1.0.20180911143226.GA (secure cluster)
Executing the above queries concurrently using a Statement object results in
successful query execution.
{noformat}
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
{noformat}
However, when the same queries listed above are executed using a
PreparedStatement object we see an NPE
{noformat}
PreparedStatement prdstmnt = conn.prepareStatement(query);
prdstmnt.executeUpdate();
{noformat}
Stack trace from drillbit.log
{noformat}
17:04:32.941 [pool-1-thread-3] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@35757005
17:04:32.941 [pool-1-thread-2] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d4413b8
17:04:32.956 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@5eb3b9ab
17:04:32.956 [pool-1-thread-4] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d9367d0
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.executeQuery(RunQuery.java:61)
at RunQuery.run(RunQuery.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:
Concurrent query execution from JDBC program using PreparedStatement results in
NPE.
Queries that were executed concurrently are (part of a query file),
{noformat}
select id from `test_tbl.json`
select count(id) from `test_tbl.json`
select count(*) from `test_tbl.json`
select * from `test_tbl.json`
{noformat}
Drill 1.14.0
git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
MapR version => 6.1.0.20180911143226.GA (secure cluster)
Executing the above queries concurrently using a Statement object results in
successful query execution.
{noformat}
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
{noformat}
However, when the same queries listed above are executed using a
PreparedStatement object we see an NPE
{noformat}
PreparedStatement prdstmnt = conn.prepareStatement(query);
prdstmnt.executeUpdate();
{noformat}
Stack trace from drillbit.log
{noformat}
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.executeQuery(RunQuery.java:61)
at RunQuery.run(RunQuery.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669081#comment-16669081
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229407973
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLStateProvider.java
##
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+/**
+ * Defines the functions required by {@link ZKACLProvider} to access ZK-ACL
related information
+ */
+
+public interface ZKACLStateProvider {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[ https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669082#comment-16669082 ] ASF GitHub Bot commented on DRILL-5671: --- bitblender commented on a change in pull request #1467: DRILL-5671: Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster URL: https://github.com/apache/drill/pull/1467#discussion_r229408012 ## File path: exec/jdbc-all/pom.xml ## @@ -511,7 +511,7 @@ This is likely due to you adding new dependencies to a java-exec and not updating the excludes in this module. This is important as it minimizes the size of the dependency of Drill application users. - 3900 + 3905 Review comment: Yes This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster > > > Key: DRILL-5671 > URL: https://issues.apache.org/jira/browse/DRILL-5671 > Project: Apache Drill > Issue Type: New Feature > Components: Server >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e. > anyone gets to do CDRWA(create, delete, read, write, admin access). This > means that even on a secure cluster anyone can perform all CRDWA actions on > the znodes. > This should be changed such that: > - In a non-secure cluster, Drill will continue using the current default > [world:all] ACL > - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the > authenticated user that created the znode gets full access. The discovery > znodes i.e. the znodes with the list of Drillbits will have an additional > [world:read] ACL, i.e. the list of Drillbits will be readable by anyone. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669078#comment-16669078
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229406948
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -50,6 +50,8 @@ private ExecConstants() {
public static final String ZK_TIMEOUT = "drill.exec.zk.timeout";
public static final String ZK_ROOT = "drill.exec.zk.root";
public static final String ZK_REFRESH = "drill.exec.zk.refresh";
+ public static final String ZK_ACL_PROVIDER = "drill.exec.zk.acl_provider";
+ public static final String ZK_APPLY_SECURE_ACL =
"drill.exec.zk.apply_secure_acl";
Review comment:
This is to maintain backward-compatibility with existing conf files.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669073#comment-16669073
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229405387
##
File path: exec/java-exec/src/main/resources/drill-module.conf
##
@@ -123,7 +123,9 @@ drill.exec: {
retry: {
count: 7200,
delay: 500
-}
+},
+apply_secure_acl: false,
+acl_provider: "org.apache.drill.exec.coord.zk.ZKSecureACLProvider"
Review comment:
This has been changed to use a provider-name instead of a class name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669072#comment-16669072
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r229405146
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProvider.java
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.curator.framework.api.ACLProvider;
+import org.apache.zookeeper.data.ACL;
+
+import java.util.List;
+
+/**
+ * This class defines the methods that a {@link ZKACLProvider} has to provide
+ * and it hides {@link ZKACLProvider}s from Curator-specific functions
+ */
Review comment:
As discussed offline, this is to make the ACLProvider to be agnostic of how
Drill communicates with ZK.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[
https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669050#comment-16669050
]
ASF GitHub Bot commented on DRILL-6798:
---
amansinha100 commented on a change in pull request #1514: DRILL-6798: Planner
changes to support semi-join.
URL: https://github.com/apache/drill/pull/1514#discussion_r229378653
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/JoinPrel.java
##
@@ -139,4 +158,57 @@ protected void buildJoinConditions(List
conditions,
}
}
+ public boolean isSemiJoin() {
+return isSemiJoin;
+ }
+
+ @Override public boolean isValid(Litmus litmus, Context context) {
Review comment:
Pls add comments about why this function had to be duplicated from the base
class and what change was needed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner changes to support semi-join
>
>
> Key: DRILL-6798
> URL: https://issues.apache.org/jira/browse/DRILL-6798
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Boaz Ben-Zvi
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[
https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669054#comment-16669054
]
ASF GitHub Bot commented on DRILL-6798:
---
amansinha100 commented on a change in pull request #1514: DRILL-6798: Planner
changes to support semi-join.
URL: https://github.com/apache/drill/pull/1514#discussion_r229388255
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestSemiJoin.java
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.join;
+
+import org.apache.drill.categories.OperatorTest;
+import org.apache.drill.categories.SlowTest;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.test.BaseTestQuery;
+import org.apache.drill.test.ClientFixture;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.junit.Test;
+import static org.junit.Assert.assertTrue;
+import org.junit.experimental.categories.Category;
+
+@Category({SlowTest.class, OperatorTest.class})
+public class TestSemiJoin extends BaseTestQuery {
Review comment:
Have you tried semi-join in conjunction with normal inner and outer joins ?
Either through unit test or manually. The join order should not change since
your new rule is applied after the LOPT optimizer but just to make sure it
works as expected. TPC-H queries that have such predicates (e.g q18 in
`TestTpchExplain.java`) may be good candidates to try.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner changes to support semi-join
>
>
> Key: DRILL-6798
> URL: https://issues.apache.org/jira/browse/DRILL-6798
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Boaz Ben-Zvi
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[
https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669057#comment-16669057
]
ASF GitHub Bot commented on DRILL-6798:
---
amansinha100 commented on a change in pull request #1514: DRILL-6798: Planner
changes to support semi-join.
URL: https://github.com/apache/drill/pull/1514#discussion_r229376473
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillJoin.java
##
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rex.RexNode;
+import org.apache.drill.exec.planner.common.DrillRelNode;
+import java.util.List;
+
+/**
+ * Interface which needs to be implemented by all the join relation
expressions.
+ */
+public interface DrillJoin extends DrillRelNode {
+
+ /*Columns of left table that are part of join condition */
Review comment:
Minor code-style..pls leave a space after the comment begin marker.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner changes to support semi-join
>
>
> Key: DRILL-6798
> URL: https://issues.apache.org/jira/browse/DRILL-6798
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Boaz Ben-Zvi
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[
https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669055#comment-16669055
]
ASF GitHub Bot commented on DRILL-6798:
---
amansinha100 commented on a change in pull request #1514: DRILL-6798: Planner
changes to support semi-join.
URL: https://github.com/apache/drill/pull/1514#discussion_r229380386
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashJoinPrel.java
##
@@ -50,14 +51,20 @@
private int joinControl;
public HashJoinPrel(RelOptCluster cluster, RelTraitSet traits, RelNode left,
RelNode right, RexNode condition,
- JoinRelType joinType) throws InvalidRelException {
-this(cluster, traits, left, right, condition, joinType, false, null,
false, JoinControl.DEFAULT);
+ JoinRelType joinType, boolean semiJoin) throws
InvalidRelException {
+this(cluster, traits, left, right, condition, joinType, false, null,
false, JoinControl.DEFAULT, semiJoin);
+ }
+
+ public HashJoinPrel(RelOptCluster cluster, RelTraitSet traits, RelNode left,
RelNode right, RexNode condition,
+ JoinRelType joinType, boolean swapped, RuntimeFilterDef
runtimeFilterDef,
+ boolean isRowKeyJoin, int joinControl) throws
InvalidRelException {
+this(cluster, traits, left, right, condition, joinType, swapped,
runtimeFilterDef, isRowKeyJoin, joinControl, false);
}
public HashJoinPrel(RelOptCluster cluster, RelTraitSet traits, RelNode left,
RelNode right, RexNode condition,
JoinRelType joinType, boolean swapped, RuntimeFilterDef runtimeFilterDef,
- boolean isRowKeyJoin, int joinControl) throws InvalidRelException {
-super(cluster, traits, left, right, condition, joinType);
+ boolean isRowKeyJoin, int joinControl, boolean semiJoin) throws
InvalidRelException {
Review comment:
Somewhere there should be a check that if `semiJoin` is `true` then
`swapped` should be `false`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner changes to support semi-join
>
>
> Key: DRILL-6798
> URL: https://issues.apache.org/jira/browse/DRILL-6798
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Boaz Ben-Zvi
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[
https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669052#comment-16669052
]
ASF GitHub Bot commented on DRILL-6798:
---
amansinha100 commented on a change in pull request #1514: DRILL-6798: Planner
changes to support semi-join.
URL: https://github.com/apache/drill/pull/1514#discussion_r229383277
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillSemiJoinRel.java
##
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinInfo;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.SemiJoin;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.ImmutableIntList;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.FieldReference;
+import org.apache.drill.common.logical.data.Join;
+import org.apache.drill.common.logical.data.JoinCondition;
+import org.apache.drill.common.logical.data.LogicalOperator;
+import org.apache.drill.common.logical.data.LogicalSemiJoin;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class DrillSemiJoinRel extends SemiJoin implements DrillJoin, DrillRel {
+
+ public DrillSemiJoinRel(
+ RelOptCluster cluster,
+ RelTraitSet traitSet,
+ RelNode left,
+ RelNode right,
+ RexNode condition,
+ ImmutableIntList leftKeys,
+ ImmutableIntList rightKeys) {
+super(cluster,
+ traitSet,
+ left,
+ right,
+ condition,
+ leftKeys,
+ rightKeys);
+ }
+
+ public static SemiJoin create(RelNode left, RelNode right, RexNode condition,
Review comment:
Since the `DrillSemiJoinRel` is allowing multi-column semi-joins, it would
be good to add unit test for it. Example:
.. WHERE (a1, b1) IN (SELECT a2, b2 FROM ...)
This should convert to a 2 column hash join.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner changes to support semi-join
>
>
> Key: DRILL-6798
> URL: https://issues.apache.org/jira/browse/DRILL-6798
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Boaz Ben-Zvi
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[
https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669056#comment-16669056
]
ASF GitHub Bot commented on DRILL-6798:
---
amansinha100 commented on a change in pull request #1514: DRILL-6798: Planner
changes to support semi-join.
URL: https://github.com/apache/drill/pull/1514#discussion_r229399354
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillSemiJoinRel.java
##
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinInfo;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.SemiJoin;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.ImmutableIntList;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.FieldReference;
+import org.apache.drill.common.logical.data.Join;
+import org.apache.drill.common.logical.data.JoinCondition;
+import org.apache.drill.common.logical.data.LogicalOperator;
+import org.apache.drill.common.logical.data.LogicalSemiJoin;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class DrillSemiJoinRel extends SemiJoin implements DrillJoin, DrillRel {
+
+ public DrillSemiJoinRel(
+ RelOptCluster cluster,
+ RelTraitSet traitSet,
Review comment:
Does the caller ensure that the `traitSet` for a SemiJoin only has the
collation and distribution properties from the left side of the join ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner changes to support semi-join
>
>
> Key: DRILL-6798
> URL: https://issues.apache.org/jira/browse/DRILL-6798
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Boaz Ben-Zvi
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[
https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669053#comment-16669053
]
ASF GitHub Bot commented on DRILL-6798:
---
amansinha100 commented on a change in pull request #1514: DRILL-6798: Planner
changes to support semi-join.
URL: https://github.com/apache/drill/pull/1514#discussion_r229398184
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillSemiJoinRel.java
##
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinInfo;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.SemiJoin;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.ImmutableIntList;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.FieldReference;
+import org.apache.drill.common.logical.data.Join;
+import org.apache.drill.common.logical.data.JoinCondition;
+import org.apache.drill.common.logical.data.LogicalOperator;
+import org.apache.drill.common.logical.data.LogicalSemiJoin;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class DrillSemiJoinRel extends SemiJoin implements DrillJoin, DrillRel {
+
+ public DrillSemiJoinRel(
+ RelOptCluster cluster,
+ RelTraitSet traitSet,
+ RelNode left,
+ RelNode right,
+ RexNode condition,
+ ImmutableIntList leftKeys,
+ ImmutableIntList rightKeys) {
+super(cluster,
+ traitSet,
+ left,
+ right,
+ condition,
+ leftKeys,
+ rightKeys);
+ }
+
+ public static SemiJoin create(RelNode left, RelNode right, RexNode condition,
+ImmutableIntList leftKeys, ImmutableIntList
rightKeys) {
+final RelOptCluster cluster = left.getCluster();
+return new DrillSemiJoinRel(cluster,
cluster.traitSetOf(DrillRel.DRILL_LOGICAL), left,
+right, condition, leftKeys, rightKeys);
+ }
+
+ @Override
+ public SemiJoin copy(RelTraitSet traitSet, RexNode condition,
+ RelNode left, RelNode right, JoinRelType
joinType, boolean semiJoinDone) {
+assert joinType == JoinRelType.INNER;
+final JoinInfo joinInfo = JoinInfo.of(left, right, condition);
+assert joinInfo.isEqui();
Review comment:
Asserts are only enabled in debug builds..best to use `Preconditions`
checks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner changes to support semi-join
>
>
> Key: DRILL-6798
> URL: https://issues.apache.org/jira/browse/DRILL-6798
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Boaz Ben-Zvi
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6798) Planner changes to support semi-join
[
https://issues.apache.org/jira/browse/DRILL-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669051#comment-16669051
]
ASF GitHub Bot commented on DRILL-6798:
---
amansinha100 commented on a change in pull request #1514: DRILL-6798: Planner
changes to support semi-join.
URL: https://github.com/apache/drill/pull/1514#discussion_r229376072
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillJoinRelBase.java
##
@@ -45,7 +46,7 @@
/**
* Base class for logical and physical Joins implemented in Drill.
*/
-public abstract class DrillJoinRelBase extends Join implements DrillRelNode {
+public abstract class DrillJoinRelBase extends Join implements DrillRelNode,
DrillJoin {
Review comment:
Since `DrillJoin` already extends `DrillRelNode`, you can remove
`DrillRelNode` from here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner changes to support semi-join
>
>
> Key: DRILL-6798
> URL: https://issues.apache.org/jira/browse/DRILL-6798
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Boaz Ben-Zvi
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6820) Implement a msgpack reader with zstandard codec
jean-claude created DRILL-6820: -- Summary: Implement a msgpack reader with zstandard codec Key: DRILL-6820 URL: https://issues.apache.org/jira/browse/DRILL-6820 Project: Apache Drill Issue Type: New Feature Reporter: jean-claude -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6814) Query performance on S3 files
[ https://issues.apache.org/jira/browse/DRILL-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6814: Fix Version/s: (was: 1.15.0) > Query performance on S3 files > - > > Key: DRILL-6814 > URL: https://issues.apache.org/jira/browse/DRILL-6814 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.14.0 > Environment: Amazon EC2 instances- > 4 Linux Redhat machines -version 7.5 > RAM- 32GB >Reporter: Ashish Shukla >Assignee: Arina Ielchiieva >Priority: Major > > I have installed 4 Node drill cluster on Amazon EC2 and trying to execute a > simple count on one Amazon S3 file. File type is CSV and size is approx- 14GB. > The query returns expected count after the execution of approx 30 minutes. > If we keep the same file in hdfs or create a table in postgres, execution > time is relatively very less (approx 2-3 minutes). > Is it normal behavior or something can be done for S3 files to make > execution time comparable ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6819) Remove invisible "back" link on query results page in Drill WebUI
[ https://issues.apache.org/jira/browse/DRILL-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6819: Priority: Minor (was: Major) > Remove invisible "back" link on query results page in Drill WebUI > - > > Key: DRILL-6819 > URL: https://issues.apache.org/jira/browse/DRILL-6819 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.15.0 >Reporter: Denys Ordynskiy >Assignee: Denys Ordynskiy >Priority: Minor > Attachments: image.png > > > In Drill WebUI on page with query results and some other pages we have > invisible link with text - "back" and URL - "/queries" > Since we have visible link "Query" on navigation tab with the same URL, > we don't need "back" link on query results page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-6425) Upgrade mapr release version
[
https://issues.apache.org/jira/browse/DRILL-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka reassigned DRILL-6425:
--
Assignee: Aman Sinha (was: Bohdan Kazydub)
> Upgrade mapr release version
>
>
> Key: DRILL-6425
> URL: https://issues.apache.org/jira/browse/DRILL-6425
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Aman Sinha
>Priority: Major
> Fix For: 1.15.0
>
>
> Upgrade MapR dependendencies from {{5.2.1}} to {{6.0.1}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6819) Remove invisible "back" link on query results page in Drill WebUI
[ https://issues.apache.org/jira/browse/DRILL-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Ordynskiy updated DRILL-6819: --- Affects Version/s: (was: 1.14.0) 1.15.0 > Remove invisible "back" link on query results page in Drill WebUI > - > > Key: DRILL-6819 > URL: https://issues.apache.org/jira/browse/DRILL-6819 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.15.0 >Reporter: Denys Ordynskiy >Assignee: Denys Ordynskiy >Priority: Major > Attachments: image.png > > > In Drill WebUI on page with query results and some other pages we have > invisible link with text - "back" and URL - "/queries" > Since we have visible link "Query" on navigation tab with the same URL, > we don't need "back" link on query results page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6819) Remove invisible "back" link on query results page in Drill WebUI
Denys Ordynskiy created DRILL-6819: -- Summary: Remove invisible "back" link on query results page in Drill WebUI Key: DRILL-6819 URL: https://issues.apache.org/jira/browse/DRILL-6819 Project: Apache Drill Issue Type: Bug Affects Versions: 1.14.0 Reporter: Denys Ordynskiy Assignee: Denys Ordynskiy Attachments: image.png In Drill WebUI on page with query results and some other pages we have invisible link with text - "back" and URL - "/queries" Since we have visible link "Query" on navigation tab with the same URL, we don't need "back" link on query results page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6717) lower and upper functions not works with national charactes
[
https://issues.apache.org/jira/browse/DRILL-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668691#comment-16668691
]
ASF GitHub Bot commented on DRILL-6717:
---
arina-ielchiieva commented on issue #1450: DRILL-6717: lower and upper
functions not works with national characters
URL: https://github.com/apache/drill/pull/1450#issuecomment-434299718
@oleg-zinovev is there any update?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> lower and upper functions not works with national charactes
> ---
>
> Key: DRILL-6717
> URL: https://issues.apache.org/jira/browse/DRILL-6717
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: result.csv, sample.json
>
>
> upper() and lower() functions works only with a ascii characters.
> sample dataset: [^sample.json]
> query:
> {code:sql}
> select
> upper(ascii_l) as upper_ascii,
> upper(national_l) as upper_national,
> lower(ascii_u) as lower_ascii,
> lower(national_u) as lower_national
> from dfs.`sample.json`
> {code}
> drill response: [^result.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6679) Error should be displayed when trying to connect Drill to unsupported version of Hive
[ https://issues.apache.org/jira/browse/DRILL-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bohdan Kazydub updated DRILL-6679: -- Fix Version/s: (was: 1.15.0) Future > Error should be displayed when trying to connect Drill to unsupported version > of Hive > - > > Key: DRILL-6679 > URL: https://issues.apache.org/jira/browse/DRILL-6679 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.14.0 >Reporter: Anton Gozhiy >Assignee: Bohdan Kazydub >Priority: Major > Fix For: Future > > > For example, there is no backward compatibility between Hive 2.3 and Hive > 2.1. But it is possible to connect Drill with Hive 2.3 client to Hive 2.1, it > just won't work correctly. So I suggest that enabling Hive storage plugin > should not be allowed if Hive version is unsupported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-4456:
Fix Version/s: 1.15.0
> Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6465) Transitive closure is not working in Drill for Join with multiple local conditions
[
https://issues.apache.org/jira/browse/DRILL-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated DRILL-6465:
---
Fix Version/s: (was: Future)
1.16.0
> Transitive closure is not working in Drill for Join with multiple local
> conditions
> --
>
> Key: DRILL-6465
> URL: https://issues.apache.org/jira/browse/DRILL-6465
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Denys Ordynskiy
>Assignee: Vitalii Diravka
>Priority: Minor
> Fix For: 1.16.0
>
> Attachments: drill.zip
>
>
> For several SQL operators Transitive closure is not working during Partition
> Pruning and Filter Pushdown for the left table in Join.
> If I use several local conditions, then Drill scans full left table in Join.
> But if we move additional conditions to the WHERE statement, then Transitive
> closure works fine for all joined tables
> *Query BETWEEN:*
> {code:java}
> EXPLAIN PLAN FOR
> SELECT * FROM hive.`h_tab1` t1
> JOIN hive.`h_tab2` t2
> ON t1.y=t2.y
> AND t2.y BETWEEN 1987 AND 1988;
> {code}
> *Expected result:*
> {code:java}
> Scan(groupscan=[HiveScan [table=Table(dbName:default, tableName:h_tab1),
> columns=[`**`], numPartitions=8, partitions= [Partition(values:[1987, 5, 1]),
> Partition(values:[1987, 5, 2]), Partition(values:[1987, 7, 1]),
> Partition(values:[1987, 7, 2]), Partition(values:[1988, 11, 1]),
> Partition(values:[1988, 11, 2]), Partition(values:[1988, 12, 1]),
> Partition(values:[1988, 12, 2])]{code}
> *Actual result:*
> {code:java}
> Scan(groupscan=[HiveScan [table=Table(dbName:default, tableName:h_tab1),
> columns=[`**`], numPartitions=16, partitions= [Partition(values:[1987, 5,
> 1]), Partition(values:[1987, 5, 2]), Partition(values:[1987, 7, 1]),
> Partition(values:[1987, 7, 2]), Partition(values:[1988, 11, 1]),
> Partition(values:[1988, 11, 2]), Partition(values:[1988, 12, 1]),
> Partition(values:[1988, 12, 2]), Partition(values:[1990, 4, 1]),
> Partition(values:[1990, 4, 2]), Partition(values:[1990, 5, 1]),
> Partition(values:[1990, 5, 2]), Partition(values:[1991, 3, 1]),
> Partition(values:[1991, 3, 2]), Partition(values:[1991, 3, 3]),
> Partition(values:[1991, 3, 4])
> ]
> {code}
> *There is the same Transitive closure behavior for this logical operators:*
> * NOT IN
> * LIKE
> * NOT LIKE
> Also Transitive closure is not working during Partition Pruning and Filter
> Pushdown for this comparison operators:
> *Query <*
> {code:java}
> EXPLAIN PLAN FOR
> SELECT * FROM hive.`h_tab1` t1
> JOIN hive.`h_tab2` t2
> ON t1.y=t2.y
> AND t2.y < 1988;
> {code}
> *Expected result:*
> {code:java}
> Scan(groupscan=[HiveScan [table=Table(dbName:default, tableName:h_tab1),
> columns=[`**`], numPartitions=4, partitions= [Partition(values:[1987, 5, 1]),
> Partition(values:[1987, 5, 2]), Partition(values:[1987, 7, 1]),
> Partition(values:[1987, 7, 2])]{code}
> *Actual result:*
> {code:java}
> 00-00 Screen
> 00-01 Project(itm=[$0], y=[$1], m=[$2], category=[$3], itm0=[$4],
> category0=[$5], y0=[$6], m0=[$7])
> 00-02 Project(itm=[$0], y=[$1], m=[$2], category=[$3], itm0=[$4],
> category0=[$5], y0=[$6], m0=[$7])
> 00-03 HashJoin(condition=[=($1, $6)], joinType=[inner])
> 00-05 Scan(groupscan=[HiveScan [table=Table(dbName:default,
> tableName:h_tab1), columns=[`**`], numPartitions=16, partitions=
> [Partition(values:[1987, 5, 1]), Partition(values:[1987, 5, 2]),
> Partition(values:[1987, 7, 1]), Partition(values:[1987, 7, 2]),
> Partition(values:[1988, 11, 1]), Partition(values:[1988, 11, 2]),
> Partition(values:[1988, 12, 1]), Partition(values:[1988, 12, 2]),
> Partition(values:[1990, 4, 1]), Partition(values:[1990, 4, 2]),
> Partition(values:[1990, 5, 1]), Partition(values:[1990, 5, 2]),
> Partition(values:[1991, 3, 1]), Partition(values:[1991, 3, 2]),
> Partition(values:[1991, 3, 3]), Partition(values:[1991, 3, 4])],
> inputDirectories=[maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/1,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/2,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/3,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/4,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/5,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/6,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/7,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/8,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/9,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/10,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/11,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/12,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/13,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/14,
>
[jira] [Updated] (DRILL-6465) Transitive closure is not working in Drill for Join with multiple local conditions
[
https://issues.apache.org/jira/browse/DRILL-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated DRILL-6465:
---
Fix Version/s: (was: 1.15.0)
Future
> Transitive closure is not working in Drill for Join with multiple local
> conditions
> --
>
> Key: DRILL-6465
> URL: https://issues.apache.org/jira/browse/DRILL-6465
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Denys Ordynskiy
>Assignee: Vitalii Diravka
>Priority: Minor
> Fix For: Future
>
> Attachments: drill.zip
>
>
> For several SQL operators Transitive closure is not working during Partition
> Pruning and Filter Pushdown for the left table in Join.
> If I use several local conditions, then Drill scans full left table in Join.
> But if we move additional conditions to the WHERE statement, then Transitive
> closure works fine for all joined tables
> *Query BETWEEN:*
> {code:java}
> EXPLAIN PLAN FOR
> SELECT * FROM hive.`h_tab1` t1
> JOIN hive.`h_tab2` t2
> ON t1.y=t2.y
> AND t2.y BETWEEN 1987 AND 1988;
> {code}
> *Expected result:*
> {code:java}
> Scan(groupscan=[HiveScan [table=Table(dbName:default, tableName:h_tab1),
> columns=[`**`], numPartitions=8, partitions= [Partition(values:[1987, 5, 1]),
> Partition(values:[1987, 5, 2]), Partition(values:[1987, 7, 1]),
> Partition(values:[1987, 7, 2]), Partition(values:[1988, 11, 1]),
> Partition(values:[1988, 11, 2]), Partition(values:[1988, 12, 1]),
> Partition(values:[1988, 12, 2])]{code}
> *Actual result:*
> {code:java}
> Scan(groupscan=[HiveScan [table=Table(dbName:default, tableName:h_tab1),
> columns=[`**`], numPartitions=16, partitions= [Partition(values:[1987, 5,
> 1]), Partition(values:[1987, 5, 2]), Partition(values:[1987, 7, 1]),
> Partition(values:[1987, 7, 2]), Partition(values:[1988, 11, 1]),
> Partition(values:[1988, 11, 2]), Partition(values:[1988, 12, 1]),
> Partition(values:[1988, 12, 2]), Partition(values:[1990, 4, 1]),
> Partition(values:[1990, 4, 2]), Partition(values:[1990, 5, 1]),
> Partition(values:[1990, 5, 2]), Partition(values:[1991, 3, 1]),
> Partition(values:[1991, 3, 2]), Partition(values:[1991, 3, 3]),
> Partition(values:[1991, 3, 4])
> ]
> {code}
> *There is the same Transitive closure behavior for this logical operators:*
> * NOT IN
> * LIKE
> * NOT LIKE
> Also Transitive closure is not working during Partition Pruning and Filter
> Pushdown for this comparison operators:
> *Query <*
> {code:java}
> EXPLAIN PLAN FOR
> SELECT * FROM hive.`h_tab1` t1
> JOIN hive.`h_tab2` t2
> ON t1.y=t2.y
> AND t2.y < 1988;
> {code}
> *Expected result:*
> {code:java}
> Scan(groupscan=[HiveScan [table=Table(dbName:default, tableName:h_tab1),
> columns=[`**`], numPartitions=4, partitions= [Partition(values:[1987, 5, 1]),
> Partition(values:[1987, 5, 2]), Partition(values:[1987, 7, 1]),
> Partition(values:[1987, 7, 2])]{code}
> *Actual result:*
> {code:java}
> 00-00 Screen
> 00-01 Project(itm=[$0], y=[$1], m=[$2], category=[$3], itm0=[$4],
> category0=[$5], y0=[$6], m0=[$7])
> 00-02 Project(itm=[$0], y=[$1], m=[$2], category=[$3], itm0=[$4],
> category0=[$5], y0=[$6], m0=[$7])
> 00-03 HashJoin(condition=[=($1, $6)], joinType=[inner])
> 00-05 Scan(groupscan=[HiveScan [table=Table(dbName:default,
> tableName:h_tab1), columns=[`**`], numPartitions=16, partitions=
> [Partition(values:[1987, 5, 1]), Partition(values:[1987, 5, 2]),
> Partition(values:[1987, 7, 1]), Partition(values:[1987, 7, 2]),
> Partition(values:[1988, 11, 1]), Partition(values:[1988, 11, 2]),
> Partition(values:[1988, 12, 1]), Partition(values:[1988, 12, 2]),
> Partition(values:[1990, 4, 1]), Partition(values:[1990, 4, 2]),
> Partition(values:[1990, 5, 1]), Partition(values:[1990, 5, 2]),
> Partition(values:[1991, 3, 1]), Partition(values:[1991, 3, 2]),
> Partition(values:[1991, 3, 3]), Partition(values:[1991, 3, 4])],
> inputDirectories=[maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/1,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/2,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/3,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/4,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/5,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/6,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/7,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/8,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/9,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/10,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/11,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/12,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/13,
> maprfs:/drill/testdata/ctas/parquet/DRILL_6173/tab1/14,
>
[jira] [Comment Edited] (DRILL-6642) Update protocol-buffers version
[
https://issues.apache.org/jira/browse/DRILL-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651579#comment-16651579
]
Vitalii Diravka edited comment on DRILL-6642 at 10/30/18 1:13 PM:
--
Some of the features of Proto2 that are not present in Proto3:
{quote}1. Required message fields. This can be useful for things like SOW
(State-of-the-World) keys.
2. Ability to set custom default values for a field.
3. Ability to determine if a missing field was not included, or was assigned
the default value. This is needed for AMPS to use delta messaging.
{quote}
The _required_ message fields are used in some proto files. It is necessary to
investigate how to replace it and whether _optional_ fit there.
"2" should not be the issue. I haven't seen custom default values in Drill.
Not sure about 3, so TBI in the process of updating.
Paul also mentioned in DRILL-5509:
{quote}This will require updating the Maven protobuf plugin, and may require
other upgrades as well.
{quote}
[~priteshm]
If all Proto2 vs Proto3 differences could be addressed the backward
compatibility is preserved.
Looks like the test cases should cover all proto messages in the process of
running _drillbit_ process and different connections to it.
The last issues related to _protobuf_ were displaying operators in Profiles
DRILL-6639, so until Drill UI is automated this part could be verified manually.
was (Author: vitalii):
Some of the features of Proto2 that are not present in Proto3:
{quote}1. Required message fields. This can be useful for things like SOW keys.
2. Ability to set custom default values for a field.
3. Ability to determine if a missing field was not included, or was assigned
the default value. This is needed for AMPS to use delta messaging.
{quote}
The _required_ message fields are used in some proto files. It is necessary to
investigate how to replace it and whether _optional_ fit there.
"2" should not be the issue. I haven't seen custom default values in Drill.
Not sure about 3, so TBI in the process of updating.
Paul also mentioned in DRILL-5509:
{quote}This will require updating the Maven protobuf plugin, and may require
other upgrades as well.
{quote}
[~priteshm]
If all Proto2 vs Proto3 differences could be addressed the backward
compatibility is preserved.
Looks like the test cases should cover all proto messages in the process of
running _drillbit_ process and different connections to it.
The last issues related to _protobuf_ were displaying operators in Profiles
DRILL-6639, so until Drill UI is automated this part could be verified manually.
> Update protocol-buffers version
> ---
>
> Key: DRILL-6642
> URL: https://issues.apache.org/jira/browse/DRILL-6642
> Project: Apache Drill
> Issue Type: Task
> Components: Tools, Build Test
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Major
> Fix For: 1.15.0
>
>
> Currently Drill uses 2.5.0 {{protocol-buffers}} version.
> The last version is 3.6.0 in maven repo:
> [https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java]
> The new version has a lot of useful enhancements, which can be used in Drill.
> One of them is using {{UNRECOGNIZED Enum NullValue}}, which can help to
> handle them in place of null values for {{ProtocolMessageEnum}} - DRILL-6639.
> Looks like the NullValue can be used instead of null returned from
> {{valueOf()}} (_or {{forNumber()}}, since {{valueOf()}} is deprecated in the
> newer protobuf version_):
>
> [https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/NullValue]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6741) CircleCI for Drill
[
https://issues.apache.org/jira/browse/DRILL-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated DRILL-6741:
---
Fix Version/s: (was: 1.15.0)
Future
> CircleCI for Drill
> --
>
> Key: DRILL-6741
> URL: https://issues.apache.org/jira/browse/DRILL-6741
> Project: Apache Drill
> Issue Type: Test
> Components: Tools, Build Test
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Major
> Fix For: Future
>
>
> There is a discussion regarding usage of CircleCI for Drill:
>
> [https://lists.apache.org/thread.html/4d8cfa074dd427ac4d495f8767752f9be1ce95cc82c51f767f7611e4@%3Cdev.drill.apache.org%3E]
> Current issue is limit of RAM for Docker images: 4Gb, but 8Gb is needed.
> Can be solved by using of CircleCI VM, the RAM limit in this case is 8Gb.
> Besides that I am looking to ask CircleCI team about providing {{large}}
> {{resource_class}} for dcoker images:
> https://circleci.com/docs/2.0/configuration-reference/#resource_class
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6643) OperatorType for Values and IteratorValidator
[
https://issues.apache.org/jira/browse/DRILL-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated DRILL-6643:
---
Fix Version/s: (was: 1.15.0)
Future
> OperatorType for Values and IteratorValidator
> -
>
> Key: DRILL-6643
> URL: https://issues.apache.org/jira/browse/DRILL-6643
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build Test
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Minor
> Fix For: Future
>
>
> {{getOperatorType()}} method of
> {{org.apache.drill.exec.physical.config.Values}} and
> {{org.apache.drill.exec.physical.config.IteratorValidator}} classes should be
> revised.
> For now it returns -1, but it could be better to return some internal type
> of operator or it should be unsupported for calling and should throw
> {{UnsupportedOperationException()}} at all.
> See: _TODO: DRILL-6643_ in the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668643#comment-16668643
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434277967
@vvysotskyi
> I think it is ok to change modifier to PRIVATE for inner and nested
classes.
The PRIVATE modifier has been removed, not added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)}}
> {{ at
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668631#comment-16668631
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev commented on issue #1446: DRILL-6349: Drill JDBC driver fails on
Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434277967
@vvysotskyi
> I think it is ok to change modifier to PRIVATE for inner and nested
classes.
I change modifier to PACKAGE LOCAL (by removing PRIVATE)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)}}
> {{ at
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668622#comment-16668622
]
ASF GitHub Bot commented on DRILL-6349:
---
vvysotskyi commented on a change in pull request #1446: DRILL-6349: Drill JDBC
driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#discussion_r229278004
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/compile/JDKClassCompiler.java
##
@@ -55,7 +55,7 @@ private JDKClassCompiler(JavaCompiler compiler, ClassLoader
classLoader, boolean
this.compiler = compiler;
this.listener = new DrillDiagnosticListener();
this.fileManager = new
DrillJavaFileManager(compiler.getStandardFileManager(listener, null,
Charsets.UTF_8), classLoader);
-this.compilerOptions = Lists.newArrayList(this.debug ?
"-g:source,lines,vars" : "-g:none");
+this.compilerOptions = Lists.newArrayList("-source", "1.8", "-target",
"1.8", this.debug ? "-g:source,lines,vars" : "-g:none");
Review comment:
I assume this change should be reverted
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
>
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668623#comment-16668623
]
ASF GitHub Bot commented on DRILL-6349:
---
vvysotskyi commented on a change in pull request #1446: DRILL-6349: Drill JDBC
driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#discussion_r229258681
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/compile/bytecode/ReplacingInterpreter.java
##
@@ -40,6 +40,7 @@
private final List valueList;
public ReplacingInterpreter(final String className, final
List valueList) {
+super(ASM5);
Review comment:
Please replace `ASM5` with `CompilationConfig.ASM_API_VERSION`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
>
[jira] [Commented] (DRILL-5550) SELECT non-existent column produces empty required VARCHAR
[
https://issues.apache.org/jira/browse/DRILL-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668606#comment-16668606
]
ASF GitHub Bot commented on DRILL-5550:
---
vdiravka commented on issue #939: DRILL-5550: Missing CSV column value set to
null
URL: https://github.com/apache/drill/pull/939#issuecomment-434269947
@prasadns14 Could you please respond to Paul's review comments?
[DRILL-6147](https://issues.apache.org/jira/browse/DRILL-6147) is resolved, so
it can also be helpful to proceed with this issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> SELECT non-existent column produces empty required VARCHAR
> --
>
> Key: DRILL-5550
> URL: https://issues.apache.org/jira/browse/DRILL-5550
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text CSV
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Prasad Nagaraj Subramanya
>Priority: Minor
> Fix For: Future
>
>
> Drill's CSV column reader supports two forms of files:
> * Files with column headers as the first line of the file.
> * Files without column headers.
> The CSV storage plugin specifies which format to use for files accessed via
> that storage plugin config.
> Suppose we have a CSV file with headers:
> {code}
> a,b,c
> 10,foo,bar
> {code}
> Suppose we configure a storage plugin to use headers:
> {code}
> TextFormatConfig csvFormat = new TextFormatConfig();
> csvFormat.fieldDelimiter = ',';
> csvFormat.skipFirstLine = false;
> csvFormat.extractHeader = true;
> {code}
> (The above can also be done using JSON when running Drill as a server.)
> Execute the following query:
> {code}
> SELECT a, c, d FROM `dfs.data.example.csv`
> {code}
> Results:
> {code}
> a,c,d
> 10,bar,
> {code}
> The actual type of column {{d}} is non-nullable VARCHAR.
> This is inconsistent with other parts of Drill in two ways, one may be a bug.
> Most other parts of Drill use a nullable INT for "missing" columns.
> 1. For CSV it makes sense for the data type to be VARCHAR, since all CSV
> columns are of that type.
> 2. It may *not* make sense for the column to be non-nullable and blank rather
> than nullable and NULL. In SQL, NULL means that the data is unknown, which is
> the case here.
> In the future, we may want to use some other indication for a missing column.
> Until then, the requested change is to make the type of a missing CSV column
> a nullable VARCHAR set to value NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-5550) SELECT non-existent column produces empty required VARCHAR
[
https://issues.apache.org/jira/browse/DRILL-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka reassigned DRILL-5550:
--
Assignee: Prasad Nagaraj Subramanya
> SELECT non-existent column produces empty required VARCHAR
> --
>
> Key: DRILL-5550
> URL: https://issues.apache.org/jira/browse/DRILL-5550
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text CSV
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Prasad Nagaraj Subramanya
>Priority: Minor
> Fix For: Future
>
>
> Drill's CSV column reader supports two forms of files:
> * Files with column headers as the first line of the file.
> * Files without column headers.
> The CSV storage plugin specifies which format to use for files accessed via
> that storage plugin config.
> Suppose we have a CSV file with headers:
> {code}
> a,b,c
> 10,foo,bar
> {code}
> Suppose we configure a storage plugin to use headers:
> {code}
> TextFormatConfig csvFormat = new TextFormatConfig();
> csvFormat.fieldDelimiter = ',';
> csvFormat.skipFirstLine = false;
> csvFormat.extractHeader = true;
> {code}
> (The above can also be done using JSON when running Drill as a server.)
> Execute the following query:
> {code}
> SELECT a, c, d FROM `dfs.data.example.csv`
> {code}
> Results:
> {code}
> a,c,d
> 10,bar,
> {code}
> The actual type of column {{d}} is non-nullable VARCHAR.
> This is inconsistent with other parts of Drill in two ways, one may be a bug.
> Most other parts of Drill use a nullable INT for "missing" columns.
> 1. For CSV it makes sense for the data type to be VARCHAR, since all CSV
> columns are of that type.
> 2. It may *not* make sense for the column to be non-nullable and blank rather
> than nullable and NULL. In SQL, NULL means that the data is unknown, which is
> the case here.
> In the future, we may want to use some other indication for a missing column.
> Until then, the requested change is to make the type of a missing CSV column
> a nullable VARCHAR set to value NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668594#comment-16668594
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434254256
@vvysotskyi
Sory, i was mistaken in
> This can be done by overriding SimpleJavaFileObject#getNestingKind, but it
will break java 11- compatibility.
NestingKind getNestingKind() exists since Java 1.6
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
>
[jira] [Updated] (DRILL-5550) SELECT non-existent column produces empty required VARCHAR
[
https://issues.apache.org/jira/browse/DRILL-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated DRILL-5550:
---
Fix Version/s: Future
> SELECT non-existent column produces empty required VARCHAR
> --
>
> Key: DRILL-5550
> URL: https://issues.apache.org/jira/browse/DRILL-5550
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text CSV
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Priority: Minor
> Fix For: Future
>
>
> Drill's CSV column reader supports two forms of files:
> * Files with column headers as the first line of the file.
> * Files without column headers.
> The CSV storage plugin specifies which format to use for files accessed via
> that storage plugin config.
> Suppose we have a CSV file with headers:
> {code}
> a,b,c
> 10,foo,bar
> {code}
> Suppose we configure a storage plugin to use headers:
> {code}
> TextFormatConfig csvFormat = new TextFormatConfig();
> csvFormat.fieldDelimiter = ',';
> csvFormat.skipFirstLine = false;
> csvFormat.extractHeader = true;
> {code}
> (The above can also be done using JSON when running Drill as a server.)
> Execute the following query:
> {code}
> SELECT a, c, d FROM `dfs.data.example.csv`
> {code}
> Results:
> {code}
> a,c,d
> 10,bar,
> {code}
> The actual type of column {{d}} is non-nullable VARCHAR.
> This is inconsistent with other parts of Drill in two ways, one may be a bug.
> Most other parts of Drill use a nullable INT for "missing" columns.
> 1. For CSV it makes sense for the data type to be VARCHAR, since all CSV
> columns are of that type.
> 2. It may *not* make sense for the column to be non-nullable and blank rather
> than nullable and NULL. In SQL, NULL means that the data is unknown, which is
> the case here.
> In the future, we may want to use some other indication for a missing column.
> Until then, the requested change is to make the type of a missing CSV column
> a nullable VARCHAR set to value NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-5970) DrillParquetReader always builds the schema with "OPTIONAL" dataMode columns instead of "REQUIRED" ones
[
https://issues.apache.org/jira/browse/DRILL-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated DRILL-5970:
---
Fix Version/s: Future
> DrillParquetReader always builds the schema with "OPTIONAL" dataMode columns
> instead of "REQUIRED" ones
> ---
>
> Key: DRILL-5970
> URL: https://issues.apache.org/jira/browse/DRILL-5970
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen, Execution - Data Types, Storage -
> Parquet
>Affects Versions: 1.11.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Major
> Fix For: Future
>
>
> The root cause of the issue is that adding REQUIRED (not-nullable) data types
> to the container in the all MapWriters is not implemented.
> It can lead to get invalid schema.
> {code}
> 0: jdbc:drill:zk=local> CREATE TABLE dfs.tmp.bof_repro_1 as select * from
> (select CONVERT_FROM('["hello","hai"]','JSON') AS MYCOL, 'Bucket1' AS Bucket
> FROM (VALUES(1)));
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 1 |
> +---++
> 1 row selected (2.376 seconds)
> {code}
> Run from Drill unit test framework (to see "data mode"):
> {code}
> @Test
> public void test() throws Exception {
> setColumnWidths(new int[] {25, 25});
> List queryDataBatches = testSqlWithResults("select * from
> dfs.tmp.bof_repro_1");
> printResult(queryDataBatches);
> }
> 1 row(s):
> ---
> | MYCOL | Bucket|
> ---
> | ["hello","hai"] | Bucket1 |
> ---
> Total record count: 1
> {code}
> {code}
> vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar
> parquet-tools-1.6.0rc3-SNAPSHOT.jar schema /tmp/bof_repro_1/0_0_0.parquet
> message root {
> repeated binary MYCOL (UTF8);
> required binary Bucket (UTF8);
> }
> {code}
> To simulate of obtaining the wrong result you can try the query with
> aggregation by using a new parquet reader (used by default for complex data
> types) and old parquet reader. False "Hash aggregate does not support schema
> changes" error will happen.
> 1) Create two parquet files.
> {code}
> 0: jdbc:drill:schema=dfs> CREATE TABLE dfs.tmp.bof_repro_1 as select * from
> (select CONVERT_FROM('["hello","hai"]','JSON') AS MYCOL, 'Bucket1' AS Bucket
> FROM (VALUES(1)));
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 1 |
> +---++
> 1 row selected (1.122 seconds)
> 0: jdbc:drill:schema=dfs> CREATE TABLE dfs.tmp.bof_repro_2 as select * from
> (select CONVERT_FROM('[]','JSON') AS MYCOL, 'Bucket1' AS Bucket FROM
> (VALUES(1)));
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 1 |
> +---++
> 1 row selected (0.552 seconds)
> 0: jdbc:drill:schema=dfs> select * from dfs.tmp.bof_repro_2;
> {code}
> 2) Copy the parquet files from bof_repro_1 to bof_repro_2.
> {code}
> [root@naravm1 ~]# hadoop fs -ls /tmp/bof_repro_1
> Found 1 items
> -rw-r--r-- 3 mapr mapr415 2017-07-25 11:46
> /tmp/bof_repro_1/0_0_0.parquet
> [root@naravm1 ~]# hadoop fs -ls /tmp/bof_repro_2
> Found 1 items
> -rw-r--r-- 3 mapr mapr368 2017-07-25 11:46
> /tmp/bof_repro_2/0_0_0.parquet
> [root@naravm1 ~]# hadoop fs -cp /tmp/bof_repro_1/0_0_0.parquet
> /tmp/bof_repro_2/0_0_1.parquet
> [root@naravm1 ~]#
> {code}
> 3) Query the table.
> {code}
> 0: jdbc:drill:schema=dfs> ALTER SESSION SET `planner.enable_streamagg`=false;
> +---++
> | ok | summary |
> +---++
> | true | planner.enable_streamagg updated. |
> +---++
> 1 row selected (0.124 seconds)
> 0: jdbc:drill:schema=dfs> select * from dfs.tmp.bof_repro_2;
> +--+--+
> | MYCOL | Bucket |
> +--+--+
> | ["hello","hai"] | Bucket1 |
> | null | Bucket1 |
> +--+--+
> 2 rows
[jira] [Commented] (DRILL-6215) Use prepared statement instead of Statement in JdbcRecordReader class
[
https://issues.apache.org/jira/browse/DRILL-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668587#comment-16668587
]
Vitalii Diravka commented on DRILL-6215:
[~khfaraaz] It may be useful to make it configurable. Possibly the type of
{{java.sql.Statement}} can be specified in the RDBMS Storage Plugin configs.
> Use prepared statement instead of Statement in JdbcRecordReader class
> -
>
> Key: DRILL-6215
> URL: https://issues.apache.org/jira/browse/DRILL-6215
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JDBC
>Affects Versions: 1.12.0
>Reporter: Khurram Faraaz
>Assignee: Khurram Faraaz
>Priority: Major
> Fix For: Future
>
>
> Use prepared statement instead of Statement in JdbcRecordReader class, which
> is more efficient and less vulnerable to SQL injection attacks.
> Apache Drill 1.13.0-SNAPSHOT, commit :
> 9073aed67d89e8b2188870d6c812706085c9c41b
> Findbugs reports the below bug and suggests that we use prepared statement
> instead of Statement.
> {noformat}
> In class org.apache.drill.exec.store.jdbc.JdbcRecordReader
> In method
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext,
> OutputMutator)
> At JdbcRecordReader.java:[line 170]
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext,
> OutputMutator) passes a nonconstant String to an execute method on an SQL
> statement
> The method invokes the execute method on an SQL statement with a String that
> seems to be dynamically generated.
> Consider using a prepared statement instead.
> It is more efficient and less vulnerable to SQL injection attacks.
> {noformat}
> LOC -
> https://github.com/apache/drill/blob/a9ea4ec1c5645ddab4b7aef9ac060ff5f109b696/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcRecordReader.java#L170
> {noformat}
> To run with findbugs:
> mvn clean install -Pfindbugs -DskipTests
> Findbugs will wirite the output to finbugsXml.html in the target directory of
> each module.
> For example the java-exec module report is located at:
> ./exec/java-exec/target/findbugs/findbugsXml.html
> Use
> find . -name "findbugsXml.html"
> to locate the files.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668584#comment-16668584
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434263357
@vvysotskyi
I managed to fix this for Java 11 by removing PRIVATE modifier from inner /
inner static classes in ClassGenerator. Is this acceptable?
P.S. I accidentally added a locale settings in surefire with last commit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
>
[jira] [Created] (DRILL-6817) Update to_number function to be consistent with CAST function
Bohdan Kazydub created DRILL-6817:
-
Summary: Update to_number function to be consistent with CAST
function
Key: DRILL-6817
URL: https://issues.apache.org/jira/browse/DRILL-6817
Project: Apache Drill
Issue Type: Improvement
Reporter: Bohdan Kazydub
Assignee: Bohdan Kazydub
{{In case when `drill.exec.functions.cast_empty_string_to_null` is enabled
casting empty string ('') to numeric types will return NULL. If `to_number` is
used to convert empty string to a number, UnsupportedOperationException will be
thrown.}}
The aim is to make these functions (CASTs and `to_number`) work consistently as
is done for date/time functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-6215) Use prepared statement instead of Statement in JdbcRecordReader class
[
https://issues.apache.org/jira/browse/DRILL-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka reassigned DRILL-6215:
--
Assignee: Khurram Faraaz
> Use prepared statement instead of Statement in JdbcRecordReader class
> -
>
> Key: DRILL-6215
> URL: https://issues.apache.org/jira/browse/DRILL-6215
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JDBC
>Affects Versions: 1.12.0
>Reporter: Khurram Faraaz
>Assignee: Khurram Faraaz
>Priority: Major
> Fix For: Future
>
>
> Use prepared statement instead of Statement in JdbcRecordReader class, which
> is more efficient and less vulnerable to SQL injection attacks.
> Apache Drill 1.13.0-SNAPSHOT, commit :
> 9073aed67d89e8b2188870d6c812706085c9c41b
> Findbugs reports the below bug and suggests that we use prepared statement
> instead of Statement.
> {noformat}
> In class org.apache.drill.exec.store.jdbc.JdbcRecordReader
> In method
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext,
> OutputMutator)
> At JdbcRecordReader.java:[line 170]
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext,
> OutputMutator) passes a nonconstant String to an execute method on an SQL
> statement
> The method invokes the execute method on an SQL statement with a String that
> seems to be dynamically generated.
> Consider using a prepared statement instead.
> It is more efficient and less vulnerable to SQL injection attacks.
> {noformat}
> LOC -
> https://github.com/apache/drill/blob/a9ea4ec1c5645ddab4b7aef9ac060ff5f109b696/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcRecordReader.java#L170
> {noformat}
> To run with findbugs:
> mvn clean install -Pfindbugs -DskipTests
> Findbugs will wirite the output to finbugsXml.html in the target directory of
> each module.
> For example the java-exec module report is located at:
> ./exec/java-exec/target/findbugs/findbugsXml.html
> Use
> find . -name "findbugsXml.html"
> to locate the files.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6215) Use prepared statement instead of Statement in JdbcRecordReader class
[
https://issues.apache.org/jira/browse/DRILL-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated DRILL-6215:
---
Fix Version/s: Future
> Use prepared statement instead of Statement in JdbcRecordReader class
> -
>
> Key: DRILL-6215
> URL: https://issues.apache.org/jira/browse/DRILL-6215
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JDBC
>Affects Versions: 1.12.0
>Reporter: Khurram Faraaz
>Priority: Major
> Fix For: Future
>
>
> Use prepared statement instead of Statement in JdbcRecordReader class, which
> is more efficient and less vulnerable to SQL injection attacks.
> Apache Drill 1.13.0-SNAPSHOT, commit :
> 9073aed67d89e8b2188870d6c812706085c9c41b
> Findbugs reports the below bug and suggests that we use prepared statement
> instead of Statement.
> {noformat}
> In class org.apache.drill.exec.store.jdbc.JdbcRecordReader
> In method
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext,
> OutputMutator)
> At JdbcRecordReader.java:[line 170]
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext,
> OutputMutator) passes a nonconstant String to an execute method on an SQL
> statement
> The method invokes the execute method on an SQL statement with a String that
> seems to be dynamically generated.
> Consider using a prepared statement instead.
> It is more efficient and less vulnerable to SQL injection attacks.
> {noformat}
> LOC -
> https://github.com/apache/drill/blob/a9ea4ec1c5645ddab4b7aef9ac060ff5f109b696/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcRecordReader.java#L170
> {noformat}
> To run with findbugs:
> mvn clean install -Pfindbugs -DskipTests
> Findbugs will wirite the output to finbugsXml.html in the target directory of
> each module.
> For example the java-exec module report is located at:
> ./exec/java-exec/target/findbugs/findbugsXml.html
> Use
> find . -name "findbugsXml.html"
> to locate the files.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668575#comment-16668575
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434263357
@vvysotskyi
I managed to fix this for Java 11 by removing PRIVATE modifier from inner /
inner static classes in ClassGenerator. Is this acceptable?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)}}
> {{ at
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668574#comment-16668574
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev commented on issue #1446: DRILL-6349: Drill JDBC driver fails on
Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434263357
@vvysotskyi
I managed to fix this by removing PRIVATE modifier from inner / inner static
classes in ClassGenerator.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)}}
> {{ at
[jira] [Commented] (DRILL-6246) Build Failing in jdbc-all artifact
[
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668573#comment-16668573
]
ASF GitHub Bot commented on DRILL-6246:
---
vdiravka commented on issue #1168: DRILL-6246: Reduced the size of the jdbc-all
jar file
URL: https://github.com/apache/drill/pull/1168#issuecomment-434263107
@rhou1 Is there public ticket for this task? Having this, we can rely on it
in other similar cases.
@krystaln @sachouche The only one case is necessary to be verified to merge
this PR to master branch.
It may also be helpful not to get blockers before Drill 1.15 release.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> * {color:#00}It was noticed that the build was failing because of the
> jdbc-all artifact{color}
> * {color:#00}The maximum compressed jar size was set to 32MB but we are
> currently creating a JAR a bit larger than 32MB {color}
> * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and
> drill-1.13.0 (on my MacOS){color}
> * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
> * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
> * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is
> roughly 32MB){color}
> * {color:#00}Compared then in more details jdbc-all-1.12.0 and
> jdbc-all-1.13.0{color}
> * {color:#00}The bulk of the increase is attributed to the calcite
> artifact{color}
> * {color:#00}Used to be 2MB (uncompressed) and now 22MB
> (uncompressed){color}
> * {color:#00}It is likely an exclusion problem {color}
> * {color:#00}The jdbc-all-1.12.0 version has only two top packages
> calcite/avatica/utils and calcite/avatica/remote{color}
> * {color:#00}The jdbc-all-1.13.0 includes new packages (within
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6223) Drill fails on Schema changes
[ https://issues.apache.org/jira/browse/DRILL-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668561#comment-16668561 ] ASF GitHub Bot commented on DRILL-6223: --- vdiravka commented on issue #1170: DRILL-6223: Fixed several Drillbit failures due to schema changes URL: https://github.com/apache/drill/pull/1170#issuecomment-434259969 @sachouche Could you please respond to review comments? Should the changes be revised? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drill fails on Schema changes > -- > > Key: DRILL-6223 > URL: https://issues.apache.org/jira/browse/DRILL-6223 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.10.0, 1.12.0 >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > > Drill Query Failing when selecting all columns from a Complex Nested Data > File (Parquet) Set). There are differences in Schema among the files: > * The Parquet files exhibit differences both at the first level and within > nested data types > * A select * will not cause an exception but using a limit clause will > * Note also this issue seems to happen only when multiple Drillbit minor > fragments are involved (concurrency higher than one) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[ https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668552#comment-16668552 ] ASF GitHub Bot commented on DRILL-6321: --- vdiravka commented on issue #1275: DRILL-6321: Lateral Join and Unnest - star column prefix URL: https://github.com/apache/drill/pull/1275#issuecomment-434258272 [DRILL-6321](https://issues.apache.org/jira/browse/DRILL-6321) is marked as resolved with 1.14 Drill version. But it involves several PRs. @chunhui-shi What is the purpose of that? Do we need reopen it or to open a new one for this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Lateral Join: Planning changes - enable submitting physical plan > > > Key: DRILL-6321 > URL: https://issues.apache.org/jira/browse/DRILL-6321 > Project: Apache Drill > Issue Type: Task >Reporter: Parth Chandra >Assignee: Chunhui Shi >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Implement changes to enable submitting a physical plan containing lateral and > unnest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6610) Add support for Minimum TLS support
[ https://issues.apache.org/jira/browse/DRILL-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668545#comment-16668545 ] ASF GitHub Bot commented on DRILL-6610: --- vdiravka commented on issue #1388: DRILL-6610: Add support for Minimum TLS restriction. URL: https://github.com/apache/drill/pull/1388#issuecomment-434256906 @superbstreak Could you please respond to review comments? In that case there these changes can be included to the Apache Drill 1.15 release version. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support for Minimum TLS support > --- > > Key: DRILL-6610 > URL: https://issues.apache.org/jira/browse/DRILL-6610 > Project: Apache Drill > Issue Type: Improvement > Components: Client - C++ >Affects Versions: 1.12.0 >Reporter: Rob Wu >Assignee: Rob Wu >Priority: Major > > Add support for minimum TLS support. > Currently, the TLSProtocol parameter only supports a specific version of TLS > to be used. > > Investigation: > Setting the default SSL context method to be sslv23 with default sslv2 and > sslv3 turned off would allow us to restrict the protocol to be TLS only. > Additional flags can be applied to further restrict the minimum TLS version: > For example: > Minimum TLS 1.0 - Sets NO_SSLv2 and NO_SSLv3 > Minimum TLS 1.1 - Sets NO_SSLv2 and NO SSLv3 and NO_TLSv1 > Minimum TLS 1.2 - Sets NO_SSLv2 and NO SSLv3 and NO_TLSv1 and NO_TLSv1_1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6806) Start moving code for handling a partition in HashAgg into a separate class
[ https://issues.apache.org/jira/browse/DRILL-6806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Diravka updated DRILL-6806: --- Fix Version/s: 1.15.0 > Start moving code for handling a partition in HashAgg into a separate class > --- > > Key: DRILL-6806 > URL: https://issues.apache.org/jira/browse/DRILL-6806 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.15.0 > > > Since this involves a lot of refactoring this will be a multiple PR effort. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668519#comment-16668519
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev commented on issue #1446: DRILL-6349: Drill JDBC driver fails on
Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434254256
@vvysotskyi
Sory, i was mistaken in
> This can be done by overriding SimpleJavaFileObject#getNestingKind, but it
will break java 11- compatibility.
NestingKind getNestingKind() exists from Java 1.6
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
>
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668521#comment-16668521
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434254256
@vvysotskyi
Sory, i was mistaken in
> This can be done by overriding SimpleJavaFileObject#getNestingKind, but it
will break java 11- compatibility.
NestingKind getNestingKind() exists from Java 1.6
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
>
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668515#comment-16668515
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434231490
@vvysotskyi
> Also, recently was released ASM 7.0, which supports JDK 11 features, so we
can update it and revert changes connected with specifying the target and
source versions in JDKClassCompiler.
There is an issue with java 11 JDK compiler and Drill code generation even
after asm version update. JDK compiler requires correctly specified "nest host"
for inner classes. So source and target version should be specified for JDK
compiler
Also, thanks for your note about HBase version. I had tried to update it to
1.1.11, but run into problems with Guava versions incompatibility.
P.S. For java 11 target support in JDK compiler
CompilationConfig#ASM_API_VERSION should be set to ASM7, but it can break
compatibility with Java 8. May be this version can be detected in runtime?
P.P.S. "Nest host" can be added with ASM ClassVisitor
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
>
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668509#comment-16668509
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434251463
@vvysotskyi
TestLargeFileCompilation.testEXTERNAL_SORT:153->BaseTestQuery.testNoResult:384
» Rpc
TestLargeFileCompilation.testPARQUET_WRITER:140->BaseTestQuery.testNoResult:384
» Rpc
TestLargeFileCompilation.testProject:172->BaseTestQuery.testNoResult:384 »
Rpc
TestLargeFileCompilation.testTOP_N_SORT:159->BaseTestQuery.testNoResult:384
» Rpc
Error text:
> Caused by: java.lang.IncompatibleClassChangeError: Type
org.apache.drill.exec.test.generated.ProjectorGen30258$ProjectorGen302580 is
not a nest member of org.apache.drill.exec.test.generated.ProjectorGen30258:
current type is not listed as a nest member
>at
org.apache.drill.exec.test.generated.ProjectorGen30258.__DRILL_INIT__(ProjectorTemplate.java)
~[na:na]
>at
org.apache.drill.exec.test.generated.ProjectorGen30258.(ProjectorTemplate.java:41)
~[na:na]
>at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[na:na]
>at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[na:na]
>at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[na:na]
>at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
~[na:na]
>at java.base/java.lang.Class.newInstance(Class.java:584) ~[na:na]
>at
org.apache.drill.exec.compile.CodeCompiler.createInstances(CodeCompiler.java:193)
~[classes/:na]
>at
org.apache.drill.exec.compile.CodeCompiler.createInstance(CodeCompiler.java:163)
~[classes/:na]
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
>
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[ https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668504#comment-16668504 ] ASF GitHub Bot commented on DRILL-6349: --- oleg-zinovev commented on issue #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM URL: https://github.com/apache/drill/pull/1446#issuecomment-434251463 @vvysotskyi TestLargeFileCompilation.testEXTERNAL_SORT:153->BaseTestQuery.testNoResult:384 » Rpc TestLargeFileCompilation.testPARQUET_WRITER:140->BaseTestQuery.testNoResult:384 » Rpc TestLargeFileCompilation.testProject:172->BaseTestQuery.testNoResult:384 » Rpc TestLargeFileCompilation.testTOP_N_SORT:159->BaseTestQuery.testNoResult:384 » Rpc Error text: Caused by: java.lang.IncompatibleClassChangeError: Type org.apache.drill.exec.test.generated.ProjectorGen30258$ProjectorGen302580 is not a nest member of org.apache.drill.exec.test.generated.ProjectorGen30258: current type is not listed as a nest member at org.apache.drill.exec.test.generated.ProjectorGen30258.__DRILL_INIT__(ProjectorTemplate.java) ~[na:na] at org.apache.drill.exec.test.generated.ProjectorGen30258.(ProjectorTemplate.java:41) ~[na:na] at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na] at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na] at java.base/java.lang.Class.newInstance(Class.java:584) ~[na:na] at org.apache.drill.exec.compile.CodeCompiler.createInstances(CodeCompiler.java:193) ~[classes/:na] at org.apache.drill.exec.compile.CodeCompiler.createInstance(CodeCompiler.java:163) ~[classes/:na] at org.apache.drill.exec.ops.BaseFragmentContext.getImplementationClass(BaseFragmentContext.java:56) ~[classes/:na] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchemaFromInput(ProjectRecordBatch.java:570) ~[classes/:na] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:583) ~[classes/:na] at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:101) ~[classes/:na] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:143) ~[classes/:na] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) ~[classes/:na] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:229) ~[classes/:na] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) ~[classes/:na] at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83) ~[classes/:na] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) ~[classes/:na] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:297) ~[classes/:na] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:284) ~[classes/:na] at java.base/java.security.AccessController.doPrivileged(Native Method) ~[na:na] at java.base/javax.security.auth.Subject.doAs(Subject.java:423) ~[na:na] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) ~[hadoop-common-2.7.4.jar:na] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:284) ~[classes/:na] This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM > --- > > Key: DRILL-6349 > URL: https://issues.apache.org/jira/browse/DRILL-6349 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.13.0 > Environment: 16:23 apache-drill-1.13.0$ uname -a > Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST > 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64 > >Reporter: Marc Prud'hommeaux >Assignee: Oleg Zinoviev >Priority: Major >
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668499#comment-16668499
]
ASF GitHub Bot commented on DRILL-6349:
---
vvysotskyi commented on issue #1446: DRILL-6349: Drill JDBC driver fails on
Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434249472
I think setting `ASM7` shouldn't break compatibility with Java 8, since it
wasn't reflected in the release notes. Could you please also specify tests that
fail due to the problem with "Nest host"?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
>
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668467#comment-16668467
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434231490
@vvysotskyi
> Also, recently was released ASM 7.0, which supports JDK 11 features, so we
can update it and revert changes connected with specifying the target and
source versions in JDKClassCompiler.
There is an issue with java 11 JDK compiler and Drill code generation even
after asm version update. JDK compiler requires correctly specified kind of
"nest" for inner classes. This can be done by overriding
SimpleJavaFileObject#getNestingKind, but it will break java 11- compatibility.
So source and target version should be specified for JDK compiler
Also, thanks for your note about HBase version. I had tried to update it to
1.1.11, but run into problems with Guava versions incompatibility.
P.S. For java 11 target support in JDK compiler
CompilationConfig#ASM_API_VERSION should be set to ASM7, but it can break
compatibility with Java 8. May be this version can be detected in runtime?
P.P.S. "Nest host" can be added with ASM ClassVisitor
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
>
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668463#comment-16668463
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434231490
@vvysotskyi
> Also, recently was released ASM 7.0, which supports JDK 11 features, so we
can update it and revert changes connected with specifying the target and
source versions in JDKClassCompiler.
There is an issue with java 11 JDK compiler and Drill code generation even
after asm version update. JDK compiler requires correctly specified kind of
"nest" for inner classes. This can be done by overriding
SimpleJavaFileObject#getNestingKind, but it will break java 11- compatibility.
So source and target version should be specified for JDK compiler
Also, thanks for your note about HBase version. I had tried to update it to
1.1.11, but run into problems with Guava versions incompatibility.
P.S. For java 11 JDK compiler support CompilationConfig#ASM_API_VERSION
should be set to ASM7, but it can break compatibility with Java 8. May be this
version can be detected in runtime?
P.P.S. "Nest host" can be added with ASM ClassVisitor
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668460#comment-16668460
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434231490
@vvysotskyi
> Also, recently was released ASM 7.0, which supports JDK 11 features, so we
can update it and revert changes connected with specifying the target and
source versions in JDKClassCompiler.
There is an issue with java 11 JDK compiler and Drill code generation even
after asm version update. JDK compiler requires correctly specified kind of
"nest" for inner classes. This can be done by overriding
SimpleJavaFileObject#getNestingKind, but it will break java 11- compatibility.
So source and target version should be specified for JDK compiler
Also, thanks for your note about HBase version. I had tried to update it to
1.1.11, but run into problems with Guava versions incompatibility.
P.S. For java 11 JDK compiler support CompilationConfig#ASM_API_VERSION
should be set to ASM7, but it can break compatibility with Java 8. May be this
version can be detected in runtime?
P.P.S. "Nestmates" can be added with ASM ClassVisitor
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668451#comment-16668451
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434231490
@vvysotskyi
> Also, recently was released ASM 7.0, which supports JDK 11 features, so we
can update it and revert changes connected with specifying the target and
source versions in JDKClassCompiler.
There is an issue with java 11 JDK compiler and Drill code generation even
after asm version update. JDK compiler requires you yo correctly specify the
kind of "nest" for inner classes. This can be done by overriding
SimpleJavaFileObject#getNestingKind, but it will break java 11- compatibility.
So source and target version should be specified for JDK compiler
Also, thanks for your note about HBase version. I had tried to update it to
1.1.11, but run into problems with Guava versions incompatibility.
P.S. May be "nestmates" can be added with ASM ClassVisitor?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668452#comment-16668452
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434231490
@vvysotskyi
> Also, recently was released ASM 7.0, which supports JDK 11 features, so we
can update it and revert changes connected with specifying the target and
source versions in JDKClassCompiler.
There is an issue with java 11 JDK compiler and Drill code generation even
after asm version update. JDK compiler requires correctly specified kind of
"nest" for inner classes. This can be done by overriding
SimpleJavaFileObject#getNestingKind, but it will break java 11- compatibility.
So source and target version should be specified for JDK compiler
Also, thanks for your note about HBase version. I had tried to update it to
1.1.11, but run into problems with Guava versions incompatibility.
P.S. May be "nestmates" can be added with ASM ClassVisitor?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
>
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668426#comment-16668426
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev edited a comment on issue #1446: DRILL-6349: Drill JDBC driver
fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434231490
@vvysotskyi
> Also, recently was released ASM 7.0, which supports JDK 11 features, so we
can update it and revert changes connected with specifying the target and
source versions in JDKClassCompiler.
There is an issue with java 11 JDK compiler and Drill code generation even
after asm version update. JDK compiler requires you yo correctly specify the
kind of "nest" for inner classes. This can be done by overriding
SimpleJavaFileObject#getNestingKind, but it will break java 11- compatibility.
So source and target version should be specified for JDK compiler
Also, thanks for your note about HBase version. I had tried to update it to
1.1.11, but run into problems with Guava versions incompatibility
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
>
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668420#comment-16668420
]
ASF GitHub Bot commented on DRILL-6349:
---
oleg-zinovev commented on issue #1446: DRILL-6349: Drill JDBC driver fails on
Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-434231490
@vvysotskyi
> Also, recently was released ASM 7.0, which supports JDK 11 features, so we
can update it and revert changes connected with specifying the target and
source versions in JDKClassCompiler.
There is an issue with java 11 JDK compiler and Drill code generation even
after asm version update. JDK compiler requires you yo correctly specify the
kind of "nest" for inner classes. This can be done by overriding
SimpleJavaFileObject#getNestingKind, but it will break java 11- compatibility.
So source and target version should be specified for JDK compiler
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at
[jira] [Created] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
Khurram Faraaz created DRILL-6816:
-
Summary: NPE - Concurrent query execution using PreparedStatement
Key: DRILL-6816
URL: https://issues.apache.org/jira/browse/DRILL-6816
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Concurrent query execution from JDBC program using PreparedStatement results in
NPE.
Queries that were executed concurrently are (part of a query file),
{noformat}
select id from `test_tbl.json`
select count(id) from `test_tbl.json`
select count(*) from `test_tbl.json`
select * from `test_tbl.json`
{noformat}
Drill 1.14.0
git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
MapR version => 6.1.0.20180911143226.GA (secure cluster)
Executing the above queries concurrently using a Statement object results in
successful query execution.
{noformat}
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
{noformat}
However, when the same queries listed above are executed using a
PreparedStatement object we see an NPE
{noformat}
PreparedStatement prdstmnt = conn.prepareStatement(query);
prdstmnt.executeUpdate();
{noformat}
Stack trace from drillbit.log
{noformat}
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.executeQuery(RunQuery.java:61)
at RunQuery.run(RunQuery.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
