[jira] [Created] (DRILL-6827) Apache Drill 1.14 on a kerberized Cloudera cluster (CDH 5.14).
Ibrahim Safieddine created DRILL-6827:
-
Summary: Apache Drill 1.14 on a kerberized Cloudera cluster (CDH
5.14).
Key: DRILL-6827
URL: https://issues.apache.org/jira/browse/DRILL-6827
Project: Apache Drill
Issue Type: Bug
Components: Security
Affects Versions: 1.14.0
Environment: * Apache Drill 1.14
* Cloudera CDH 5.14
Reporter: Ibrahim Safieddine
Hello,
I'am using apache Drill 1.14 on a kerberized Cloudera cluster (CDH 5.14).
When I activate kerberos authentification, drill server refuse to start with
error:
{color:#ff}_org.apache.drill.exec.exception.DrillbitStartupException:
Authentication is enabled for WebServer but none of the security mechanism was
configured properly. Please verify the configurations and try again._{color}
I can see in the logs that the kerberos authentification is ok:
[main] INFO o.a.d.exec.server.BootStrapContext - Process user name: 'root' and
logged in successfully as 'tata/xx.yy...@xx.yy'
Can you help me please?
Based on the Apache Drill documentation, there is my conf/drill-override.conf:
drill.exec: {
cluster-id: "drillbits1",
zk.connect: "xx.yy.zz:2181",
service_name: "service1",
impersonation: {
enabled: true,
max_chained_user_hops: 3
},
security: {
user.auth.enabled:true,
auth.mechanisms:["KERBEROS"],
auth.principal:"tata/xx.yy...@xx.yy",
auth.keytab:"keytab1.keytab",
drill.exec.security.auth.auth_to_local:hive,
auth.realm: "XX.YY",
user.encryption.sasl.enabled: true,
user.encryption.sasl.max_wrapped_size: 65536
},
security.user.encryption.ssl: {
enabled: true,
keyPassword: "X",
handshakeTimeout: 1,
provider: "JDK"
},
ssl: {
keyStorePath: "X",
keyStorePassword: "X",
trustStorePath: "X",
trustStorePassword: "X"
},
http: {
enabled: true,
auth.enabled: false,
auth.mechanisms: ["KERBEROS"],
ssl_enabled: true,
port: 8047
session_max_idle_secs: 3600, # Default value 1hr
cors: {
enabled: false,
allowedOrigins: ["null"],
allowedMethods: ["GET", "POST", "HEAD", "OPTIONS"],
allowedHeaders: ["X-Requested-With", "Content-Type", "Accept", "Origin"],
credentials: true
}
}
}
Thank you
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6720) Unable to read more than 1000 records from DB2 table
[
https://issues.apache.org/jira/browse/DRILL-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672830#comment-16672830
]
kshitij commented on DRILL-6720:
[~kkhatua] As mentioned in my earlier comment, I have tried using the same JDBC
driver in a simple java program that connects to the database and reads through
all the records in the table. It works without any issue.
PFB the Java code snippet:
Connection con = DriverManager.getConnection(url, user, password);
Statement stmt = con.createStatement();
ResultSet rs = stmt.executeQuery("SELECT * FROM SCHEMA.TABLE");
int count = 0;
while(rs.next()){
System.out.println("Record number "+ ++count +":"+rs.getString(2));
}
rs.close();
con.close();
> Unable to read more than 1000 records from DB2 table
>
>
> Key: DRILL-6720
> URL: https://issues.apache.org/jira/browse/DRILL-6720
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Storage - JDBC
>Affects Versions: 1.9.0
>Reporter: kshitij
>Priority: Critical
> Fix For: 1.9.0
>
> Attachments: drill_DB2_query_error_log.txt, drill_db2_query_status.PNG
>
>
> We have created a storage plugin in drill for DB2 database, PFB the details:
> {
> "type": "jdbc",
> "driver": "com.ibm.db2.jcc.DB2Driver",
> "url": "jdbc:db2://server:port/databasename",
> "username": "user",
> "password": "password",
> "enabled": true
> }
> Version of DB2 is 10.1. We are using a type 4 JDBC driver (db2jcc4.jar).
> When we try to read the data in any of the tables, the query fails with
> following error from drill:
> org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR:
> Failure while attempting to read from database. sql SELECT * FROM
> schema.table plugin DB2_PLUGIN Fragment 0:0 [Error Id:
> 1404544f-bb5e-439b-b1a8-679388bb344d on server:port]
> The error logs from drill have been attached.
> One interesting observation - when we put a LIMIT clause of <=1000 to the
> query, the query works and returns the data. Anything more than 1000 in LIMIT
> clause throws back the same error as above.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673030#comment-16673030
]
ASF GitHub Bot commented on DRILL-6349:
---
vvysotskyi commented on issue #1446: DRILL-6349: Drill JDBC driver fails on
Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#issuecomment-435362811
@oleg-zinovev, thanks for pointing to
[SUREFIRE-1588](https://issues.apache.org/jira/browse/SUREFIRE-1588) issue.
Let's set `false` for
`maven-surefire-plugin` to avoid possible problems until it is resolved in the
newer plugin version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.15.0
>
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
> java.base/jdk.interna
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673187#comment-16673187
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r230401090
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/coord/zk/TestZKACL.java
##
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import com.typesafe.config.ConfigValueFactory;
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.api.ACLProvider;
+import org.apache.curator.retry.RetryNTimes;
+import org.apache.curator.test.TestingServer;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.scanner.ClassPathScanner;
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.server.BootStrapContext;
+import org.apache.drill.exec.server.options.SystemOptionManager;
+import org.apache.zookeeper.data.ACL;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import java.util.List;
+
+public class TestZKACL {
+
+ private TestingServer server;
+ private final static String cluster_config_znode =
"test-cluster_config_znode";
+ private final static byte[] cluster_config_data = "drill-node-1".getBytes();
+ private final static String drill_zk_root = "drill-test-drill_zk_root";
+ private final static String drill_cluster_name = "test-drillbits";
+ private static final String drillClusterPath = "/" + drill_zk_root + "/" +
drill_cluster_name;
+ private static final RetryPolicy retryPolicy = new RetryNTimes(1, 1000);
+
+ private static final String drillUDFName = "test-udfs";
+ private final static byte[] udf_data = "test-udf-1".getBytes();
+ private static final String drillUDFPath = "/" + drill_zk_root + "/" +
drillUDFName;
+ private static ACLProvider aclProviderDelegate;
+
+ private static CuratorFramework client;
+
+ @Before
+ public void setUp() throws Exception {
+System.setProperty("zookeeper.authProvider.1",
"org.apache.zookeeper.server.auth.SASLAuthenticationProvider");
+String configPath =
(ClassLoader.getSystemResource("zkacltest.conf")).getPath();
+System.setProperty("java.security.auth.login.config", configPath);
+server = new TestingServer();
+
+
+final DrillConfig config = new
DrillConfig(DrillConfig.create().withValue(ExecConstants.ZK_ACL_PROVIDER,
+ConfigValueFactory.fromAnyRef("creator-all")
+).withValue(ExecConstants.ZK_APPLY_SECURE_ACL,
ConfigValueFactory.fromAnyRef(true)));
+
+final ScanResult result = ClassPathScanner.fromPrescan(config);
+final BootStrapContext bootStrapContext =
+new BootStrapContext(config,
SystemOptionManager.createDefaultOptionDefinitions(), result);
+aclProviderDelegate = ZKACLProviderFactory.getACLProvider(config,
drillClusterPath, bootStrapContext);
+
+server.start();
+
+client = CuratorFrameworkFactory.builder().
+retryPolicy(retryPolicy).
+connectString(server.getConnectString()).
+aclProvider(aclProviderDelegate).
+build();
+client.start();
+ }
+
+ /**
+ * Test ACLs on znodes required to discover the cluster
+ */
+ // ZK libraries only supports one client instance per-machine per-server and
it is cached.
+ // This test will fail when run after other ZK tests that setup the client
in a way that will cause this test to fail.
+ @Ignore
+ @Test
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673185#comment-16673185
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r230401022
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKSecureACLProvider.java
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.data.ACL;
+
+import java.util.List;
+
+/**
+ * ZKSecureACLProvider restricts access to znodes created by Drill in a secure
installation.
+ *
+ * The cluster discovery znode i.e. the znode containing the list of Drillbits
is
+ * readable by anyone.
+ *
+ * For all other znodes, only the creator of the znode, i.e the Drillbit user,
has full access.
+ *
+ */
+@ZKACLProviderTemplate(type = "creator-all")
+public class ZKSecureACLProvider implements ZKACLProvider {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKSecureACLProvider.class);
+
+/**
+ * DEFAULT_ACL gives the creator of a znode full-access to it
+ */
+static ImmutableList DEFAULT_ACL = new ImmutableList.Builder()
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+ .build();
+/**
+ * DRILL_CLUSTER_ACL gives the creator full access and everyone else only
read access.
+ * Used on the Drillbit discovery znode (znode path
//)
+ * i.e. the node that contains the list of Drillbits in the cluster.
+ */
+ static ImmutableList DRILL_CLUSTER_ACL = new
ImmutableList.Builder()
+
.addAll(Ids.READ_ACL_UNSAFE.iterator())
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+.build();
+final String drillClusterPath;
+
+public ZKSecureACLProvider(ZKACLContextProvider state) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673189#comment-16673189
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r230401310
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderFactory.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.drill.common.config.DrillConfig;
+import static org.apache.drill.exec.ExecConstants.ZK_ACL_PROVIDER;
+import static org.apache.drill.exec.ExecConstants.ZK_APPLY_SECURE_ACL;
+
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.apache.drill.exec.exception.DrillbitStartupException;
+import org.apache.drill.exec.server.BootStrapContext;
+import org.apache.drill.shaded.guava.com.google.common.base.Strings;
+import java.lang.reflect.InvocationTargetException;
+
+import java.lang.reflect.Constructor;
+import java.util.Collection;
+
+/**
+ * This function returns a {@link ZKACLProviderDelegate} which will be used to
set ACLs on Drill ZK nodes
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673192#comment-16673192
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r230401716
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/coord/zk/TestZKACL.java
##
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import com.typesafe.config.ConfigValueFactory;
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.api.ACLProvider;
+import org.apache.curator.retry.RetryNTimes;
+import org.apache.curator.test.TestingServer;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.scanner.ClassPathScanner;
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.server.BootStrapContext;
+import org.apache.drill.exec.server.options.SystemOptionManager;
+import org.apache.zookeeper.data.ACL;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import java.util.List;
+
+public class TestZKACL {
+
+ private TestingServer server;
+ private final static String cluster_config_znode =
"test-cluster_config_znode";
+ private final static byte[] cluster_config_data = "drill-node-1".getBytes();
+ private final static String drill_zk_root = "drill-test-drill_zk_root";
+ private final static String drill_cluster_name = "test-drillbits";
+ private static final String drillClusterPath = "/" + drill_zk_root + "/" +
drill_cluster_name;
+ private static final RetryPolicy retryPolicy = new RetryNTimes(1, 1000);
+
+ private static final String drillUDFName = "test-udfs";
+ private final static byte[] udf_data = "test-udf-1".getBytes();
+ private static final String drillUDFPath = "/" + drill_zk_root + "/" +
drillUDFName;
+ private static ACLProvider aclProviderDelegate;
+
+ private static CuratorFramework client;
+
+ @Before
+ public void setUp() throws Exception {
+System.setProperty("zookeeper.authProvider.1",
"org.apache.zookeeper.server.auth.SASLAuthenticationProvider");
+String configPath =
(ClassLoader.getSystemResource("zkacltest.conf")).getPath();
+System.setProperty("java.security.auth.login.config", configPath);
+server = new TestingServer();
+
+
+final DrillConfig config = new
DrillConfig(DrillConfig.create().withValue(ExecConstants.ZK_ACL_PROVIDER,
+ConfigValueFactory.fromAnyRef("creator-all")
+).withValue(ExecConstants.ZK_APPLY_SECURE_ACL,
ConfigValueFactory.fromAnyRef(true)));
+
+final ScanResult result = ClassPathScanner.fromPrescan(config);
+final BootStrapContext bootStrapContext =
+new BootStrapContext(config,
SystemOptionManager.createDefaultOptionDefinitions(), result);
+aclProviderDelegate = ZKACLProviderFactory.getACLProvider(config,
drillClusterPath, bootStrapContext);
+
+server.start();
+
+client = CuratorFrameworkFactory.builder().
+retryPolicy(retryPolicy).
+connectString(server.getConnectString()).
+aclProvider(aclProviderDelegate).
+build();
+client.start();
+ }
+
+ /**
+ * Test ACLs on znodes required to discover the cluster
+ */
+ // ZK libraries only supports one client instance per-machine per-server and
it is cached.
+ // This test will fail when run after other ZK tests that setup the client
in a way that will cause this test to fail.
+ @Ignore
+ @Test
+ public void testClusterDiscoveryPaths() {
+try {
+ String path = PathUtils.join(drillClusterPath, cluster_config_znode);
+ client.create().creatingParentsIfNeeded().forPath(path,
cluster_config_data);
+ List remoteACLs =
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673190#comment-16673190
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r230401665
##
File path: exec/java-exec/src/main/resources/drill-module.conf
##
@@ -123,7 +125,9 @@ drill.exec: {
retry: {
count: 7200,
delay: 500
-}
+},
+apply_secure_acl: false,
+acl_provider: "creator-all"
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673191#comment-16673191
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r230401680
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKSecureACLProvider.java
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.data.ACL;
+
+import java.util.List;
+
+/**
+ * ZKSecureACLProvider restricts access to znodes created by Drill in a secure
installation.
+ *
+ * The cluster discovery znode i.e. the znode containing the list of Drillbits
is
+ * readable by anyone.
+ *
+ * For all other znodes, only the creator of the znode, i.e the Drillbit user,
has full access.
+ *
+ */
+@ZKACLProviderTemplate(type = "creator-all")
+public class ZKSecureACLProvider implements ZKACLProvider {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKSecureACLProvider.class);
+
+/**
+ * DEFAULT_ACL gives the creator of a znode full-access to it
+ */
+static ImmutableList DEFAULT_ACL = new ImmutableList.Builder()
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+ .build();
+/**
+ * DRILL_CLUSTER_ACL gives the creator full access and everyone else only
read access.
+ * Used on the Drillbit discovery znode (znode path
//)
+ * i.e. the node that contains the list of Drillbits in the cluster.
+ */
+ static ImmutableList DRILL_CLUSTER_ACL = new
ImmutableList.Builder()
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673202#comment-16673202
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r230403320
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKSecureACLProvider.java
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.data.ACL;
+
+import java.util.List;
+
+/**
+ * ZKSecureACLProvider restricts access to znodes created by Drill in a secure
installation.
+ *
+ * The cluster discovery znode i.e. the znode containing the list of Drillbits
is
+ * readable by anyone.
+ *
+ * For all other znodes, only the creator of the znode, i.e the Drillbit user,
has full access.
+ *
+ */
+@ZKACLProviderTemplate(type = "creator-all")
+public class ZKSecureACLProvider implements ZKACLProvider {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKSecureACLProvider.class);
+
+/**
+ * DEFAULT_ACL gives the creator of a znode full-access to it
+ */
+static ImmutableList DEFAULT_ACL = new ImmutableList.Builder()
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+ .build();
+/**
+ * DRILL_CLUSTER_ACL gives the creator full access and everyone else only
read access.
+ * Used on the Drillbit discovery znode (znode path
//)
+ * i.e. the node that contains the list of Drillbits in the cluster.
+ */
+ static ImmutableList DRILL_CLUSTER_ACL = new
ImmutableList.Builder()
+
.addAll(Ids.READ_ACL_UNSAFE.iterator())
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+.build();
+final String drillClusterPath;
+
+public ZKSecureACLProvider(ZKACLContextProvider state) {
+this.drillClusterPath = state.getClusterPath();
+}
+
+@Override
+public List getDrillDefaultAcl() {
+return DEFAULT_ACL;
+}
+
+@Override
+public List getDrillAclForPath(String path) {
+logger.trace("getAclForPath " + path);
+if(path.startsWith(drillClusterPath)) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additi
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673205#comment-16673205
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r230403715
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderDelegate.java
##
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.curator.framework.api.ACLProvider;
+import
org.apache.drill.shaded.guava.com.google.common.annotations.VisibleForTesting;
+import org.apache.zookeeper.data.ACL;
+
+import java.util.List;
+
+/**
+ * This class hides the {@link ZKACLProvider} from Curator-specific functions
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673204#comment-16673204
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r230403657
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderFactory.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.drill.common.config.DrillConfig;
+import static org.apache.drill.exec.ExecConstants.ZK_ACL_PROVIDER;
+import static org.apache.drill.exec.ExecConstants.ZK_APPLY_SECURE_ACL;
+
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.apache.drill.exec.exception.DrillbitStartupException;
+import org.apache.drill.exec.server.BootStrapContext;
+import org.apache.drill.shaded.guava.com.google.common.base.Strings;
+import java.lang.reflect.InvocationTargetException;
+
+import java.lang.reflect.Constructor;
+import java.util.Collection;
+
+/**
+ * This function returns a {@link ZKACLProviderDelegate} which will be used to
set ACLs on Drill ZK nodes
+ * If secure ACLs are required, the {@link ZKACLProviderFactory} looks up and
instantiates a {@link ZKACLProviderDelegate}
+ * specified in the config file. Else it returns the {@link
ZKDefaultACLProvider}
+ */
+public class ZKACLProviderFactory {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKACLProviderFactory.class);
+
+public static ZKACLProviderDelegate getACLProvider(DrillConfig config,
String drillClusterPath, BootStrapContext context)
+throws DrillbitStartupException {
+ZKACLContextProvider stateProvider = new
ZKACLContextProviderImpl(drillClusterPath);
+
+if (config.getBoolean(ZK_APPLY_SECURE_ACL)) {
+logger.trace("Using secure ZK ACL. Drill cluster path " +
drillClusterPath);
+ZKACLProviderDelegate aclProvider = findACLProvider(config,
stateProvider, context);
+return aclProvider;
+}
+logger.trace("Using un-secure default ZK ACL");
+final ZKDefaultACLProvider defaultAclProvider = new
ZKDefaultACLProvider(stateProvider);
+return new ZKACLProviderDelegate(defaultAclProvider);
+}
+
+public static ZKACLProviderDelegate findACLProvider(DrillConfig config,
ZKACLContextProvider contextProvider,
+BootStrapContext
context) throws DrillbitStartupException {
+if (!config.hasPath(ZK_ACL_PROVIDER)) {
+throw new DrillbitStartupException(String.format("BOOT option '%s'
is missing in config.", ZK_ACL_PROVIDER));
+}
+
+final String aclProviderName = config.getString(ZK_ACL_PROVIDER);
+
+if (Strings.isNullOrEmpty(aclProviderName)) {
+throw new DrillbitStartupException(String.format("Invalid value
'%s' for BOOT option '%s'", aclProviderName,
+ZK_ACL_PROVIDER));
+}
+
+ScanResult scan = context.getClasspathScan();
+final Collection> aclProviderImpls =
+scan.getImplementations(ZKACLProvider.class);
+logger.debug("Found ZkACLProvider implementations: {}",
aclProviderImpls);
+
+for (Class clazz : aclProviderImpls) {
+ final ZKACLProviderTemplate template =
clazz.getAnnotation(ZKACLProviderTemplate.class);
+ if (template == null) {
+logger.warn("{} doesn't have {} annotation. Skipping.",
clazz.getCanonicalName(),
+ZKACLProviderTemplate.class);
+continue;
+ }
+
+ if (template.type().equalsIgnoreCase(aclProviderName)) {
+Constructor validConstructor = null;
+Class constructorArgumentClass = ZKACLContextProvider.class;
+for (Constructor c : clazz.getConstructors()) {
+ Class[] params = c.getParameterTypes();
+ if (par
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673206#comment-16673206
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r230403743
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderFactory.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.drill.common.config.DrillConfig;
+import static org.apache.drill.exec.ExecConstants.ZK_ACL_PROVIDER;
+import static org.apache.drill.exec.ExecConstants.ZK_APPLY_SECURE_ACL;
+
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.apache.drill.exec.exception.DrillbitStartupException;
+import org.apache.drill.exec.server.BootStrapContext;
+import org.apache.drill.shaded.guava.com.google.common.base.Strings;
+import java.lang.reflect.InvocationTargetException;
+
+import java.lang.reflect.Constructor;
+import java.util.Collection;
+
+/**
+ * This function returns a {@link ZKACLProviderDelegate} which will be used to
set ACLs on Drill ZK nodes
+ * If secure ACLs are required, the {@link ZKACLProviderFactory} looks up and
instantiates a {@link ZKACLProviderDelegate}
+ * specified in the config file. Else it returns the {@link
ZKDefaultACLProvider}
+ */
+public class ZKACLProviderFactory {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKACLProviderFactory.class);
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673219#comment-16673219
]
ASF GitHub Bot commented on DRILL-5671:
---
bitblender commented on a change in pull request #1467: DRILL-5671: Set secure
ACLs (Access Control List) for Drill ZK nodes in a secure cluster
URL: https://github.com/apache/drill/pull/1467#discussion_r230405672
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderTemplate.java
##
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.coord.zk;
+
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * Annotation for {@link ZKACLProviderDelegate} implementation to identify the
+ * implementation type. Implementation type is set in BOOT option
drill.exec.zk.acl_provider.
+ */
+@Retention(RetentionPolicy.RUNTIME)
+@Target({ElementType.TYPE})
+public @interface ZKACLProviderTemplate {
+ /**
+ * {@link ZKACLProviderDelegate} implementation type.
+ * @return
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[ https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673221#comment-16673221 ] ASF GitHub Bot commented on DRILL-5671: --- bitblender commented on issue #1467: DRILL-5671: Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster URL: https://github.com/apache/drill/pull/1467#issuecomment-435409843 @arina-ielchiieva @sohami Update the PR. Please take a look This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster > > > Key: DRILL-5671 > URL: https://issues.apache.org/jira/browse/DRILL-5671 > Project: Apache Drill > Issue Type: New Feature > Components: Server >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e. > anyone gets to do CDRWA(create, delete, read, write, admin access). This > means that even on a secure cluster anyone can perform all CRDWA actions on > the znodes. > This should be changed such that: > - In a non-secure cluster, Drill will continue using the current default > [world:all] ACL > - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the > authenticated user that created the znode gets full access. The discovery > znodes i.e. the znodes with the list of Drillbits will have an additional > [world:read] ACL, i.e. the list of Drillbits will be readable by anyone. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673478#comment-16673478
]
Oleg Zinoviev commented on DRILL-6722:
--
At this stage, it seems to me that the whole logic of the MergeAdapter is
broken. Neither the variable index in ValueHolderSub, nor mapping local
variable to frame slot in MethodAnalyzer does not track source code control
flow.
Bug reason:
Then
ALOAD[P]
ASTORE[N]
executes, ValueHolderSub linked with variable N changes it's `first` field to P
holder `first` field. Since ValueHolderSub shared between all variables, linked
to him, it leads to all this varaibles become assigned to P.
If I make first field final, CASE fails:
CASE compiles into
var res;
{code:java}
if (expression) {
res = v1;
else {
res = v2;
}
{code}
Then Analyzer calculates Frame variables, it ignores if - else logic and
assigne v2 to res slot.
P.S.
https://stackoverflow.com/questions/25109942/is-there-a-better-explanation-of-stack-map-frames
Can be Analyzed is intended only for variable type inferrence?
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673478#comment-16673478
]
Oleg Zinoviev edited comment on DRILL-6722 at 11/2/18 5:47 PM:
---
At this stage, it seems to me that the whole logic of the MergeAdapter is
broken. Neither the variable index in ValueHolderSub, nor mapping local
variable to frame slot in MethodAnalyzer does not track source code control
flow.
Bug reason:
Then
ALOAD[P]
ASTORE[N]
executes, ValueHolderSub linked with variable N changes it's `first` field to P
holder `first` field. Since ValueHolderSub shared between all variables, linked
to him, it leads to all this varaibles become assigned to P.
If I make first field final, CASE fails:
CASE compiles into
{code:java}
var res;
if (expression) {
res = v1;
else {
res = v2;
}
{code}
Then Analyzer calculates Frame variables, it ignores if - else logic and
assigne v2 to res slot.
P.S.
https://stackoverflow.com/questions/25109942/is-there-a-better-explanation-of-stack-map-frames
Can be Analyzed is intended only for variable type inferrence?
was (Author: le.louch):
At this stage, it seems to me that the whole logic of the MergeAdapter is
broken. Neither the variable index in ValueHolderSub, nor mapping local
variable to frame slot in MethodAnalyzer does not track source code control
flow.
Bug reason:
Then
ALOAD[P]
ASTORE[N]
executes, ValueHolderSub linked with variable N changes it's `first` field to P
holder `first` field. Since ValueHolderSub shared between all variables, linked
to him, it leads to all this varaibles become assigned to P.
If I make first field final, CASE fails:
CASE compiles into
var res;
{code:java}
if (expression) {
res = v1;
else {
res = v2;
}
{code}
Then Analyzer calculates Frame variables, it ignores if - else logic and
assigne v2 to res slot.
P.S.
https://stackoverflow.com/questions/25109942/is-there-a-better-explanation-of-stack-map-frames
Can be Analyzed is intended only for variable type inferrence?
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673478#comment-16673478
]
Oleg Zinoviev edited comment on DRILL-6722 at 11/2/18 5:48 PM:
---
At this stage, it seems to me that the whole logic of the MergeAdapter is
broken. Neither the variable index in ValueHolderSub, nor mapping local
variable to frame slot in MethodAnalyzer does not track source code control
flow.
Bug reason:
Then
ALOAD[P]
ASTORE[N]
executes, ValueHolderSub linked with variable N changes it's `first` field to P
holder `first` field. Since ValueHolderSub shared between all variables, linked
to him, it leads to all this varaibles become assigned to P.
If I make first field final, CASE fails:
CASE compiles into
{code:java}
var res;
if (expression) {
res = v1;
else {
res = v2;
}
{code}
Then Analyzer calculates Frame variables, it ignores if - else logic and
assigne v2 to res slot.
P.S.
https://stackoverflow.com/questions/25109942/is-there-a-better-explanation-of-stack-map-frames
Can be Analyzed is intended only for variable type inferrence?
was (Author: le.louch):
At this stage, it seems to me that the whole logic of the MergeAdapter is
broken. Neither the variable index in ValueHolderSub, nor mapping local
variable to frame slot in MethodAnalyzer does not track source code control
flow.
Bug reason:
Then
ALOAD[P]
ASTORE[N]
executes, ValueHolderSub linked with variable N changes it's `first` field to P
holder `first` field. Since ValueHolderSub shared between all variables, linked
to him, it leads to all this varaibles become assigned to P.
If I make first field final, CASE fails:
CASE compiles into
{code:java}
var res;
if (expression) {
res = v1;
else {
res = v2;
}
{code}
Then Analyzer calculates Frame variables, it ignores if - else logic and
assigne v2 to res slot.
P.S.
https://stackoverflow.com/questions/25109942/is-there-a-better-explanation-of-stack-map-frames
Can be Analyzed is intended only for variable type inferrence?
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673478#comment-16673478
]
Oleg Zinoviev edited comment on DRILL-6722 at 11/2/18 5:51 PM:
---
At this stage, it seems to me that the whole logic of the ScalarReplacementNode
is broken. Neither the variable index in ValueHolderSub, nor mapping local
variable to frame slot in MethodAnalyzer does not track source code control
flow.
Bug reason:
Then
ALOAD[P]
ASTORE[N]
executes, ValueHolderSub linked with variable N changes it's `first` field to P
holder `first` field. Since ValueHolderSub shared between all variables, linked
to him, it leads to all this varaibles become assigned to P.
If I make first field final, CASE fails:
CASE compiles into
{code:java}
var res;
if (expression) {
res = v1;
else {
res = v2;
}
{code}
Then Analyzer calculates Frame variables, it ignores if - else logic and
assigne v2 to res slot.
P.S.
https://stackoverflow.com/questions/25109942/is-there-a-better-explanation-of-stack-map-frames
Can be Analyzed is intended only for variable type inferrence?
was (Author: le.louch):
At this stage, it seems to me that the whole logic of the MergeAdapter is
broken. Neither the variable index in ValueHolderSub, nor mapping local
variable to frame slot in MethodAnalyzer does not track source code control
flow.
Bug reason:
Then
ALOAD[P]
ASTORE[N]
executes, ValueHolderSub linked with variable N changes it's `first` field to P
holder `first` field. Since ValueHolderSub shared between all variables, linked
to him, it leads to all this varaibles become assigned to P.
If I make first field final, CASE fails:
CASE compiles into
{code:java}
var res;
if (expression) {
res = v1;
else {
res = v2;
}
{code}
Then Analyzer calculates Frame variables, it ignores if - else logic and
assigne v2 to res slot.
P.S.
https://stackoverflow.com/questions/25109942/is-there-a-better-explanation-of-stack-map-frames
Can be Analyzed is intended only for variable type inferrence?
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673478#comment-16673478
]
Oleg Zinoviev edited comment on DRILL-6722 at 11/2/18 5:53 PM:
---
At this stage, it seems to me that the whole logic of the ScalarReplacementNode
is broken. Neither the variable index in ValueHolderSub, nor mapping local
variable to frame slot in MethodAnalyzer does not track source code control
flow.
Bug reason:
Then
ALOAD[P]
ASTORE[N]
executes, ValueHolderSub linked with variable P changes it's `first` field to N
holder `first` field. Since ValueHolderSub shared between all variables, linked
to him, it leads to all this varaibles become assigned to N.
If I make first field final, CASE fails:
CASE compiles into
{code:java}
var res;
if (expression) {
res = v1;
else {
res = v2;
}
{code}
Then Analyzer calculates Frame variables, it ignores if - else logic and
assigne v2 to res slot.
P.S.
https://stackoverflow.com/questions/25109942/is-there-a-better-explanation-of-stack-map-frames
Can be Analyzed is intended only for variable type inferrence?
was (Author: le.louch):
At this stage, it seems to me that the whole logic of the ScalarReplacementNode
is broken. Neither the variable index in ValueHolderSub, nor mapping local
variable to frame slot in MethodAnalyzer does not track source code control
flow.
Bug reason:
Then
ALOAD[P]
ASTORE[N]
executes, ValueHolderSub linked with variable P changes it's `first` field to N
holder `first` field. Since ValueHolderSub shared between all variables, linked
to him, it leads to all this varaibles become assigned to P.
If I make first field final, CASE fails:
CASE compiles into
{code:java}
var res;
if (expression) {
res = v1;
else {
res = v2;
}
{code}
Then Analyzer calculates Frame variables, it ignores if - else logic and
assigne v2 to res slot.
P.S.
https://stackoverflow.com/questions/25109942/is-there-a-better-explanation-of-stack-map-frames
Can be Analyzed is intended only for variable type inferrence?
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673478#comment-16673478
]
Oleg Zinoviev edited comment on DRILL-6722 at 11/2/18 5:53 PM:
---
At this stage, it seems to me that the whole logic of the ScalarReplacementNode
is broken. Neither the variable index in ValueHolderSub, nor mapping local
variable to frame slot in MethodAnalyzer does not track source code control
flow.
Bug reason:
Then
ALOAD[P]
ASTORE[N]
executes, ValueHolderSub linked with variable P changes it's `first` field to N
holder `first` field. Since ValueHolderSub shared between all variables, linked
to him, it leads to all this varaibles become assigned to P.
If I make first field final, CASE fails:
CASE compiles into
{code:java}
var res;
if (expression) {
res = v1;
else {
res = v2;
}
{code}
Then Analyzer calculates Frame variables, it ignores if - else logic and
assigne v2 to res slot.
P.S.
https://stackoverflow.com/questions/25109942/is-there-a-better-explanation-of-stack-map-frames
Can be Analyzed is intended only for variable type inferrence?
was (Author: le.louch):
At this stage, it seems to me that the whole logic of the ScalarReplacementNode
is broken. Neither the variable index in ValueHolderSub, nor mapping local
variable to frame slot in MethodAnalyzer does not track source code control
flow.
Bug reason:
Then
ALOAD[P]
ASTORE[N]
executes, ValueHolderSub linked with variable N changes it's `first` field to P
holder `first` field. Since ValueHolderSub shared between all variables, linked
to him, it leads to all this varaibles become assigned to P.
If I make first field final, CASE fails:
CASE compiles into
{code:java}
var res;
if (expression) {
res = v1;
else {
res = v2;
}
{code}
Then Analyzer calculates Frame variables, it ignores if - else logic and
assigne v2 to res slot.
P.S.
https://stackoverflow.com/questions/25109942/is-there-a-better-explanation-of-stack-map-frames
Can be Analyzed is intended only for variable type inferrence?
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6828) Hit UnrecognizedPropertyException when run tpch queries
Dechang Gu created DRILL-6828:
-
Summary: Hit UnrecognizedPropertyException when run tpch queries
Key: DRILL-6828
URL: https://issues.apache.org/jira/browse/DRILL-6828
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.15.0
Environment: RHEL 7, Apache Drill commit id:
18e09a1b1c801f2691a05ae7db543bf71874cfea
Reporter: Dechang Gu
Assignee: Boaz Ben-Zvi
Fix For: 1.15.0
Installed Apache Drill 1.15.0 commit id:
18e09a1b1c801f2691a05ae7db543bf71874cfea DRILL-6763: Codegen optimization of
SQL functions with constant values(\#1481)
Hit the following errors:
{code}
java.sql.SQLException: SYSTEM ERROR: UnrecognizedPropertyException:
Unrecognized field "outgoingBatchSize" (class
org.apache.drill.exec.physical.config.HashPartitionSender), not marked as
ignorable (9 known properties: "receiver-major-fragment", "initialAllocation",
"expr", "userName", "@id", "child", "cost", "destinations", "maxAllocation"])
at [Source: (StringReader); line: 1000, column: 29] (through reference chain:
org.apache.drill.exec.physical.config.HashPartitionSender["outgoingBatchSize"])
Fragment 3:175
Please, refer to logs for more information.
[Error Id: cc023cdb-9a46-4edd-ad0b-6da1e9085291 on ucs-node6.perf.lab:31010]
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528)
at
org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:600)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1288)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
at
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
at
org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1109)
at
org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1120)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:196)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at PipSQueak.executeQuery(PipSQueak.java:289)
at PipSQueak.runTest(PipSQueak.java:104)
at PipSQueak.main(PipSQueak.java:477)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: UnrecognizedPropertyException: Unrecognized field "outgoingBatchSize"
(class org.apache.drill.exec.physical.config.HashPartitionSender), not marked
as ignorable (9 known properties: "receiver-major-fragment",
"initialAllocation", "expr", "userName", "@id", "child", "cost",
"destinations", "maxAllocation"])
at [Source: (StringReader); line: 1000, column: 29] (through reference chain:
org.apache.drill.exec.physical.config.HashPartitionSender["outgoingBatchSize"])
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (DRILL-6722) Query from parquet with case-then and arithmetic operation returns a wrong result
[
https://issues.apache.org/jira/browse/DRILL-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zinoviev updated DRILL-6722:
-
Comment: was deleted
(was: At this stage, it seems to me that the whole logic of the
ScalarReplacementNode is broken. Neither the variable index in ValueHolderSub,
nor mapping local variable to frame slot in MethodAnalyzer does not track
source code control flow.
Bug reason:
Then
ALOAD[P]
ASTORE[N]
executes, ValueHolderSub linked with variable P changes it's `first` field to N
holder `first` field. Since ValueHolderSub shared between all variables, linked
to him, it leads to all this varaibles become assigned to N.
If I make first field final, CASE fails:
CASE compiles into
{code:java}
var res;
if (expression) {
res = v1;
else {
res = v2;
}
{code}
Then Analyzer calculates Frame variables, it ignores if - else logic and
assigne v2 to res slot.
P.S.
https://stackoverflow.com/questions/25109942/is-there-a-better-explanation-of-stack-map-frames
Can be Analyzed is intended only for variable type inferrence?
)
> Query from parquet with case-then and arithmetic operation returns a wrong
> result
> -
>
> Key: DRILL-6722
> URL: https://issues.apache.org/jira/browse/DRILL-6722
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: JaininoJava.class, JaininoJava2_merged.class,
> correct.csv, result.csv
>
>
> Steps to reproduce:
> 1) Create sample table:
> {code:sql}
> create table dfs.tmp.test as
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> {code}
> 2) Execute query:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from dfs.tmp.test s
> {code}
> 3) Drill returns: [^result.csv]
> 4) Result of query without parquet:
> {code:sql}
> select
> case when s.a > s.b then s.a else s.b end as b,
> abs(s.a - s.b) as d
> from (
> select 1 as a, 2 as b
> union all
> select 3 as a, 2 as b
> union all
> select 1 as a, 4 as b
> union all
> select 2 as a, 2 as b
> ) s
> {code}
> [^correct.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6809) Handle repeated map in schema inference
[
https://issues.apache.org/jira/browse/DRILL-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673693#comment-16673693
]
ASF GitHub Bot commented on DRILL-6809:
---
sohami closed pull request #1513: DRILL-6809: Handle repeated map in schema
inference
URL: https://github.com/apache/drill/pull/1513
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/model/single/SingleSchemaInference.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/model/single/SingleSchemaInference.java
index 2a0e3cc6ab1..ad0c8c0454b 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/model/single/SingleSchemaInference.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/model/single/SingleSchemaInference.java
@@ -87,7 +87,7 @@ private AbstractColumnMetadata inferVector(ValueVector
vector) {
private TupleSchema inferMapSchema(AbstractMapVector vector) {
final List columns = new ArrayList<>();
-for (int i = 0; i < vector.getField().getChildren().size(); i++) {
+for (int i = 0; i < vector.size(); i++) {
columns.add(inferVector(vector.getChildByOrdinal(i)));
}
return MetadataUtils.fromColumns(columns);
diff --git
a/exec/java-exec/src/test/java/org/apache/drill/exec/physical/rowSet/impl/TestResultSetLoaderMapArray.java
b/exec/java-exec/src/test/java/org/apache/drill/exec/physical/rowSet/impl/TestResultSetLoaderMapArray.java
index 3d816c32900..a1d250242f4 100644
---
a/exec/java-exec/src/test/java/org/apache/drill/exec/physical/rowSet/impl/TestResultSetLoaderMapArray.java
+++
b/exec/java-exec/src/test/java/org/apache/drill/exec/physical/rowSet/impl/TestResultSetLoaderMapArray.java
@@ -45,8 +45,8 @@
import org.apache.drill.test.SubOperatorTest;
import org.apache.drill.test.rowSet.RowSet;
import org.apache.drill.test.rowSet.RowSet.SingleRowSet;
-import org.apache.drill.test.rowSet.RowSetComparison;
import org.apache.drill.test.rowSet.RowSetReader;
+import org.apache.drill.test.rowSet.RowSetUtilities;
import org.apache.drill.test.rowSet.schema.SchemaBuilder;
import org.junit.Test;
@@ -123,7 +123,7 @@ public void testBasics() {
mapValue(320, "d3.2"),
mapValue(330, "d3.3")))
.build();
-new RowSetComparison(expected).verifyAndClearAll(actual);
+RowSetUtilities.verify(expected, actual);
// In the second, create a row, then add a map member.
// Should be back-filled to empty for the first row.
@@ -172,7 +172,7 @@ public void testBasics() {
mapValue(620, "d6.2", null),
mapValue(630, "d6.3", "e6.3")))
.build();
-new RowSetComparison(expected).verifyAndClearAll(actual);
+RowSetUtilities.verify(expected, actual);
rsLoader.close();
}
@@ -220,13 +220,13 @@ public void testNestedArray() {
mapValue(320, strArray()),
mapValue(330, strArray("d3.3.1", "d1.2.2"
.build();
-new RowSetComparison(expected).verifyAndClearAll(actual);
+RowSetUtilities.verify(expected, actual);
rsLoader.close();
}
/**
- * Test a doubly-nested arrays of maps.
+ * Test a doubly-nested array of maps.
*/
@Test
diff --git
a/exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java
b/exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java
index e6645fb100b..ff9b4f456a2 100644
---
a/exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java
+++
b/exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java
@@ -22,7 +22,6 @@
import org.apache.drill.exec.record.BatchSchema;
import org.apache.drill.exec.record.metadata.MetadataUtils;
import org.apache.drill.exec.record.metadata.TupleMetadata;
-import org.apache.drill.exec.vector.accessor.TupleWriter;
import org.apache.drill.test.rowSet.RowSet.SingleRowSet;
import java.util.Set;
@@ -65,7 +64,7 @@ public RowSetBuilder(BufferAllocator allocator, TupleMetadata
schema, int capaci
writer = rowSet.writer(capacity);
}
- public TupleWriter writer() { return writer; }
+ public RowSetWriter writer() { return writer; }
/**
* Add a new row using column values passed as variable-length arguments.
Expects
diff --git
a/exec/java-exec/src/test/java/org/apache/drill/test/rowSet/test/TestMapAccessors.java
b/exec/java-exec/src/test/java/org/apache/drill/test/rowSet/test/TestMapAccessors.java
new file mode 100644
index 000..ca660c9313a
--- /dev/null
+++
b/exec/java-exec/src/test/java/org/apache/drill/test/rowSet/test/TestMapAccessors.java
@@ -0,0 +1,429
[jira] [Commented] (DRILL-6810) Disable NULL_IF_NULL NullHandling for functions with ComplexWriter
[
https://issues.apache.org/jira/browse/DRILL-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673692#comment-16673692
]
ASF GitHub Bot commented on DRILL-6810:
---
sohami closed pull request #1509: DRILL-6810: Disable NULL_IF_NULL NullHandling
for functions with Comp…
URL: https://github.com/apache/drill/pull/1509
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillSimpleFunc.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillSimpleFunc.java
index 3600f810e7d..72b08ba6e2b 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillSimpleFunc.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillSimpleFunc.java
@@ -17,7 +17,7 @@
*/
package org.apache.drill.exec.expr;
-public interface DrillSimpleFunc extends DrillFunc{
- public void setup();
- public void eval();
+public interface DrillSimpleFunc extends DrillFunc {
+ void setup();
+ void eval();
}
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/annotations/FunctionTemplate.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/annotations/FunctionTemplate.java
index 3aa73650b7b..5bfb2388afa 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/annotations/FunctionTemplate.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/annotations/FunctionTemplate.java
@@ -103,13 +103,13 @@
DEFAULT(OutputWidthCalculators.DefaultOutputWidthCalculator.INSTANCE),
CLONE(OutputWidthCalculators.CloneOutputWidthCalculator.INSTANCE),
CONCAT(OutputWidthCalculators.ConcatOutputWidthCalculator.INSTANCE),
-// Custom calculator are required for functions that don't fall in to any
pre-defined
+// Custom calculator is required for functions that don't fall into any
predefined
// calculator categories - like replace and lpad
// place holder markers on functions until support
// for CUSTOM calculators is implemented
-// CUSTOM_FIXED_WIDTH_DEFUALT will default to a fixed size - for functions
like
+// CUSTOM_FIXED_WIDTH_DEFAULT will default to a fixed size - for functions
like
// lpad() where the ouput size does not easily map to the input size
-
CUSTOM_FIXED_WIDTH_DEFUALT(OutputWidthCalculators.DefaultOutputWidthCalculator.INSTANCE),
+
CUSTOM_FIXED_WIDTH_DEFAULT(OutputWidthCalculators.DefaultOutputWidthCalculator.INSTANCE),
// CUSTOM CLONE will default to CLONE - for functions like replace() where
the output
// size does not easily map to the input size but is likely to be at most
the size of the input.
CUSTOM_CLONE_DEFAULT(OutputWidthCalculators.CloneOutputWidthCalculator.INSTANCE);
@@ -138,6 +138,9 @@
* Indicates that a method's associated logical operation returns NULL if
* either input is NULL, and therefore that the method must not be called
* with null inputs. (The calling framework must handle NULLs.)
+ *
+ * Not Supported for aggregate functions and for functions with {@link
Output} of type
+ * {@link
org.apache.drill.exec.vector.complex.writer.BaseWriter.ComplexWriter}.
*/
NULL_IF_NULL
}
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillAggFuncHolder.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillAggFuncHolder.java
index e6930f30972..76d416e6a4b 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillAggFuncHolder.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillAggFuncHolder.java
@@ -65,7 +65,6 @@ public DrillAggFuncHolder(
FunctionAttributes attributes,
FunctionInitializer initializer) {
super(attributes, initializer);
-checkArgument(attributes.getNullHandling() == NullHandling.INTERNAL, "An
aggregation function is required to do its own null handling.");
}
@Override
@@ -263,4 +262,9 @@ private void addProtectedBlockHA(ClassGenerator g,
JBlock sub, String body, H
}
+ @Override
+ protected void checkNullHandling(NullHandling nullHandling) {
+checkArgument(nullHandling == NullHandling.INTERNAL,
+"An aggregate function is required to handle null input(s) on its
own.");
+ }
}
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillComplexWriterFuncHolder.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillComplexWriterFuncHolder.java
index 9be1b3a72e9..cb78bec45e9 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillComplexWriterFuncHolder.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillCompl
[jira] [Commented] (DRILL-6806) Start moving code for handling a partition in HashAgg into a separate class
[ https://issues.apache.org/jira/browse/DRILL-6806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673690#comment-16673690 ] ASF GitHub Bot commented on DRILL-6806: --- ilooner commented on issue #1515: DRILL-6806: Moving code for a HashAgg partition into separate class. URL: https://github.com/apache/drill/pull/1515#issuecomment-435512604 @Ben-Zvi Added description of the changes This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Start moving code for handling a partition in HashAgg into a separate class > --- > > Key: DRILL-6806 > URL: https://issues.apache.org/jira/browse/DRILL-6806 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.15.0 > > > Since this involves a lot of refactoring this will be a multiple PR effort. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6768) Improve to_date, to_time and to_timestamp and corresponding cast functions to handle empty string when `drill.exec.functions.cast_empty_string_to_null` option is enable
[
https://issues.apache.org/jira/browse/DRILL-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673694#comment-16673694
]
ASF GitHub Bot commented on DRILL-6768:
---
sohami closed pull request #1494: DRILL-6768: Improve to_date, to_time and
to_timestamp and correspondi…
URL: https://github.com/apache/drill/pull/1494
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/exec/java-exec/src/main/codegen/data/Casts.tdd
b/exec/java-exec/src/main/codegen/data/Casts.tdd
index e43572a8781..31eef193ec5 100644
--- a/exec/java-exec/src/main/codegen/data/Casts.tdd
+++ b/exec/java-exec/src/main/codegen/data/Casts.tdd
@@ -61,10 +61,30 @@
{from: "VarChar", to: "TimeStamp", major: "VarCharDate", alias:
"timestamptype"},
{from: "VarChar", to: "Time", major: "VarCharDate", alias: "timetype"},
+{from: "VarChar", to: "NullableDate", major: "NullableVarCharDate"},
+{from: "VarChar", to: "NullableTimeStamp", major: "NullableVarCharDate"},
+{from: "VarChar", to: "NullableTime", major: "NullableVarCharDate"},
+
+{from: "NullableVarChar", to: "NullableDate", major:
"NullableVarCharDate"},
+{from: "NullableVarChar", to: "NullableTimeStamp", major:
"NullableVarCharDate"},
+{from: "NullableVarChar", to: "NullableTime", major:
"NullableVarCharDate"},
+
{from: "VarBinary", to: "Date", major: "VarBinaryDate", alias: "datetype"},
{from: "VarBinary", to: "TimeStamp", major: "VarBinaryDate", alias:
"timestamptype"},
{from: "VarBinary", to: "Time", major: "VarBinaryDate", alias: "timetype"},
+{from: "VarBinary", to: "NullableDate", major: "NullableVarCharDate"},
+{from: "VarBinary", to: "NullableTimeStamp", major: "NullableVarCharDate"},
+{from: "VarBinary", to: "NullableTime", major: "NullableVarCharDate"},
+
+{from: "NullableVarBinary", to: "NullableDate", major:
"NullableVarCharDate"},
+{from: "NullableVarBinary", to: "NullableTimeStamp", major:
"NullableVarCharDate"},
+{from: "NullableVarBinary", to: "NullableTime", major:
"NullableVarCharDate"},
+
+{from: "NullableVar16Char", to: "NullableDate", major:
"NullableVarCharDate"},
+{from: "NullableVar16Char", to: "NullableTimeStamp", major:
"NullableVarCharDate"},
+{from: "NullableVar16Char", to: "NullableTime", major:
"NullableVarCharDate"},
+
{from: "Date", to: "VarChar", major: "DateVarChar", bufferLength: "10"}
{from: "TimeStamp", to: "VarChar", major: "DateVarChar", bufferLength:
"23"},
{from: "Time", to: "VarChar", major: "DateVarChar", bufferLength: "12"},
@@ -73,6 +93,14 @@
{from: "VarChar", to: "IntervalDay", major: "VarCharInterval"},
{from: "VarChar", to: "IntervalYear", major: "VarCharInterval"},
+{from: "VarChar", to: "NullableInterval", major:
"NullableVarCharInterval"},
+{from: "VarChar", to: "NullableIntervalDay", major:
"NullableVarCharInterval"},
+{from: "VarChar", to: "NullableIntervalYear", major:
"NullableVarCharInterval"},
+
+{from: "NullableVarChar", to: "NullableInterval", major:
"NullableVarCharInterval"},
+{from: "NullableVarChar", to: "NullableIntervalDay", major:
"NullableVarCharInterval"},
+{from: "NullableVarChar", to: "NullableIntervalYear", major:
"NullableVarCharInterval"},
+
{from: "Interval", to: "VarChar", major: "IntervalVarChar", bufferLength:
"65"},
{from: "IntervalYear", to: "VarChar", major: "IntervalYearVarChar",
bufferLength: "35"},
{from: "IntervalDay", to: "VarChar", major: "IntervalDayVarChar",
bufferLength: "43"},
@@ -118,27 +146,27 @@
{from: "VarChar", to: "NullableFloat4", major: "EmptyString",
javaType:"Float", parse:"Float"},
{from: "VarChar", to: "NullableFloat8", major: "EmptyString",
javaType:"Double", parse:"Double"},
-{from: "VarChar", to: "NullableVarDecimal", major:
"EmptyStringVarCharDecimalComplex"},
+{from: "VarChar", to: "NullableVarDecimal", major:
"NullableVarCharDecimalComplex"},
{from: "NullableVarChar", to: "NullableInt", major: "EmptyString",
javaType:"Integer", primeType:"int"},
{from: "NullableVarChar", to: "NullableBigInt", major: "EmptyString",
javaType: "Long", primeType: "long"},
{from: "NullableVarChar", to: "NullableFloat4", major: "EmptyString",
javaType:"Float", parse:"Float"},
{from: "NullableVarChar", to: "NullableFloat8", major: "EmptyString",
javaType:"Double", parse:"Double"},
-{from: "NullableVarChar", to: "NullableVarDecimal", major:
"EmptyStringVarCharDecimalComplex"},
+{from: "NullableVarChar", to: "NullableVarDecimal", major:
"NullableVarCharDecimalComplex"},
{from: "NullableVar16Char", to: "NullableInt", major: "EmptyString",
javaType:"Int
[jira] [Commented] (DRILL-6824) Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the only distinct Values are NULL and Text
[
https://issues.apache.org/jira/browse/DRILL-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673691#comment-16673691
]
ASF GitHub Bot commented on DRILL-6824:
---
sohami closed pull request #1518: DRILL-6824: Handle schema changes in
MapRDBJsonRecordReader
URL: https://github.com/apache/drill/pull/1518
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/MaprDBJsonRecordReader.java
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/MaprDBJsonRecordReader.java
index 0be44e84b09..5b849ea0508 100644
---
a/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/MaprDBJsonRecordReader.java
+++
b/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/MaprDBJsonRecordReader.java
@@ -45,6 +45,7 @@
import org.apache.drill.exec.vector.complex.fn.JsonReaderUtils;
import org.apache.drill.exec.vector.complex.impl.VectorContainerWriter;
import org.apache.hadoop.fs.Path;
+import org.ojai.Document;
import org.ojai.DocumentReader;
import org.ojai.DocumentStream;
import org.ojai.FieldPath;
@@ -77,6 +78,7 @@
public class MaprDBJsonRecordReader extends AbstractRecordReader {
private static final Logger logger =
LoggerFactory.getLogger(MaprDBJsonRecordReader.class);
+ protected enum SchemaState {SCHEMA_UNKNOWN, SCHEMA_INIT, SCHEMA_CHANGE};
protected static final FieldPath[] ID_ONLY_PROJECTION = { ID_FIELD };
@@ -94,16 +96,19 @@
private OperatorContext operatorContext;
protected VectorContainerWriter vectorWriter;
private DBDocumentReaderBase reader;
+ Document document;
+ protected OutputMutator vectorWriterMutator;
private DrillBuf buffer;
private DocumentStream documentStream;
private Iterator documentReaderIterators;
+ private Iterator documentIterator;
private boolean includeId;
private boolean idOnly;
-
+ private SchemaState schemaState;
private boolean projectWholeDocument;
private FieldProjector projector;
@@ -121,11 +126,16 @@
protected OjaiValueWriter valueWriter;
protected DocumentReaderVectorWriter documentWriter;
protected int maxRecordsToRead = -1;
+ protected DBDocumentReaderBase lastDocumentReader;
+ protected Document lastDocument;
public MaprDBJsonRecordReader(MapRDBSubScanSpec subScanSpec,
MapRDBFormatPlugin formatPlugin,
List projectedColumns,
FragmentContext context, int maxRecords) {
this(subScanSpec, formatPlugin, projectedColumns, context);
this.maxRecordsToRead = maxRecords;
+this.lastDocumentReader = null;
+this.lastDocument = null;
+this.schemaState = SchemaState.SCHEMA_UNKNOWN;
}
protected MaprDBJsonRecordReader(MapRDBSubScanSpec subScanSpec,
MapRDBFormatPlugin formatPlugin,
@@ -264,34 +274,40 @@ protected boolean getIgnoreSchemaChange() {
@Override
public void setup(OperatorContext context, OutputMutator output) throws
ExecutionSetupException {
this.vectorWriter = new VectorContainerWriter(output, unionEnabled);
+this.vectorWriterMutator = output;
this.operatorContext = context;
try {
table.setOption(TableOption.EXCLUDEID, !includeId);
documentStream = table.find(condition, scannedFields);
- documentReaderIterators = documentStream.documentReaders().iterator();
-
- if (allTextMode) {
-valueWriter = new AllTextValueWriter(buffer);
- } else if (readNumbersAsDouble) {
-valueWriter = new NumbersAsDoubleValueWriter(buffer);
- } else {
-valueWriter = new OjaiValueWriter(buffer);
- }
-
- if (projectWholeDocument) {
-documentWriter = new ProjectionPassthroughVectorWriter(valueWriter,
projector, includeId);
- } else if (isSkipQuery()) {
-documentWriter = new RowCountVectorWriter(valueWriter);
- } else if (idOnly) {
-documentWriter = new IdOnlyVectorWriter(valueWriter);
- } else {
-documentWriter = new FieldTransferVectorWriter(valueWriter);
- }
+ documentIterator = documentStream.iterator();
+ setupWriter();
} catch (DBException ex) {
throw new ExecutionSetupException(ex);
}
}
+ /*
+ * Setup the valueWriter and documentWriters based on config options
+ */
+ private void setupWriter() {
+if (allTextMode) {
+ valueWriter = new AllTextValueWriter(buffer);
+} else if (readNumbersAsDouble) {
+ valueWriter = new NumbersAsDoubleValueWriter(buffer);
+} else {
+ valueWriter = new OjaiValueWriter(buffer);
+}
+
+if (projectWholeDocument) {
+ documentWriter = new ProjectionPassth
[jira] [Updated] (DRILL-6824) Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the only distinct Values are NULL and Text
[ https://issues.apache.org/jira/browse/DRILL-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6824: - Component/s: Execution - Data Types > Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the > only distinct Values are NULL and Text > > > Key: DRILL-6824 > URL: https://issues.apache.org/jira/browse/DRILL-6824 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Gautam Parai >Assignee: Gautam Parai >Priority: Major > Labels: ready-to-commit > Fix For: 1.15.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Drill Query on MapR DB JSON Table or it View fails with a > SchemaChangeException. The only distinct values is NULL and some Text. > The issue is that when Drill sees a NULL or does not see any values for a > column in the first batch it assumes the type as NULLABLE INT. Subsequently, > when the data shows up and it is different from NULLABLE INT there is a > schema change. Operators e.g. aggregators etc. cannot handle such a Schema > Change and throw a SchemaChangeException. > > One of the short-term solution implemented in this fix: Add a CAST expression > which will cast null values to the target type. Hence, we would never see a > SchemaChange due to NULLs. However, the MapRDB Reader code was written > differently than other reader and was hitting a SchemaChangeException. The > code was changed to make it similar to other scans i.e. emit a new batch > whenever a schema change is encountered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6824) Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the only distinct Values are NULL and Text
[ https://issues.apache.org/jira/browse/DRILL-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6824: - Component/s: (was: Execution - Data Types) > Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the > only distinct Values are NULL and Text > > > Key: DRILL-6824 > URL: https://issues.apache.org/jira/browse/DRILL-6824 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Gautam Parai >Assignee: Gautam Parai >Priority: Major > Labels: ready-to-commit > Fix For: 1.15.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Drill Query on MapR DB JSON Table or it View fails with a > SchemaChangeException. The only distinct values is NULL and some Text. > The issue is that when Drill sees a NULL or does not see any values for a > column in the first batch it assumes the type as NULLABLE INT. Subsequently, > when the data shows up and it is different from NULLABLE INT there is a > schema change. Operators e.g. aggregators etc. cannot handle such a Schema > Change and throw a SchemaChangeException. > > One of the short-term solution implemented in this fix: Add a CAST expression > which will cast null values to the target type. Hence, we would never see a > SchemaChange due to NULLs. However, the MapRDB Reader code was written > differently than other reader and was hitting a SchemaChangeException. The > code was changed to make it similar to other scans i.e. emit a new batch > whenever a schema change is encountered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6824) Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the only distinct Values are NULL and Text
[ https://issues.apache.org/jira/browse/DRILL-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6824: - Component/s: Storage - MapRDB > Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the > only distinct Values are NULL and Text > > > Key: DRILL-6824 > URL: https://issues.apache.org/jira/browse/DRILL-6824 > Project: Apache Drill > Issue Type: Bug > Components: Storage - MapRDB >Affects Versions: 1.14.0 >Reporter: Gautam Parai >Assignee: Gautam Parai >Priority: Major > Labels: ready-to-commit > Fix For: 1.15.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Drill Query on MapR DB JSON Table or it View fails with a > SchemaChangeException. The only distinct values is NULL and some Text. > The issue is that when Drill sees a NULL or does not see any values for a > column in the first batch it assumes the type as NULLABLE INT. Subsequently, > when the data shows up and it is different from NULLABLE INT there is a > schema change. Operators e.g. aggregators etc. cannot handle such a Schema > Change and throw a SchemaChangeException. > > One of the short-term solution implemented in this fix: Add a CAST expression > which will cast null values to the target type. Hence, we would never see a > SchemaChange due to NULLs. However, the MapRDB Reader code was written > differently than other reader and was hitting a SchemaChangeException. The > code was changed to make it similar to other scans i.e. emit a new batch > whenever a schema change is encountered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673788#comment-16673788
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r230527544
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java
##
@@ -362,12 +368,35 @@ public boolean isUserAuthenticationEnabled() {
@Override
public void addRuntimeFilter(RuntimeFilterWritable runtimeFilter) {
-this.runtimeFilterSink.aggregate(runtimeFilter);
+this.runtimeFilterWritable = runtimeFilter;
+if (enableRFWaiting) {
+ condition.signal();
+}
+ }
+
+ @Override
+ public RuntimeFilterWritable getRuntimeFilter() {
+return runtimeFilterWritable;
}
@Override
- public RuntimeFilterSink getRuntimeFilterSink() {
-return runtimeFilterSink;
+ public RuntimeFilterWritable getRuntimeFilter(long maxWaitTime, TimeUnit
timeUnit) {
+if (runtimeFilterWritable != null) {
+ return runtimeFilterWritable;
+}
+if (enableRFWaiting) {
+ lock.lock();
+ try {
+if (runtimeFilterWritable != null) {
+ condition.await(maxWaitTime, timeUnit);
+}
+ } catch (InterruptedException e) {
+logger.debug("Condition was interrupted", e);
+ } finally {
+lock.unlock();
+ }
+}
+return runtimeFilterWritable;
Review comment:
```
lockForRF.lock();
try {
if (runtimeFilterWritable != null) {
return runtimeFilterWritable;
}
if (enableRFWaiting) {
conditionForRF.await(maxWaitTime, timeUnit);
}
return runtimeFilterWritable;
} catch (InterruptedException ex) {
logger.debug("Conditional wait for RF was interrupted", ex);
} finally {
lockForRF.unlock();
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673784#comment-16673784 ] ASF GitHub Bot commented on DRILL-6792: --- sohami commented on a change in pull request #1504: DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf… URL: https://github.com/apache/drill/pull/1504#discussion_r230527876 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ## @@ -159,18 +158,22 @@ BufferAllocator getNewChildAllocator(final String operatorName, @Override void close(); - - /** - * @return - */ - RuntimeFilterSink getRuntimeFilterSink(); - /** * add a RuntimeFilter when the RuntimeFilter receiver belongs to the same MinorFragment * @param runtimeFilter */ public void addRuntimeFilter(RuntimeFilterWritable runtimeFilter); + public RuntimeFilterWritable getRuntimeFilter(); + + /** + * get the RuntimeFilter with a blocking wait, if the waiting option is enabled + * @param maxWaitTime + * @param timeUnit + * @return Review comment: add `description` for `return` tag otherwise there is warnings for it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Find the right probe side fragment to any storage plugin > > > Key: DRILL-6792 > URL: https://issues.apache.org/jira/browse/DRILL-6792 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Fix For: 1.15.0 > > > The current implementation of JPPD to find the probe side wrapper depends on > the GroupScan's digest. But there's no promise the GroupScan's digest will > not be changed since it is attached to the RuntimeFilterDef by different > storage plugin implementation logic.So here we assign a unique identifier to > the RuntimeFilter operator, and find the right probe side fragment wrapper by > the runtime filter identifier at the RuntimeFilterRouter class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673787#comment-16673787 ] ASF GitHub Bot commented on DRILL-6792: --- sohami commented on a change in pull request #1504: DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf… URL: https://github.com/apache/drill/pull/1504#discussion_r230519811 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java ## @@ -115,6 +117,10 @@ private final BufferManager bufferManager; private ExecutorState executorState; private final ExecutionControls executionControls; + private boolean enableRuntimeFilter; + private boolean enableRFWaiting; + private Lock lock; + private Condition condition; Review comment: please consider changing name of variable to `lockForRF` and `conditionForRF` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Find the right probe side fragment to any storage plugin > > > Key: DRILL-6792 > URL: https://issues.apache.org/jira/browse/DRILL-6792 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Fix For: 1.15.0 > > > The current implementation of JPPD to find the probe side wrapper depends on > the GroupScan's digest. But there's no promise the GroupScan's digest will > not be changed since it is attached to the RuntimeFilterDef by different > storage plugin implementation logic.So here we assign a unique identifier to > the RuntimeFilter operator, and find the right probe side fragment wrapper by > the runtime filter identifier at the RuntimeFilterRouter class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673789#comment-16673789
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r230520412
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java
##
@@ -362,12 +368,35 @@ public boolean isUserAuthenticationEnabled() {
@Override
public void addRuntimeFilter(RuntimeFilterWritable runtimeFilter) {
-this.runtimeFilterSink.aggregate(runtimeFilter);
+this.runtimeFilterWritable = runtimeFilter;
+if (enableRFWaiting) {
+ condition.signal();
+}
Review comment:
We can add the runtimeFilterWritable under `lock()` to prevent any race
condition although there is no harm with this race. But there is no issue in
adding it under lock either that way its clear that lock and condition variable
is for `runtimeFilterWritable`.
```
lockForRF.lock()
try () {
this.runtimeFilterWritable = runtimeFilter;
if (enableRFWaiting) {
// wake up the waiting thread in RuntimeFilterRecordBatch
conditionForRF.signal();
}
} finally {
lockForRF.unlock()
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673785#comment-16673785
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r230520575
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java
##
@@ -362,12 +368,35 @@ public boolean isUserAuthenticationEnabled() {
@Override
public void addRuntimeFilter(RuntimeFilterWritable runtimeFilter) {
-this.runtimeFilterSink.aggregate(runtimeFilter);
+this.runtimeFilterWritable = runtimeFilter;
+if (enableRFWaiting) {
+ condition.signal();
+}
+ }
+
+ @Override
+ public RuntimeFilterWritable getRuntimeFilter() {
+return runtimeFilterWritable;
Review comment:
```
lockForRF.lock();
try {
return runtimeFilterWritable;
} finally {
lockForRF.unlock();
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673786#comment-16673786 ] ASF GitHub Bot commented on DRILL-6792: --- sohami commented on a change in pull request #1504: DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf… URL: https://github.com/apache/drill/pull/1504#discussion_r230515093 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MurmurHash3.java ## @@ -20,7 +20,6 @@ import io.netty.buffer.DrillBuf; import io.netty.util.internal.PlatformDependent; - Review comment: Still seeing this file. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Find the right probe side fragment to any storage plugin > > > Key: DRILL-6792 > URL: https://issues.apache.org/jira/browse/DRILL-6792 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Fix For: 1.15.0 > > > The current implementation of JPPD to find the probe side wrapper depends on > the GroupScan's digest. But there's no promise the GroupScan's digest will > not be changed since it is attached to the RuntimeFilterDef by different > storage plugin implementation logic.So here we assign a unique identifier to > the RuntimeFilter operator, and find the right probe side fragment wrapper by > the runtime filter identifier at the RuntimeFilterRouter class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6825) Applying different hash function according to data types and data size
[ https://issues.apache.org/jira/browse/DRILL-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673860#comment-16673860 ] Boaz Ben-Zvi commented on DRILL-6825: - We were talking a while back about changing the use of hash functions, instead of generating code – make a virtual call that computes the hash value for each type of vector (similar to the `copyEntry()` in the `ValueVector`). And then compute the hash value by iterating over the key columns (similar to `appendRow()` in `VectorContainer` - though need to know which columns belong to the key). Also this would remove the hash value computation from the HashTable. Don't remember if a Jira was opened for that work. This would definitely simplify using different hash functions, per each datatype. One last point - may need to keep various integers hashing compatibility - so best if `HashValue(X as smallIint) == HashValue(X as int) == HashValue(X as bigint)` > Applying different hash function according to data types and data size > -- > > Key: DRILL-6825 > URL: https://issues.apache.org/jira/browse/DRILL-6825 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Codegen >Reporter: weijie.tong >Priority: Major > Fix For: 1.16.0 > > > Different hash functions have different performance according to different > data types and data size. We should choose a right one to apply not just > Murmurhash. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6611) Add [meta]-[Enter] js handler for query form submission
[
https://issues.apache.org/jira/browse/DRILL-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673864#comment-16673864
]
ASF GitHub Bot commented on DRILL-6611:
---
kkhatua opened a new pull request #1521: DRILL-6611: Add Ctrl+Enter support for
query submission
URL: https://github.com/apache/drill/pull/1521
1. BugFix on parent commit: Ensure query submission is done with user name
when impersonation is enabled.
2. Support non-Mac browsers
3. Support keyboard submission for profile pages with Edit Query tab.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add [meta]-[Enter] js handler for query form submission
> ---
>
> Key: DRILL-6611
> URL: https://issues.apache.org/jira/browse/DRILL-6611
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.14.0
>Reporter: Bob Rudis
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> The new ACE-based SQL query editor is great. Being able to submit the form
> without using a mouse would be even better.
> Adding:
>
> {noformat}
> document.getElementById('queryForm')
> .addEventListener('keydown', function(e) {
> if (!(e.keyCode == 13 && e.metaKey)) return;
> if (e.target.form) doSubmitQueryWithUserName();
> });
> {noformat}
> {{to ./exec/java-exec/src/main/resources/rest/query/query.ftl adds such
> support.}}
> I can file a PR with the code if desired.
> --
> Functionality (for the documentation):
> This JIRA's commit introduces the following to Drill:
> When composing queries in the web query editor it is now possible to submit
> the query text by using the {{Meta+Enter}} key combination. This will trigger
> the same action as pressing the {{Submit}} button. On Mac keyboards
> {{Meta+Enter}} is {{Cmd+Enter}}. On Windows or Linux is {{Ctrl+Enter}} though
> Linux users may have keymapped the {{Meta}} key to another physical keyboard
> key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6611) Add [meta]-[Enter] js handler for query form submission
[
https://issues.apache.org/jira/browse/DRILL-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673865#comment-16673865
]
ASF GitHub Bot commented on DRILL-6611:
---
kkhatua commented on issue #1521: DRILL-6611: Add Ctrl+Enter support for query
submission
URL: https://github.com/apache/drill/pull/1521#issuecomment-435548255
@arina-ielchiieva please review this. There are fixes on top of @hrbrmstr
's original commit.
Tested with impersonation enabled, which was the only bug. Added support for
non-Mac machines as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add [meta]-[Enter] js handler for query form submission
> ---
>
> Key: DRILL-6611
> URL: https://issues.apache.org/jira/browse/DRILL-6611
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.14.0
>Reporter: Bob Rudis
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> The new ACE-based SQL query editor is great. Being able to submit the form
> without using a mouse would be even better.
> Adding:
>
> {noformat}
> document.getElementById('queryForm')
> .addEventListener('keydown', function(e) {
> if (!(e.keyCode == 13 && e.metaKey)) return;
> if (e.target.form) doSubmitQueryWithUserName();
> });
> {noformat}
> {{to ./exec/java-exec/src/main/resources/rest/query/query.ftl adds such
> support.}}
> I can file a PR with the code if desired.
> --
> Functionality (for the documentation):
> This JIRA's commit introduces the following to Drill:
> When composing queries in the web query editor it is now possible to submit
> the query text by using the {{Meta+Enter}} key combination. This will trigger
> the same action as pressing the {{Submit}} button. On Mac keyboards
> {{Meta+Enter}} is {{Cmd+Enter}}. On Windows or Linux is {{Ctrl+Enter}} though
> Linux users may have keymapped the {{Meta}} key to another physical keyboard
> key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673904#comment-16673904
]
ASF GitHub Bot commented on DRILL-6792:
---
weijietong commented on a change in pull request #1504: DRILL-6792: Find the
right probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r230543377
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java
##
@@ -362,12 +368,35 @@ public boolean isUserAuthenticationEnabled() {
@Override
public void addRuntimeFilter(RuntimeFilterWritable runtimeFilter) {
-this.runtimeFilterSink.aggregate(runtimeFilter);
+this.runtimeFilterWritable = runtimeFilter;
+if (enableRFWaiting) {
+ condition.signal();
+}
Review comment:
Here there's a usage bug about the signal ,I have fixed that on the latest
commit. But I don't want to add unnecessary lock to the runtimeFilterWritable
var. Drill's code has lots of sync usage. I always try to thin the block size,
though it maybe introduce bugs. If you insist on this, I am open to do it at
another commit. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6825) Applying different hash function according to data types and data size
[ https://issues.apache.org/jira/browse/DRILL-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673915#comment-16673915 ] weijie.tong commented on DRILL-6825: Thanks for your advice [~ben-zvi]. I am evaluating the hash function choosing to different data types , data size, hash code size by reference the ClickHouse's implementation. Your advice about the implementation style sounds good to me. So there's no issue about that now. I think there's no need to keep hashing compatibility,just need to make the choosing strategy is all applied by different parts of Drill's codes like hash exchange, hash aggregate, hash join. To make hash value by different hash functions have the same hash value is not viable. > Applying different hash function according to data types and data size > -- > > Key: DRILL-6825 > URL: https://issues.apache.org/jira/browse/DRILL-6825 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Codegen >Reporter: weijie.tong >Priority: Major > Fix For: 1.16.0 > > > Different hash functions have different performance according to different > data types and data size. We should choose a right one to apply not just > Murmurhash. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
