[jira] [Created] (DRILL-8200) Update hadoop-common to ≥ 3.2.3 for CVE-2022-26612
James Turton created DRILL-8200: --- Summary: Update hadoop-common to ≥ 3.2.3 for CVE-2022-26612 Key: DRILL-8200 URL: https://issues.apache.org/jira/browse/DRILL-8200 Project: Apache Drill Issue Type: Bug Components: library Affects Versions: 1.20.0 Reporter: James Turton Assignee: James Turton Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (DRILL-8155) Introduce new plugin authentication modes
[
https://issues.apache.org/jira/browse/DRILL-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527577#comment-17527577
]
ASF GitHub Bot commented on DRILL-8155:
---
vvysotskyi commented on code in PR #2516:
URL: https://github.com/apache/drill/pull/2516#discussion_r857767899
##
contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcConventionFactory.java:
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.jdbc;
+
+import org.apache.calcite.sql.SqlDialect;
+
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public class JdbcConventionFactory {
+
+ private final Map CACHE = new
ConcurrentHashMap<>();
Review Comment:
Yes, it would be also nice to have an expiry time on it
> Introduce new plugin authentication modes
> -
>
> Key: DRILL-8155
> URL: https://issues.apache.org/jira/browse/DRILL-8155
> Project: Apache Drill
> Issue Type: Improvement
> Components: Security
>Affects Versions: 1.20.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: Future
>
>
> At present, Drill storage plugins can use a shared set of credentials to
> access storage on behalf of Drill users or, in a subset of cases belonging to
> the broader Hadoop family, they can impersonate the Drill user when
> drill.exec.impersonation.enabled = true. An important but missing auth mode
> is [what is termed "user translation" in
> Trino|[https://docs.starburst.io/latest/security/impersonation.html].] Under
> user translation, the active Drill user is translated to a user known to the
> external storage by means of a translation table that associates Drill users
> with their credentials for the external storage. No support for user
> impersonation in the external storage is required in this mode. This ticket
> proposes that we add establish a design pattern that adds support for this
> auth mode to Drill storage plugins.
> Another present day limitation is that impersonation, for the plugins that
> support it, is toggled by a global switch. We propose here that the auth
> mode chosen for a plugin should be independent of the auth modes chosen for
> other plugins, by a move of this option into their respective storage configs.
> Finally, while a standardised means of choosing an authentication mode is
> desired, note that not every storage plugin needs to, or can, support every
> mode.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (DRILL-8155) Introduce new plugin authentication modes
[
https://issues.apache.org/jira/browse/DRILL-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527493#comment-17527493
]
ASF GitHub Bot commented on DRILL-8155:
---
jnturton commented on code in PR #2516:
URL: https://github.com/apache/drill/pull/2516#discussion_r857621992
##
contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcConventionFactory.java:
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.jdbc;
+
+import org.apache.calcite.sql.SqlDialect;
+
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public class JdbcConventionFactory {
+
+ private final Map CACHE = new
ConcurrentHashMap<>();
Review Comment:
@vvysotskyi Thanks, I've replaced it with a Guava cache that has a maximum
size of 100 before LRU evictions begin. I wonder if I should set an expiry time
on it too?
> Introduce new plugin authentication modes
> -
>
> Key: DRILL-8155
> URL: https://issues.apache.org/jira/browse/DRILL-8155
> Project: Apache Drill
> Issue Type: Improvement
> Components: Security
>Affects Versions: 1.20.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: Future
>
>
> At present, Drill storage plugins can use a shared set of credentials to
> access storage on behalf of Drill users or, in a subset of cases belonging to
> the broader Hadoop family, they can impersonate the Drill user when
> drill.exec.impersonation.enabled = true. An important but missing auth mode
> is [what is termed "user translation" in
> Trino|[https://docs.starburst.io/latest/security/impersonation.html].] Under
> user translation, the active Drill user is translated to a user known to the
> external storage by means of a translation table that associates Drill users
> with their credentials for the external storage. No support for user
> impersonation in the external storage is required in this mode. This ticket
> proposes that we add establish a design pattern that adds support for this
> auth mode to Drill storage plugins.
> Another present day limitation is that impersonation, for the plugins that
> support it, is toggled by a global switch. We propose here that the auth
> mode chosen for a plugin should be independent of the auth modes chosen for
> other plugins, by a move of this option into their respective storage configs.
> Finally, while a standardised means of choosing an authentication mode is
> desired, note that not every storage plugin needs to, or can, support every
> mode.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (DRILL-8155) Introduce new plugin authentication modes
[
https://issues.apache.org/jira/browse/DRILL-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527491#comment-17527491
]
ASF GitHub Bot commented on DRILL-8155:
---
jnturton commented on code in PR #2516:
URL: https://github.com/apache/drill/pull/2516#discussion_r857621992
##
contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcConventionFactory.java:
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.jdbc;
+
+import org.apache.calcite.sql.SqlDialect;
+
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public class JdbcConventionFactory {
+
+ private final Map CACHE = new
ConcurrentHashMap<>();
Review Comment:
Thanks, I've replaced it with a Guava cache that has a maximum size of 100
before evictions LRU begin. I wonder if I should set an expiry time on it too?
> Introduce new plugin authentication modes
> -
>
> Key: DRILL-8155
> URL: https://issues.apache.org/jira/browse/DRILL-8155
> Project: Apache Drill
> Issue Type: Improvement
> Components: Security
>Affects Versions: 1.20.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: Future
>
>
> At present, Drill storage plugins can use a shared set of credentials to
> access storage on behalf of Drill users or, in a subset of cases belonging to
> the broader Hadoop family, they can impersonate the Drill user when
> drill.exec.impersonation.enabled = true. An important but missing auth mode
> is [what is termed "user translation" in
> Trino|[https://docs.starburst.io/latest/security/impersonation.html].] Under
> user translation, the active Drill user is translated to a user known to the
> external storage by means of a translation table that associates Drill users
> with their credentials for the external storage. No support for user
> impersonation in the external storage is required in this mode. This ticket
> proposes that we add establish a design pattern that adds support for this
> auth mode to Drill storage plugins.
> Another present day limitation is that impersonation, for the plugins that
> support it, is toggled by a global switch. We propose here that the auth
> mode chosen for a plugin should be independent of the auth modes chosen for
> other plugins, by a move of this option into their respective storage configs.
> Finally, while a standardised means of choosing an authentication mode is
> desired, note that not every storage plugin needs to, or can, support every
> mode.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (DRILL-8155) Introduce new plugin authentication modes
[
https://issues.apache.org/jira/browse/DRILL-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527492#comment-17527492
]
ASF GitHub Bot commented on DRILL-8155:
---
jnturton commented on code in PR #2516:
URL: https://github.com/apache/drill/pull/2516#discussion_r857621992
##
contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcConventionFactory.java:
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.jdbc;
+
+import org.apache.calcite.sql.SqlDialect;
+
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public class JdbcConventionFactory {
+
+ private final Map CACHE = new
ConcurrentHashMap<>();
Review Comment:
@vvysotskyi Thanks, I've replaced it with a Guava cache that has a maximum
size of 100 before evictions LRU begin. I wonder if I should set an expiry time
on it too?
> Introduce new plugin authentication modes
> -
>
> Key: DRILL-8155
> URL: https://issues.apache.org/jira/browse/DRILL-8155
> Project: Apache Drill
> Issue Type: Improvement
> Components: Security
>Affects Versions: 1.20.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: Future
>
>
> At present, Drill storage plugins can use a shared set of credentials to
> access storage on behalf of Drill users or, in a subset of cases belonging to
> the broader Hadoop family, they can impersonate the Drill user when
> drill.exec.impersonation.enabled = true. An important but missing auth mode
> is [what is termed "user translation" in
> Trino|[https://docs.starburst.io/latest/security/impersonation.html].] Under
> user translation, the active Drill user is translated to a user known to the
> external storage by means of a translation table that associates Drill users
> with their credentials for the external storage. No support for user
> impersonation in the external storage is required in this mode. This ticket
> proposes that we add establish a design pattern that adds support for this
> auth mode to Drill storage plugins.
> Another present day limitation is that impersonation, for the plugins that
> support it, is toggled by a global switch. We propose here that the auth
> mode chosen for a plugin should be independent of the auth modes chosen for
> other plugins, by a move of this option into their respective storage configs.
> Finally, while a standardised means of choosing an authentication mode is
> desired, note that not every storage plugin needs to, or can, support every
> mode.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (DRILL-8188) Convert HDF5 format to EVF2
[ https://issues.apache.org/jira/browse/DRILL-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527369#comment-17527369 ] ASF GitHub Bot commented on DRILL-8188: --- luocooong commented on PR #2515: URL: https://github.com/apache/drill/pull/2515#issuecomment-1108226622 > Found the bug. It is in `ColumnBuilder` which seems to be missing code to handle an unprojected repeated list. This bug then caused the other "bugs" that we discussed in the review: those bits of code are working as they should. The problem is that the result set loader is materializing a vector when it should not. It will take some time to remember how all this stuff works. Stay tuned. @paul-rogers Great! Thank you for the quick work. From the end of the most recent discussion, I completely rejected my previous code revision, guess that the unprojected handle might have been lost. Actually, I've added this function locally, but I'm not sure it's correct. Would you mind checking mine before you submit the new revision? > Convert HDF5 format to EVF2 > --- > > Key: DRILL-8188 > URL: https://issues.apache.org/jira/browse/DRILL-8188 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.0 >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > Use EVF V2 instead of old V1. > Also, fixed a few bugs in V2 framework. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (DRILL-8188) Convert HDF5 format to EVF2
[ https://issues.apache.org/jira/browse/DRILL-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527363#comment-17527363 ] ASF GitHub Bot commented on DRILL-8188: --- paul-rogers commented on PR #2515: URL: https://github.com/apache/drill/pull/2515#issuecomment-1108207457 Found the bug. It is in `ColumnBuilder` which seems to be missing code to handle an unprojected repeated list. This bug then caused the other "bugs" that we discussed in the review: those bits of code are working as they should. The problem is that the result set loader is materializing a vector when it should not. It will take some time to remember how all this stuff works. Stay tuned. > Convert HDF5 format to EVF2 > --- > > Key: DRILL-8188 > URL: https://issues.apache.org/jira/browse/DRILL-8188 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.0 >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > Use EVF V2 instead of old V1. > Also, fixed a few bugs in V2 framework. -- This message was sent by Atlassian Jira (v8.20.7#820007)
