[jira] [Commented] (HIVE-24083) hcatalog error in Hadoop 3.3.0: authentication type needed
[
https://issues.apache.org/jira/browse/HIVE-24083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607039#comment-17607039
]
Stamatis Zampetakis commented on HIVE-24083:
{quote}Should I create another PR for master branch?{quote}
Yes a separate PR is needed for master.
> hcatalog error in Hadoop 3.3.0: authentication type needed
> --
>
> Key: HIVE-24083
> URL: https://issues.apache.org/jira/browse/HIVE-24083
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 3.1.2
>Reporter: Javier J. Salmeron Garcia
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Using Hive 3.1.2, webhcat fails to start in Hadoop 3.3.0 with the following
> error:
> ```
> javax.servlet.ServletException: Authentication type must be specified:
> simple|kerberos
> ```
> I tried in Hadoop 3.2.1 with the exact settings and it starts without issues:
>
> ```
> webhcat: /tmp/hadoop-3.2.1//bin/hadoop jar
> /opt/bitnami/hadoop/hive/hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-3.1.2.jar
> org.apache.hive.hcatalog.templeton.Main
> webhcat: starting ... started.
> webhcat: done
> ```
>
> I can provide more logs if needed. Detected authentication settings:
>
> ```
> hadoop.http.authentication.simple.anonymous.allowed=true
> hadoop.http.authentication.type=simple
> hadoop.security.authentication=simple
> ipc.client.fallback-to-simple-auth-allowed=false
> yarn.timeline-service.http-authentication.simple.anonymous.allowed=true
> yarn.timeline-service.http-authentication.type=simple
> ```
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-24083) hcatalog error in Hadoop 3.3.0: authentication type needed
[ https://issues.apache.org/jira/browse/HIVE-24083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607036#comment-17607036 ] Stamatis Zampetakis commented on HIVE-24083: I suppose this is the same problem with the one I reported just now under HIVE-26549. > hcatalog error in Hadoop 3.3.0: authentication type needed > -- > > Key: HIVE-24083 > URL: https://issues.apache.org/jira/browse/HIVE-24083 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 3.1.2 >Reporter: Javier J. Salmeron Garcia >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Using Hive 3.1.2, webhcat fails to start in Hadoop 3.3.0 with the following > error: > ``` > javax.servlet.ServletException: Authentication type must be specified: > simple|kerberos > ``` > I tried in Hadoop 3.2.1 with the exact settings and it starts without issues: > > ``` > webhcat: /tmp/hadoop-3.2.1//bin/hadoop jar > /opt/bitnami/hadoop/hive/hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-3.1.2.jar > org.apache.hive.hcatalog.templeton.Main > webhcat: starting ... started. > webhcat: done > ``` > > I can provide more logs if needed. Detected authentication settings: > > ``` > hadoop.http.authentication.simple.anonymous.allowed=true > hadoop.http.authentication.type=simple > hadoop.security.authentication=simple > ipc.client.fallback-to-simple-auth-allowed=false > yarn.timeline-service.http-authentication.simple.anonymous.allowed=true > yarn.timeline-service.http-authentication.type=simple > ``` > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-24083) hcatalog error in Hadoop 3.3.0: authentication type needed
[ https://issues.apache.org/jira/browse/HIVE-24083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583670#comment-17583670 ] Zhiguo Wu commented on HIVE-24083: -- [~zabetak] Also, I prefer if we can keep track and fix the compatible issue with Hive 3.1.3 and Hadoop 3.3.4, Since they are included in Apache Bigtop stack(which provide a fully compatible version of bigdata components, and now Bigtop 3.2.0 contains Hive 3.1.3 and Hadoop 3.3.4), it'll affect lots of users if there are compatible issues and we didn't fix it. But if you are too busy to work on this, I'll do my best and help with it. > hcatalog error in Hadoop 3.3.0: authentication type needed > -- > > Key: HIVE-24083 > URL: https://issues.apache.org/jira/browse/HIVE-24083 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 3.1.2 >Reporter: Javier J. Salmeron Garcia >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Using Hive 3.1.2, webhcat fails to start in Hadoop 3.3.0 with the following > error: > ``` > javax.servlet.ServletException: Authentication type must be specified: > simple|kerberos > ``` > I tried in Hadoop 3.2.1 with the exact settings and it starts without issues: > > ``` > webhcat: /tmp/hadoop-3.2.1//bin/hadoop jar > /opt/bitnami/hadoop/hive/hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-3.1.2.jar > org.apache.hive.hcatalog.templeton.Main > webhcat: starting ... started. > webhcat: done > ``` > > I can provide more logs if needed. Detected authentication settings: > > ``` > hadoop.http.authentication.simple.anonymous.allowed=true > hadoop.http.authentication.type=simple > hadoop.security.authentication=simple > ipc.client.fallback-to-simple-auth-allowed=false > yarn.timeline-service.http-authentication.simple.anonymous.allowed=true > yarn.timeline-service.http-authentication.type=simple > ``` > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-24083) hcatalog error in Hadoop 3.3.0: authentication type needed
[
https://issues.apache.org/jira/browse/HIVE-24083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583397#comment-17583397
]
Zhiguo Wu commented on HIVE-24083:
--
[~zabetak]
With patch: [https://github.com/apache/hive/pull/3543]
Now this can be reproduced in master branch
{code:java}
ERROR | 23 Aug 2022 06:34:27,027 | org.apache.hive.hcatalog.templeton.Main |
Server failed to start:
javax.servlet.ServletException: Authentication type must be specified:
simple|kerberos|
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:164)
~[hadoop-auth-3.3.4.jar:?]
at
org.apache.hadoop.security.authentication.server.ProxyUserAuthenticationFilter.init(ProxyUserAuthenticationFilter.java:57)
~[hadoop-common-3.3.4.jar:?]
at
org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:140)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:731)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
~[?:1.8.0_342]
at
java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
~[?:1.8.0_342]
at
java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
~[?:1.8.0_342]
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:755)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at org.eclipse.jetty.server.Server.start(Server.java:423)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
~[jetty-runner-9.4.40.v20210413.jar:9.4.40.v20210413]
at org.apache.hive.hcatalog.templeton.Main.runServer(Main.java:255)
~[hive-webhcat-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
at org.apache.hive.hcatalog.templeton.Main.run(Main.java:147)
[hive-webhcat-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
at org.apache.hive.hcatalog.templeton.Main.main(Main.java:394)
[hive-webhcat-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_342]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_342]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_342]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_342]
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
[hadoop-common-3.3.4.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
[hadoop-common-3.3.4.jar:?] {code}
> hcatalog error in Hadoop 3.3.0: authentication type needed
> --
>
> Key: HIVE-24083
> URL: https://issues.apache.org/jira/browse/HIVE-24083
> Project: Hive
> Issue Type: Bug
> Components:
[jira] [Commented] (HIVE-24083) hcatalog error in Hadoop 3.3.0: authentication type needed
[ https://issues.apache.org/jira/browse/HIVE-24083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1758#comment-1758 ] Zhiguo Wu commented on HIVE-24083: -- [~zabetak] Hi Stamatis. I tried to reproduce this in master branch, but it seems there is another problem we need to fix first. https://issues.apache.org/jira/browse/HIVE-26286 > hcatalog error in Hadoop 3.3.0: authentication type needed > -- > > Key: HIVE-24083 > URL: https://issues.apache.org/jira/browse/HIVE-24083 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 3.1.2 >Reporter: Javier J. Salmeron Garcia >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Using Hive 3.1.2, webhcat fails to start in Hadoop 3.3.0 with the following > error: > ``` > javax.servlet.ServletException: Authentication type must be specified: > simple|kerberos > ``` > I tried in Hadoop 3.2.1 with the exact settings and it starts without issues: > > ``` > webhcat: /tmp/hadoop-3.2.1//bin/hadoop jar > /opt/bitnami/hadoop/hive/hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-3.1.2.jar > org.apache.hive.hcatalog.templeton.Main > webhcat: starting ... started. > webhcat: done > ``` > > I can provide more logs if needed. Detected authentication settings: > > ``` > hadoop.http.authentication.simple.anonymous.allowed=true > hadoop.http.authentication.type=simple > hadoop.security.authentication=simple > ipc.client.fallback-to-simple-auth-allowed=false > yarn.timeline-service.http-authentication.simple.anonymous.allowed=true > yarn.timeline-service.http-authentication.type=simple > ``` > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-24083) hcatalog error in Hadoop 3.3.0: authentication type needed
[ https://issues.apache.org/jira/browse/HIVE-24083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17582936#comment-17582936 ] Stamatis Zampetakis commented on HIVE-24083: Hive does not claim to support multiple Hadoop versions. From what I can see for Hive 3.x the supported Hadoop version is [3.1.0|https://github.com/apache/hive/blob/01924c8d54b146dee44c30d104fb60fbcdec0e87/pom.xml#L151]. There are no guarantees or requirements to work with newer or older Hadoop versions so this may never get fixed in 3.x releases. If the problem affects master then please include steps to reproduce and unit/integration tests as part of the PR. > hcatalog error in Hadoop 3.3.0: authentication type needed > -- > > Key: HIVE-24083 > URL: https://issues.apache.org/jira/browse/HIVE-24083 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 3.1.2 >Reporter: Javier J. Salmeron Garcia >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Using Hive 3.1.2, webhcat fails to start in Hadoop 3.3.0 with the following > error: > ``` > javax.servlet.ServletException: Authentication type must be specified: > simple|kerberos > ``` > I tried in Hadoop 3.2.1 with the exact settings and it starts without issues: > > ``` > webhcat: /tmp/hadoop-3.2.1//bin/hadoop jar > /opt/bitnami/hadoop/hive/hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-3.1.2.jar > org.apache.hive.hcatalog.templeton.Main > webhcat: starting ... started. > webhcat: done > ``` > > I can provide more logs if needed. Detected authentication settings: > > ``` > hadoop.http.authentication.simple.anonymous.allowed=true > hadoop.http.authentication.type=simple > hadoop.security.authentication=simple > ipc.client.fallback-to-simple-auth-allowed=false > yarn.timeline-service.http-authentication.simple.anonymous.allowed=true > yarn.timeline-service.http-authentication.type=simple > ``` > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-24083) hcatalog error in Hadoop 3.3.0: authentication type needed
[
https://issues.apache.org/jira/browse/HIVE-24083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564223#comment-17564223
]
Anmol Sundaram commented on HIVE-24083:
---
I was able to get it run by the following (temporary?) fix :
diff --git
a/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java
b/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java
index d183b2e61b..5e5c4132f4 100644
---
a/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java
+++
b/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java
@@ -38,6 +38,8 @@
import org.apache.hadoop.hive.shims.Utils;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.security.authentication.client.PseudoAuthenticator;
+import org.apache.hadoop.security.authentication.server.AuthenticationFilter;
+import
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler;
import
org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler;
import org.apache.hadoop.security.SecurityUtil;
import org.apache.hadoop.util.GenericOptionsParser;
@@ -269,6 +271,11 @@ public FilterHolder makeAuthFilter() throws IOException \{
authFilter.setInitParameter("dfs.web.authentication.kerberos.keytab",
conf.kerberosKeytab());
}
+
+authFilter.setInitParameter(AuthenticationFilter.AUTH_TYPE,
UserGroupInformation.isSecurityEnabled() ?
+KerberosAuthenticationHandler.TYPE :
+PseudoAuthenticationHandler.TYPE);
+
return authFilter;
}
> hcatalog error in Hadoop 3.3.0: authentication type needed
> --
>
> Key: HIVE-24083
> URL: https://issues.apache.org/jira/browse/HIVE-24083
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 3.1.2
>Reporter: Javier J. Salmeron Garcia
>Priority: Minor
>
> Using Hive 3.1.2, webhcat fails to start in Hadoop 3.3.0 with the following
> error:
> ```
> javax.servlet.ServletException: Authentication type must be specified:
> simple|kerberos
> ```
> I tried in Hadoop 3.2.1 with the exact settings and it starts without issues:
>
> ```
> webhcat: /tmp/hadoop-3.2.1//bin/hadoop jar
> /opt/bitnami/hadoop/hive/hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-3.1.2.jar
> org.apache.hive.hcatalog.templeton.Main
> webhcat: starting ... started.
> webhcat: done
> ```
>
> I can provide more logs if needed. Detected authentication settings:
>
> ```
> hadoop.http.authentication.simple.anonymous.allowed=true
> hadoop.http.authentication.type=simple
> hadoop.security.authentication=simple
> ipc.client.fallback-to-simple-auth-allowed=false
> yarn.timeline-service.http-authentication.simple.anonymous.allowed=true
> yarn.timeline-service.http-authentication.type=simple
> ```
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-24083) hcatalog error in Hadoop 3.3.0: authentication type needed
[ https://issues.apache.org/jira/browse/HIVE-24083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342320#comment-17342320 ] Ahzaz Hingora commented on HIVE-24083: -- Hadoop 3.3.0 [changed the way AuthFilter|https://github.com/apache/hadoop/commit/4ea6c2f457496461afc63f38ef4cef3ab0efce49#diff-680cfa76ea617a05a7a414540076bf6570a82f244f46e94bd745e3c4d259304a] is initialized. The 3.1.2 webhcat uses [AuthFilter|https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java#L258] directly, rather than using AuthFilterInitializer which puts the required properties in the filter. > hcatalog error in Hadoop 3.3.0: authentication type needed > -- > > Key: HIVE-24083 > URL: https://issues.apache.org/jira/browse/HIVE-24083 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 3.1.2 >Reporter: Javier J. Salmeron Garcia >Priority: Minor > > Using Hive 3.1.2, webhcat fails to start in Hadoop 3.3.0 with the following > error: > ``` > javax.servlet.ServletException: Authentication type must be specified: > simple|kerberos > ``` > I tried in Hadoop 3.2.1 with the exact settings and it starts without issues: > > ``` > webhcat: /tmp/hadoop-3.2.1//bin/hadoop jar > /opt/bitnami/hadoop/hive/hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-3.1.2.jar > org.apache.hive.hcatalog.templeton.Main > webhcat: starting ... started. > webhcat: done > ``` > > I can provide more logs if needed. Detected authentication settings: > > ``` > hadoop.http.authentication.simple.anonymous.allowed=true > hadoop.http.authentication.type=simple > hadoop.security.authentication=simple > ipc.client.fallback-to-simple-auth-allowed=false > yarn.timeline-service.http-authentication.simple.anonymous.allowed=true > yarn.timeline-service.http-authentication.type=simple > ``` > -- This message was sent by Atlassian Jira (v8.3.4#803005)
