Hello ozone community
I'm getting the following error when trying to run any action in SCM with
kerberos enabled:
Caused by: java.io.IOException: Couldn't setup connection for
myu...@example.com to scm-server.example.com/10.XXX.YYY.ZZZ:9961
...
Caused by:
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
GSS initiate failed
when I check the SCM logs and the service starts properly
[Listener at 0.0.0.0/9860] INFO
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
Using keytab /opt/ozone/keytabs/spnego.keytab, for principal HTTP/
scm-server.example....@example.com
[Listener at 0.0.0.0/9860] INFO
org.eclipse.jetty.server.handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@24a1c17f
{logs,/logs,file:///var/log/ozone/,AVAILABLE}
[Listener at 0.0.0.0/9860] INFO
org.eclipse.jetty.server.handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@532721fd
{static,/static,jar:file:/opt/ozone/share/ozone/lib/hadoop-hdds-server-scm-1.0.0.jar!/webapps/static,AVAILABLE}
[Listener at 0.0.0.0/9860] INFO
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
Using keytab /opt/ozone/keytabs/spnego.keytab, for principal HTTP/
scm-server.example....@example.com
[Listener at 0.0.0.0/9860] INFO
org.eclipse.jetty.server.handler.ContextHandler: Started
o.e.j.w.WebAppContext@62ddd21b
{scm,/,file:///tmp/jetty-0_0_0_0-9877-hadoop-hdds-server-scm-1_0_0_jar-_-any-4975172152496600541.dir/webapp/,AVAILABLE}{jar:file:/opt/ozone/share/ozone/lib/hadoop-hdds-server-scm-1.0.0.jar!/webapps/scm}
[Listener at 0.0.0.0/9860] INFO
org.eclipse.jetty.util.ssl.SslContextFactory: x509=X509@783efb48(
scm-server.example.com,h=[scm-server.example.com, scm-server],w=[]) for
Server@c446b14
[provider=null,keyStore=file:///opt/ozone/jks/scm-server.jks,trustStore=file:///opt/ozone/jks/cacerts]
[Listener at 0.0.0.0/9860] INFO org.eclipse.jetty.server.AbstractConnector:
Started ServerConnector@13006998{SSL,[ssl, http/1.1]}{0.0.0.0:9877}
[Listener at 0.0.0.0/9860] INFO org.eclipse.jetty.server.Server: Started
@2301ms
[Listener at 0.0.0.0/9860] INFO
org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink prometheus started
[Listener at 0.0.0.0/9860] INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Registered sink
prometheus
[Listener at 0.0.0.0/9860] INFO
org.apache.hadoop.hdds.server.http.BaseHttpServer: HTTPS server of scm
listening at https://0.0.0.0:9877
and I only see this message:
[Socket Reader #1 for port 9961] WARN
SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for
10.XXX.YYY.ZZZ:52306:null
(GSS initiate failed) with true cause: (GSS initiate failed)
I used this template to build my configs:
https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
my auth_to_local currently is like this:
<property>
<name>hadoop.security.auth_to_local</name>
<value>RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//
RULE:[2:$1@$0](.*)s/.*/scm/
DEFAULT</value>
</property>
I already tested the rules and they seems to be OK:
$ hadoop kerbname myu...@example.com
Name: myu...@example.com to myuser
$ hadoop kerbname scm/scm-server.example....@example.com
Name: scm/scm-server.example....@example.com to scm
Also tried the following auth_to_local, but the result is the same:
<property>
<name>hadoop.security.auth_to_local</name>
<value>RULE:[1:$1@$0](s...@example.com)s/.*/scm/
RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//
RULE:[2:$1@$0](.*)s/.*/root/
DEFAULT</value>
</property>
Have you had this issue before? Could you provide any suggestion for this?
I'm also attaching as reference my templates for the core-site, ozone-site
and hdfs-site.
I'm using ozone 1.0 downloaded from the site
build =
https://github.com/apache/hadoop-ozone.git/28d372ca903b4741131bace09e0339e9161257bb
; compiled by 'sammi' on 2020-08-25T13:04Z
java = 1.8.0_201
But it is the same issue when I build it from source.
Thank you in advance.
Diego
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<property>
<name>dfs.data.transfer.protection</name>
<value>authentication,privacy</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:1019</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:1012</value>
</property>
<property>
<name>dfs.datanode.kerberos.principal</name>
<value>dn/_h...@example.com</value>
</property>
<property>
<name>dfs.datanode.keytab.file</name>
<value>{{ keytabs_dir }}/dn.keytab</value>
</property>
<property>
<name>dfs.web.authentication.kerberos.principal</name>
<value>HTTP/_h...@example.com</value>
</property>
<property>
<name>dfs.web.authentication.kerberos.keytab</name>
<value>{{ keytabs_dir }}/spnego.keytab</value>
</property>
<property>
<name>rpc.metrics.quantile.enable</name>
<value>true</value>
</property>
<property>
<name>rpc.metrics.percentiles.intervals</name>
<value>60,300</value>
</property>
</configuration>
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<property>
<name>fs.ofs.impl</name>
<value>org.apache.hadoop.fs.ozone.RootedOzoneFileSystem</value>
</property>
<property>
<name>fs.o3fs.impl</name>
<value>org.apache.hadoop.fs.ozone.OzoneFileSystem</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>ofs://{{ ozone_om }}</value>
</property>
<property>
<name>dfs.data.transfer.protection</name>
<value>authentication</value>
</property>
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hadoop.http.authentication.simple.anonymous.allowed</name>
<value>false</value>
</property>
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
<property>
<name>hadoop.http.authentication.signature.secret.file</name>
<value>/etc/security/http_secret</value>
</property>
<property>
<name>hadoop.http.authentication.type</name>
<value>kerberos</value>
</property>
<property>
<name>hadoop.http.authentication.kerberos.principal</name>
<value>HTTP/_h...@example.com</value>
</property>
<property>
<name>hadoop.http.authentication.kerberos.keytab</name>
<value>{{ keytabs_dir }}/spnego.keytab</value>
</property>
<property>
<name>hadoop.proxyuser.HTTP.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.HTTP.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.ssl.require.client.cert</name>
<value>false</value>
</property>
<property>
<name>hadoop.ssl.hostname.verifier</name>
<value>DEFAULT</value>
</property>
<property>
<name>hadoop.ssl.keystores.factory.class</name>
<value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
</property>
<property>
<name>hadoop.ssl.server.conf</name>
<value>ssl-server.xml</value>
</property>
<property>
<name>hadoop.ssl.client.conf</name>
<value>ssl-client.xml</value>
</property>
<property>
<name>hadoop.security.key.provider.path</name>
<value>kms://https@{{ kms }}/kms</value>
</property>
<property>
<name>hadoop.security.auth_to_local</name>
<value>RULE:[1:$1@$0](s...@example.com)s/.*/scm/
RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//
RULE:[2:$1@$0](.*)s/.*/root/
DEFAULT</value>
</property>
</configuration>
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<property>
<name>ozone.replication</name>
<value>3</value>
</property>
<property>
<name>ozone.metadata.dirs</name>
<value>{{ ozone_dir }}/meta</value>
</property>
<property>
<name>ozone.handler.type</name>
<value>distributed</value>
</property>
<property>
<name>hdds.block.token.enabled</name>
<value>true</value>
</property>
<!--OZONE SECURITY -->
<property>
<name>ozone.security.enabled</name>
<value>true</value>
</property>
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>ozone.security.http.kerberos.enabled</name>
<value>true</value>
</property>
<property>
<name>ozone.http.filter.initializers</name>
<value>org.apache.hadoop.security.AuthenticationFilterInitializer</value>
</property>
<property>
<name>ozone.acl.enabled</name>
<value>true</value>
</property>
<property>
<name>ozone.acl.authorizer.class</name>
<value>org.apache.hadoop.ozone.security.acl.OzoneNativeAuthorizer</value>
</property>
<property>
<name>ozone.administrators</name>
<value>"{{ ozone_admins }}"</value>
</property>
<!--SCM CONFIGS -->
<property>
<name>ozone.scm.names</name>
<value>{{ ozone_scm_names }}</value>
</property>
<property>
<name>ozone.scm.client.address</name>
<value>{{ ozone_scm_client_address }}</value>
</property>
<property>
<name>ozone.scm.block.client.address</name>
<value>{{ ozone_scm_block_client_address }}</value>
</property>
<property>
<name>ozone.scm.pipeline.creation.interval</name>
<value>30s</value>
</property>
<property>
<name>ozone.scm.pipeline.owner.container.count</name>
<value>1</value>
</property>
<property>
<name>ozone.scm.container.size</name>
<value>{{ ozone_scm_container_size }}</value>
</property>
<property>
<name>ozone.scm.datanode.id.dir</name>
<value>{{ ozone_dir }}/</value>
</property>
<property>
<name>ozone.scm.datanode.id</name>
<value>{{ ozone_dir }}/datanode.id</value>
</property>
<!--SCM SECURITY CONFIGS -->
<property>
<name>hdds.scm.kerberos.principal</name>
<value>scm/_h...@example.com</value>
</property>
<property>
<name>hdds.scm.kerberos.keytab.file</name>
<value>{{ keytabs_dir }}/scm.keytab</value>
</property>
<property>
<name>hdds.scm.http.auth.type</name>
<value>kerberos</value>
</property>
<property>
<name>hdds.scm.http.auth.kerberos.principal</name>
<value>HTTP/_h...@example.com</value>
</property>
<property>
<name>hdds.scm.http.auth.kerberos.keytab</name>
<value>{{ keytabs_dir }}/spnego.keytab</value>
</property>
<property>
<name>ozone.scm.http.auth.type</name>
<value>kerberos</value>
</property>
<property>
<name>ozone.om.address</name>
<value>{{ ozone_om_address }}</value>
</property>
<property>
<name>ozone.om.http-address</name>
<value>{{ ozone_om_http_address }}</value>
</property>
<!--OM SECURITY CONFIGS -->
<property>
<name>ozone.om.volume.listall.allowed</name>
<value>false</value>
</property>
<property>
<name>ozone.om.kerberos.principal</name>
<value>om/_h...@example.com</value>
</property>
<property>
<name>ozone.om.kerberos.keytab.file</name>
<value>{{ keytabs_dir }}/om.keytab</value>
</property>
<property>
<name>ozone.om.http.auth.type</name>
<value>kerberos</value>
</property>
<property>
<name>ozone.om.http.auth.kerberos.principal</name>
<value>HTTP/_h...@example.com</value>
</property>
<property>
<name>ozone.om.http.auth.kerberos.keytab</name>
<value>{{ keytabs_dir }}/spnego.keytab</value>
</property>
<!--RECON CONFIGS -->
<property>
<name>ozone.recon.address</name>
<value>{{ ozone_recon_address }}</value>
</property>
<property>
<name>ozone.recon.db.dir</name>
<value>{{ ozone_dir }}/recon</value>
</property>
<property>
<name>ozone.recon.om.snapshot.task.interval.delay</name>
<value>1m</value>
</property>
<property>
<name>ozone.recon.om.snapshot.task.initial.delay</name>
<value>20s</value>
</property>
<!--RECON SECURITY CONFIGS -->
<property>
<name>ozone.recon.http.auth.type</name>
<value>kerberos</value>
</property>
<property>
<name>ozone.recon.kerberos.principal</name>
<value>recon/_h...@example.com</value>
</property>
<property>
<name>ozone.recon.kerberos.keytab.file</name>
<value>{{ keytabs_dir }}/recon.keytab</value>
</property>
<property>
<name>ozone.recon.http.auth.kerberos.principal</name>
<value>HTTP/_h...@example.com</value>
</property>
<property>
<name>ozone.recon.http.auth.kerberos.keytab</name>
<value>{{ keytabs_dir }}/spnego.keytab</value>
</property>
<!--S3G SECURITY CONFIGS -->
<property>
<name>ozone.s3g.http.auth.type</name>
<value>kerberos</value>
</property>
<property>
<name>ozone.s3g.kerberos.principal</name>
<value>s3g/_h...@example.com</value>
</property>
<property>
<name>ozone.s3g.kerberos.keytab.file</name>
<value>{{ keytabs_dir }}/s3g.keytab</value>
</property>
<property>
<name>ozone.s3g.http.auth.kerberos.principal</name>
<value>HTTP/_h...@example.com</value>
</property>
<property>
<name>ozone.s3g.http.auth.kerberos.keytab</name>
<value>{{ keytabs_dir }}/spnego.keytab</value>
</property>
<!--DATANODE CONFIGS -->
<property>
<name>hdds.datanode.dir</name>
<value>/disk1/dn,/disk2/dn,/disk3/dn</value>
</property>
<!--DATANODE SECURITY CONFIGS -->
<property>
<name>dfs.datanode.kerberos.principal</name>
<value>dn/_h...@example.com</value>
</property>
<property>
<name>dfs.datanode.keytab.file</name>
<value>{{ keytabs_dir }}/dn.keytab</value>
</property>
<property>
<name>hdds.datanode.http.kerberos.principal</name>
<value>HTTP/_h...@example.com</value>
</property>
<property>
<name>hdds.datanode.http.kerberos.keytab</name>
<value>{{ keytabs_dir }}/spnego.keytab</value>
</property>
<property>
<name>hdds.datanode.http.auth.type</name>
<value>kerberos</value>
</property>
<property>
<name>hdds.datanode.http.auth.kerberos.principal</name>
<value>HTTP/_h...@example.com</value>
</property>
<property>
<name>hdds.datanode.http.auth.kerberos.keytab</name>
<value>{{ keytabs_dir }}/spnego.keytab</value>
</property>
<property>
<name>hadoop.http.max.request.header.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.http.authentication.signature.secret.file</name>
<value>/etc/security/http_secret</value>
</property>
<!--SSL CONFIGURATION -->
<property>
<name>ozone.http.policy</name>
<value>HTTPS_ONLY</value>
</property>
<property>
<name>ozone.https.client.need-auth</name>
<value>false</value>
<description>
Whether SSL client certificate authentication is required (2-WAY SSL)
</description>
</property>
<property>
<name>ozone.https.client.keystore.resource</name>
<value>ssl-client.xml</value>
</property>
<property>
<name>ozone.https.server.keystore.resource</name>
<value>ssl-server.xml</value>
</property>
<property>
<name>ssl.server.keystore.keypassword</name>
<value>{{ keystore_password }}</value>
</property>
<property>
<name>ssl.server.keystore.location</name>
<value>{{ ssl_location }}/{{ keystore }}.jks</value>
</property>
<property>
<name>ssl.server.keystore.password</name>
<value>{{ keystore_password }}</value>
</property>
<property>
<name>ssl.server.truststore.location</name>
<value>{{ ssl_location }}/cacerts</value>
</property>
<property>
<name>ssl.server.truststore.password</name>
<value>{{ truststore_password }}</value>
</property>
<!--Tracing -->
<property>
<name>ozone.trace.enabled</name>
<value>true</value>
</property>
</configuration>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org
