Hello ozone community I'm getting the following error when trying to run any action in SCM with kerberos enabled:
Caused by: java.io.IOException: Couldn't setup connection for myu...@example.com to scm-server.example.com/10.XXX.YYY.ZZZ:9961 ... Caused by: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed when I check the SCM logs and the service starts properly [Listener at 0.0.0.0/9860] INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Using keytab /opt/ozone/keytabs/spnego.keytab, for principal HTTP/ scm-server.example....@example.com [Listener at 0.0.0.0/9860] INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@24a1c17f {logs,/logs,file:///var/log/ozone/,AVAILABLE} [Listener at 0.0.0.0/9860] INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@532721fd {static,/static,jar:file:/opt/ozone/share/ozone/lib/hadoop-hdds-server-scm-1.0.0.jar!/webapps/static,AVAILABLE} [Listener at 0.0.0.0/9860] INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Using keytab /opt/ozone/keytabs/spnego.keytab, for principal HTTP/ scm-server.example....@example.com [Listener at 0.0.0.0/9860] INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@62ddd21b {scm,/,file:///tmp/jetty-0_0_0_0-9877-hadoop-hdds-server-scm-1_0_0_jar-_-any-4975172152496600541.dir/webapp/,AVAILABLE}{jar:file:/opt/ozone/share/ozone/lib/hadoop-hdds-server-scm-1.0.0.jar!/webapps/scm} [Listener at 0.0.0.0/9860] INFO org.eclipse.jetty.util.ssl.SslContextFactory: x509=X509@783efb48( scm-server.example.com,h=[scm-server.example.com, scm-server],w=[]) for Server@c446b14 [provider=null,keyStore=file:///opt/ozone/jks/scm-server.jks,trustStore=file:///opt/ozone/jks/cacerts] [Listener at 0.0.0.0/9860] INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@13006998{SSL,[ssl, http/1.1]}{0.0.0.0:9877} [Listener at 0.0.0.0/9860] INFO org.eclipse.jetty.server.Server: Started @2301ms [Listener at 0.0.0.0/9860] INFO org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink prometheus started [Listener at 0.0.0.0/9860] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Registered sink prometheus [Listener at 0.0.0.0/9860] INFO org.apache.hadoop.hdds.server.http.BaseHttpServer: HTTPS server of scm listening at https://0.0.0.0:9877 and I only see this message: [Socket Reader #1 for port 9961] WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.XXX.YYY.ZZZ:52306:null (GSS initiate failed) with true cause: (GSS initiate failed) I used this template to build my configs: https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config my auth_to_local currently is like this: <property> <name>hadoop.security.auth_to_local</name> <value>RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*// RULE:[2:$1@$0](.*)s/.*/scm/ DEFAULT</value> </property> I already tested the rules and they seems to be OK: $ hadoop kerbname myu...@example.com Name: myu...@example.com to myuser $ hadoop kerbname scm/scm-server.example....@example.com Name: scm/scm-server.example....@example.com to scm Also tried the following auth_to_local, but the result is the same: <property> <name>hadoop.security.auth_to_local</name> <value>RULE:[1:$1@$0](s...@example.com)s/.*/scm/ RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*// RULE:[2:$1@$0](.*)s/.*/root/ DEFAULT</value> </property> Have you had this issue before? Could you provide any suggestion for this? I'm also attaching as reference my templates for the core-site, ozone-site and hdfs-site. I'm using ozone 1.0 downloaded from the site build = https://github.com/apache/hadoop-ozone.git/28d372ca903b4741131bace09e0339e9161257bb ; compiled by 'sammi' on 2020-08-25T13:04Z java = 1.8.0_201 But it is the same issue when I build it from source. Thank you in advance. Diego
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <configuration> <property> <name>dfs.data.transfer.protection</name> <value>authentication,privacy</value> </property> <property> <name>dfs.datanode.address</name> <value>0.0.0.0:1019</value> </property> <property> <name>dfs.datanode.http.address</name> <value>0.0.0.0:1012</value> </property> <property> <name>dfs.datanode.kerberos.principal</name> <value>dn/_h...@example.com</value> </property> <property> <name>dfs.datanode.keytab.file</name> <value>{{ keytabs_dir }}/dn.keytab</value> </property> <property> <name>dfs.web.authentication.kerberos.principal</name> <value>HTTP/_h...@example.com</value> </property> <property> <name>dfs.web.authentication.kerberos.keytab</name> <value>{{ keytabs_dir }}/spnego.keytab</value> </property> <property> <name>rpc.metrics.quantile.enable</name> <value>true</value> </property> <property> <name>rpc.metrics.percentiles.intervals</name> <value>60,300</value> </property> </configuration>
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <configuration> <property> <name>fs.ofs.impl</name> <value>org.apache.hadoop.fs.ozone.RootedOzoneFileSystem</value> </property> <property> <name>fs.o3fs.impl</name> <value>org.apache.hadoop.fs.ozone.OzoneFileSystem</value> </property> <property> <name>fs.defaultFS</name> <value>ofs://{{ ozone_om }}</value> </property> <property> <name>dfs.data.transfer.protection</name> <value>authentication</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hadoop.http.authentication.simple.anonymous.allowed</name> <value>false</value> </property> <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.http.authentication.signature.secret.file</name> <value>/etc/security/http_secret</value> </property> <property> <name>hadoop.http.authentication.type</name> <value>kerberos</value> </property> <property> <name>hadoop.http.authentication.kerberos.principal</name> <value>HTTP/_h...@example.com</value> </property> <property> <name>hadoop.http.authentication.kerberos.keytab</name> <value>{{ keytabs_dir }}/spnego.keytab</value> </property> <property> <name>hadoop.proxyuser.HTTP.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.HTTP.hosts</name> <value>*</value> </property> <property> <name>hadoop.ssl.require.client.cert</name> <value>false</value> </property> <property> <name>hadoop.ssl.hostname.verifier</name> <value>DEFAULT</value> </property> <property> <name>hadoop.ssl.keystores.factory.class</name> <value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value> </property> <property> <name>hadoop.ssl.server.conf</name> <value>ssl-server.xml</value> </property> <property> <name>hadoop.ssl.client.conf</name> <value>ssl-client.xml</value> </property> <property> <name>hadoop.security.key.provider.path</name> <value>kms://https@{{ kms }}/kms</value> </property> <property> <name>hadoop.security.auth_to_local</name> <value>RULE:[1:$1@$0](s...@example.com)s/.*/scm/ RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*// RULE:[2:$1@$0](.*)s/.*/root/ DEFAULT</value> </property> </configuration>
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <configuration> <property> <name>ozone.replication</name> <value>3</value> </property> <property> <name>ozone.metadata.dirs</name> <value>{{ ozone_dir }}/meta</value> </property> <property> <name>ozone.handler.type</name> <value>distributed</value> </property> <property> <name>hdds.block.token.enabled</name> <value>true</value> </property> <!--OZONE SECURITY --> <property> <name>ozone.security.enabled</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>ozone.security.http.kerberos.enabled</name> <value>true</value> </property> <property> <name>ozone.http.filter.initializers</name> <value>org.apache.hadoop.security.AuthenticationFilterInitializer</value> </property> <property> <name>ozone.acl.enabled</name> <value>true</value> </property> <property> <name>ozone.acl.authorizer.class</name> <value>org.apache.hadoop.ozone.security.acl.OzoneNativeAuthorizer</value> </property> <property> <name>ozone.administrators</name> <value>"{{ ozone_admins }}"</value> </property> <!--SCM CONFIGS --> <property> <name>ozone.scm.names</name> <value>{{ ozone_scm_names }}</value> </property> <property> <name>ozone.scm.client.address</name> <value>{{ ozone_scm_client_address }}</value> </property> <property> <name>ozone.scm.block.client.address</name> <value>{{ ozone_scm_block_client_address }}</value> </property> <property> <name>ozone.scm.pipeline.creation.interval</name> <value>30s</value> </property> <property> <name>ozone.scm.pipeline.owner.container.count</name> <value>1</value> </property> <property> <name>ozone.scm.container.size</name> <value>{{ ozone_scm_container_size }}</value> </property> <property> <name>ozone.scm.datanode.id.dir</name> <value>{{ ozone_dir }}/</value> </property> <property> <name>ozone.scm.datanode.id</name> <value>{{ ozone_dir }}/datanode.id</value> </property> <!--SCM SECURITY CONFIGS --> <property> <name>hdds.scm.kerberos.principal</name> <value>scm/_h...@example.com</value> </property> <property> <name>hdds.scm.kerberos.keytab.file</name> <value>{{ keytabs_dir }}/scm.keytab</value> </property> <property> <name>hdds.scm.http.auth.type</name> <value>kerberos</value> </property> <property> <name>hdds.scm.http.auth.kerberos.principal</name> <value>HTTP/_h...@example.com</value> </property> <property> <name>hdds.scm.http.auth.kerberos.keytab</name> <value>{{ keytabs_dir }}/spnego.keytab</value> </property> <property> <name>ozone.scm.http.auth.type</name> <value>kerberos</value> </property> <property> <name>ozone.om.address</name> <value>{{ ozone_om_address }}</value> </property> <property> <name>ozone.om.http-address</name> <value>{{ ozone_om_http_address }}</value> </property> <!--OM SECURITY CONFIGS --> <property> <name>ozone.om.volume.listall.allowed</name> <value>false</value> </property> <property> <name>ozone.om.kerberos.principal</name> <value>om/_h...@example.com</value> </property> <property> <name>ozone.om.kerberos.keytab.file</name> <value>{{ keytabs_dir }}/om.keytab</value> </property> <property> <name>ozone.om.http.auth.type</name> <value>kerberos</value> </property> <property> <name>ozone.om.http.auth.kerberos.principal</name> <value>HTTP/_h...@example.com</value> </property> <property> <name>ozone.om.http.auth.kerberos.keytab</name> <value>{{ keytabs_dir }}/spnego.keytab</value> </property> <!--RECON CONFIGS --> <property> <name>ozone.recon.address</name> <value>{{ ozone_recon_address }}</value> </property> <property> <name>ozone.recon.db.dir</name> <value>{{ ozone_dir }}/recon</value> </property> <property> <name>ozone.recon.om.snapshot.task.interval.delay</name> <value>1m</value> </property> <property> <name>ozone.recon.om.snapshot.task.initial.delay</name> <value>20s</value> </property> <!--RECON SECURITY CONFIGS --> <property> <name>ozone.recon.http.auth.type</name> <value>kerberos</value> </property> <property> <name>ozone.recon.kerberos.principal</name> <value>recon/_h...@example.com</value> </property> <property> <name>ozone.recon.kerberos.keytab.file</name> <value>{{ keytabs_dir }}/recon.keytab</value> </property> <property> <name>ozone.recon.http.auth.kerberos.principal</name> <value>HTTP/_h...@example.com</value> </property> <property> <name>ozone.recon.http.auth.kerberos.keytab</name> <value>{{ keytabs_dir }}/spnego.keytab</value> </property> <!--S3G SECURITY CONFIGS --> <property> <name>ozone.s3g.http.auth.type</name> <value>kerberos</value> </property> <property> <name>ozone.s3g.kerberos.principal</name> <value>s3g/_h...@example.com</value> </property> <property> <name>ozone.s3g.kerberos.keytab.file</name> <value>{{ keytabs_dir }}/s3g.keytab</value> </property> <property> <name>ozone.s3g.http.auth.kerberos.principal</name> <value>HTTP/_h...@example.com</value> </property> <property> <name>ozone.s3g.http.auth.kerberos.keytab</name> <value>{{ keytabs_dir }}/spnego.keytab</value> </property> <!--DATANODE CONFIGS --> <property> <name>hdds.datanode.dir</name> <value>/disk1/dn,/disk2/dn,/disk3/dn</value> </property> <!--DATANODE SECURITY CONFIGS --> <property> <name>dfs.datanode.kerberos.principal</name> <value>dn/_h...@example.com</value> </property> <property> <name>dfs.datanode.keytab.file</name> <value>{{ keytabs_dir }}/dn.keytab</value> </property> <property> <name>hdds.datanode.http.kerberos.principal</name> <value>HTTP/_h...@example.com</value> </property> <property> <name>hdds.datanode.http.kerberos.keytab</name> <value>{{ keytabs_dir }}/spnego.keytab</value> </property> <property> <name>hdds.datanode.http.auth.type</name> <value>kerberos</value> </property> <property> <name>hdds.datanode.http.auth.kerberos.principal</name> <value>HTTP/_h...@example.com</value> </property> <property> <name>hdds.datanode.http.auth.kerberos.keytab</name> <value>{{ keytabs_dir }}/spnego.keytab</value> </property> <property> <name>hadoop.http.max.request.header.size</name> <value>131072</value> </property> <property> <name>hadoop.http.authentication.signature.secret.file</name> <value>/etc/security/http_secret</value> </property> <!--SSL CONFIGURATION --> <property> <name>ozone.http.policy</name> <value>HTTPS_ONLY</value> </property> <property> <name>ozone.https.client.need-auth</name> <value>false</value> <description> Whether SSL client certificate authentication is required (2-WAY SSL) </description> </property> <property> <name>ozone.https.client.keystore.resource</name> <value>ssl-client.xml</value> </property> <property> <name>ozone.https.server.keystore.resource</name> <value>ssl-server.xml</value> </property> <property> <name>ssl.server.keystore.keypassword</name> <value>{{ keystore_password }}</value> </property> <property> <name>ssl.server.keystore.location</name> <value>{{ ssl_location }}/{{ keystore }}.jks</value> </property> <property> <name>ssl.server.keystore.password</name> <value>{{ keystore_password }}</value> </property> <property> <name>ssl.server.truststore.location</name> <value>{{ ssl_location }}/cacerts</value> </property> <property> <name>ssl.server.truststore.password</name> <value>{{ truststore_password }}</value> </property> <!--Tracing --> <property> <name>ozone.trace.enabled</name> <value>true</value> </property> </configuration>
--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org