[ https://issues.apache.org/jira/browse/FLINK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550701#comment-16550701 ]
ASF GitHub Bot commented on FLINK-8981: --------------------------------------- Github user dawidwys commented on a diff in the pull request: https://github.com/apache/flink/pull/6377#discussion_r204020995 --- Diff: flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster/bootstrap.sh --- @@ -0,0 +1,121 @@ +#!/bin/bash +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +: ${HADOOP_PREFIX:=/usr/local/hadoop} + +$HADOOP_PREFIX/etc/hadoop/hadoop-env.sh + +rm /tmp/*.pid + +# installing libraries if any - (resource urls added comma separated to the ACP system variable) +cd $HADOOP_PREFIX/share/hadoop/common ; for cp in ${ACP//,/ }; do echo == $cp; curl -LO $cp ; done; cd - + +# kerberos client +sed -i "s/EXAMPLE.COM/${KRB_REALM}/g" /etc/krb5.conf +sed -i "s/example.com/${DOMAIN_REALM}/g" /etc/krb5.conf + +# update config files +sed -i "s/HOSTNAME/$(hostname -f)/g" $HADOOP_PREFIX/etc/hadoop/core-site.xml +sed -i "s/EXAMPLE.COM/${KRB_REALM}/g" $HADOOP_PREFIX/etc/hadoop/core-site.xml +sed -i "s#/etc/security/keytabs#${KEYTAB_DIR}#g" $HADOOP_PREFIX/etc/hadoop/core-site.xml + +sed -i "s/EXAMPLE.COM/${KRB_REALM}/g" $HADOOP_PREFIX/etc/hadoop/hdfs-site.xml +sed -i "s/HOSTNAME/$(hostname -f)/g" $HADOOP_PREFIX/etc/hadoop/hdfs-site.xml +sed -i "s#/etc/security/keytabs#${KEYTAB_DIR}#g" $HADOOP_PREFIX/etc/hadoop/hdfs-site.xml + +sed -i "s/EXAMPLE.COM/${KRB_REALM}/g" $HADOOP_PREFIX/etc/hadoop/yarn-site.xml +sed -i "s/HOSTNAME/$(hostname -f)/g" $HADOOP_PREFIX/etc/hadoop/yarn-site.xml +sed -i "s#/etc/security/keytabs#${KEYTAB_DIR}#g" $HADOOP_PREFIX/etc/hadoop/yarn-site.xml + +sed -i "s/EXAMPLE.COM/${KRB_REALM}/g" $HADOOP_PREFIX/etc/hadoop/mapred-site.xml +sed -i "s/HOSTNAME/$(hostname -f)/g" $HADOOP_PREFIX/etc/hadoop/mapred-site.xml +sed -i "s#/etc/security/keytabs#${KEYTAB_DIR}#g" $HADOOP_PREFIX/etc/hadoop/mapred-site.xml + +sed -i "s#/usr/local/hadoop/bin/container-executor#${NM_CONTAINER_EXECUTOR_PATH}#g" $HADOOP_PREFIX/etc/hadoop/yarn-site.xml + +# create namenode kerberos principal and keytab +kadmin -p ${KERBEROS_ADMIN} -w ${KERBEROS_ADMIN_PASSWORD} -q "addprinc -randkey hdfs/$(hostname -f)@${KRB_REALM}" +kadmin -p ${KERBEROS_ADMIN} -w ${KERBEROS_ADMIN_PASSWORD} -q "addprinc -randkey mapred/$(hostname -f)@${KRB_REALM}" +kadmin -p ${KERBEROS_ADMIN} -w ${KERBEROS_ADMIN_PASSWORD} -q "addprinc -randkey yarn/$(hostname -f)@${KRB_REALM}" +kadmin -p ${KERBEROS_ADMIN} -w ${KERBEROS_ADMIN_PASSWORD} -q "addprinc -randkey HTTP/$(hostname -f)@${KRB_REALM}" + +kadmin -p ${KERBEROS_ADMIN} -w ${KERBEROS_ADMIN_PASSWORD} -q "xst -k hdfs.keytab hdfs/$(hostname -f) HTTP/$(hostname -f)" +kadmin -p ${KERBEROS_ADMIN} -w ${KERBEROS_ADMIN_PASSWORD} -q "xst -k mapred.keytab mapred/$(hostname -f) HTTP/$(hostname -f)" +kadmin -p ${KERBEROS_ADMIN} -w ${KERBEROS_ADMIN_PASSWORD} -q "xst -k yarn.keytab yarn/$(hostname -f) HTTP/$(hostname -f)" + +mkdir -p ${KEYTAB_DIR} +mv hdfs.keytab ${KEYTAB_DIR} +mv mapred.keytab ${KEYTAB_DIR} +mv yarn.keytab ${KEYTAB_DIR} +chmod 400 ${KEYTAB_DIR}/hdfs.keytab +chmod 400 ${KEYTAB_DIR}/mapred.keytab +chmod 400 ${KEYTAB_DIR}/yarn.keytab +chown hdfs:hadoop ${KEYTAB_DIR}/hdfs.keytab +chown mapred:hadoop ${KEYTAB_DIR}/mapred.keytab +chown yarn:hadoop ${KEYTAB_DIR}/yarn.keytab + +service ssh start --- End diff -- Can we just make ssh start automatically in Dockerfile? > Add end-to-end test for running on YARN with Kerberos > ----------------------------------------------------- > > Key: FLINK-8981 > URL: https://issues.apache.org/jira/browse/FLINK-8981 > Project: Flink > Issue Type: Sub-task > Components: Security, Tests > Affects Versions: 1.5.0 > Reporter: Till Rohrmann > Assignee: Aljoscha Krettek > Priority: Blocker > Labels: pull-request-available > Fix For: 1.6.0 > > > We should add an end-to-end test which verifies Flink's integration with > Kerberos security. In order to do this, we should start a Kerberos secured > Hadoop, ZooKeeper and Kafka cluster. Then we should start a Flink cluster > with HA enabled and run a job which reads from and writes to Kafka. We could > use a simple pipe job for that purpose which has some state for checkpointing > to HDFS. > See [security docs| > https://ci.apache.org/projects/flink/flink-docs-master/ops/security-kerberos.html] > for how more information about Flink's Kerberos integration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)