Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/6377#discussion_r204308017 --- Diff: flink-end-to-end-tests/test-scripts/test_yarn_kerberos_docker.sh --- @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ +set -o pipefail + +source "$(dirname "$0")"/common.sh + +FLINK_TARBALL_DIR=$TEST_DATA_DIR +FLINK_TARBALL=flink.tar.gz +FLINK_DIRNAME=$(basename $FLINK_DIR) + +echo "Flink Tarball directory $FLINK_TARBALL_DIR" +echo "Flink tarball filename $FLINK_TARBALL" +echo "Flink distribution directory name $FLINK_DIRNAME" +echo "End-to-end directory $END_TO_END_DIR" +docker --version +docker-compose --version + +mkdir -p $FLINK_TARBALL_DIR +tar czf $FLINK_TARBALL_DIR/$FLINK_TARBALL -C $(dirname $FLINK_DIR) . + +echo "Building Hadoop Docker container" +until docker build -f $END_TO_END_DIR/test-scripts/docker-hadoop-secure-cluster/Dockerfile -t flink/docker-hadoop-secure-cluster:latest $END_TO_END_DIR/test-scripts/docker-hadoop-secure-cluster/; do + # with all the downloading and ubuntu updating a lot of flakiness can happen, make sure + # we don't immediately fail + echo "Something went wrong while building the Docker image, retrying ..." + sleep 2 +done + +echo "Starting Hadoop cluster" +docker-compose -f $END_TO_END_DIR/test-scripts/docker-hadoop-secure-cluster/docker-compose.yml up -d + +# make sure we stop our cluster at the end +function cluster_shutdown { + # don't call ourselves again for another signal interruption + trap "exit -1" INT + # don't call ourselves again for normal exit + trap "" EXIT + + docker-compose -f $END_TO_END_DIR/test-scripts/docker-hadoop-secure-cluster/docker-compose.yml down + rm $FLINK_TARBALL_DIR/$FLINK_TARBALL +} +trap cluster_shutdown INT +trap cluster_shutdown EXIT + +until docker cp $FLINK_TARBALL_DIR/$FLINK_TARBALL master:/home/hadoop-user/; do --- End diff -- I did it like this so that rebuilding Flink does not require building the docker image. I know I could do it as one of the last steps but with repeatedly running the test locally I think it's still easier this way. WDYT?
---