Remove Yarn from minicluster by default. (2nd try) Remove Yarn from minicluster by default.
Turns out that we start Yarn as part of the minicluster, but we never use it. (HiveServer2 is configured to run MR jobs "locally" in process.) Likely, this Yarn integration is a vestige of Yarn/Llama integration. We can save memory by not starting it by default. There are some less-common tooks like tests/comparison/cluster.py which use Yarn (and Hadoop Streaming). In deference to those tools, I've left a mechanism to start Yarn rather than excising it altogether. After running buildall the regular way, add Yarn to the cluster by running: testdata/cluster/admin -y start_cluster I tested by running core tests. I did not test the kerberized minicluster. [Due to a git mishap, a version of this was previously checked in and reverted.] Change-Id: I97053a44bbe32048e6c35cc28680d1c7696af13f Reviewed-on: http://gerrit.cloudera.org:8080/9970 Reviewed-by: Michael Brown <mi...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/impala/repo Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/c2e4c43b Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/c2e4c43b Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/c2e4c43b Branch: refs/heads/2.x Commit: c2e4c43b6e3265cd9e92f6afaa4a10a307919ca7 Parents: f09f68a Author: Philip Zeyliger <phi...@cloudera.com> Authored: Tue Apr 10 09:21:20 2018 -0700 Committer: Impala Public Jenkins <impala-public-jenk...@gerrit.cloudera.org> Committed: Wed Apr 11 22:56:01 2018 +0000 ---------------------------------------------------------------------- testdata/cluster/admin | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/impala/blob/c2e4c43b/testdata/cluster/admin ---------------------------------------------------------------------- diff --git a/testdata/cluster/admin b/testdata/cluster/admin index cb33e21..74b5a9c 100755 --- a/testdata/cluster/admin +++ b/testdata/cluster/admin @@ -18,7 +18,7 @@ # specific language governing permissions and limitations # under the License. -# This will create/control/destroy a local hdfs+yarn cluster. +# This will create/control/destroy a local hdfs/yarn/kms/kudu cluster. # # All roles run on 127.0.0.1, just like the standard mini cluster included with hadoop. # The difference is with this cluster, each role runs in its own process and has its own @@ -31,12 +31,14 @@ set -euo pipefail trap 'echo Error in $0 at line $LINENO: $(awk "NR == $LINENO" $0)' ERR : ${IMPALA_KERBERIZE=} +: ${INCLUDE_YARN=} -while getopts vk OPT; do +while getopts vky OPT; do case $OPT in v) set -x;; k) export IMPALA_KERBERIZE=1;; - ?) echo "Usage: $0 [-v (verbose) -k (kerberize)] ACTION (see source...)"; exit 1;; + y) export INCLUDE_YARN=1;; + ?) echo "Usage: $0 [-v (verbose) -k (kerberize) -y (yarn)] ACTION (see source...)"; exit 1;; esac done shift $(($OPTIND-1)) @@ -54,7 +56,10 @@ export KILL_CLUSTER_MARKER=IBelongToTheMiniCluster if [[ "$TARGET_FILESYSTEM" == "hdfs" ]]; then # The check above indicates that the regular mini-cluster is in use. - SUPPORTED_SERVICES=(hdfs yarn kms) + SUPPORTED_SERVICES=(hdfs kms) + if [ -n "${INCLUDE_YARN}" ]; then + SUPPORTED_SERVICES+=(yarn) + fi else # Either a remote distributed file system or a local non-distributed file system is # in use. Currently the only service that is expected to work is Kudu, though in theory
