Elek, Marton created HADOOP-16063: ------------------------------------- Summary: Docker based pseudo-cluster definitions and test scripts for Hdfs/Yarn Key: HADOOP-16063 URL: https://issues.apache.org/jira/browse/HADOOP-16063 Project: Hadoop Common Issue Type: New Feature Reporter: Elek, Marton
During the recent releases of Apache Hadoop Ozone we had multiple experiments using docker/docker-compose to support the development of ozone. As of now the hadoop-ozone distribution contains two directories in additional the regular hadoop directories (bin, share/lib, etc h3. compose The ./compose directory of the distribution contains different type of pseudo-cluster definitions. To start an ozone cluster is as easy as "cd compose/ozone && docker-compose up-d" The clusters also could be scaled up and down (docker-compose scale datanode=3) There are multiple cluster definitions for different use cases (for example ozone+s3 or hdfs+ozone). The docker-compose files are based on apache/hadoop-runner image which is an "empty" image. It doesnt' contain any hadoop distribution. Instead the current hadoop is used (the ../.. is mapped as a volume at /opt/hadoop) With this approach it's very easy to 1) start a cluster from the distribution 2) test any patch from the dev tree, as after any build a new cluster can be started easily (with multiple nodes and datanodes) h3. smoketest We also started to use a simple robotframework based test suite. (see ./smoketest directory). It's a high level test definition very similar to the smoketests which are executed manually by the contributors during a release vote. But it's a formal definition to start cluster from different docker-compose definitions and execute simple shell scripts (and compare the output). I believe that both approaches helped a lot during the development of ozone and I propose to do the same improvements on the main hadoop distribution. I propose to provide docker-compose based example cluster definitions for yarn/hdfs and for different use cases (simple hdfs, router based federation, etc.) It can help to understand the different configuration and try out new features with predefined config set. Long term we can also add robottests to help the release votes (basic wordcount/mr tests could be scripted) -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org