Elek, Marton created HADOOP-16063:
-------------------------------------
Summary: Docker based pseudo-cluster definitions and test scripts
for Hdfs/Yarn
Key: HADOOP-16063
URL: https://issues.apache.org/jira/browse/HADOOP-16063
Project: Hadoop Common
Issue Type: New Feature
Reporter: Elek, Marton
During the recent releases of Apache Hadoop Ozone we had multiple experiments
using docker/docker-compose to support the development of ozone.
As of now the hadoop-ozone distribution contains two directories in additional
the regular hadoop directories (bin, share/lib, etc
h3. compose
The ./compose directory of the distribution contains different type of
pseudo-cluster definitions. To start an ozone cluster is as easy as "cd
compose/ozone && docker-compose up-d"
The clusters also could be scaled up and down (docker-compose scale datanode=3)
There are multiple cluster definitions for different use cases (for example
ozone+s3 or hdfs+ozone).
The docker-compose files are based on apache/hadoop-runner image which is an
"empty" image. It doesnt' contain any hadoop distribution. Instead the current
hadoop is used (the ../.. is mapped as a volume at /opt/hadoop)
With this approach it's very easy to 1) start a cluster from the distribution
2) test any patch from the dev tree, as after any build a new cluster can be
started easily (with multiple nodes and datanodes)
h3. smoketest
We also started to use a simple robotframework based test suite. (see
./smoketest directory). It's a high level test definition very similar to the
smoketests which are executed manually by the contributors during a release
vote.
But it's a formal definition to start cluster from different docker-compose
definitions and execute simple shell scripts (and compare the output).
I believe that both approaches helped a lot during the development of ozone and
I propose to do the same improvements on the main hadoop distribution.
I propose to provide docker-compose based example cluster definitions for
yarn/hdfs and for different use cases (simple hdfs, router based federation,
etc.)
It can help to understand the different configuration and try out new features
with predefined config set.
Long term we can also add robottests to help the release votes (basic
wordcount/mr tests could be scripted)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
