[GitHub] [submarine] yuanzac commented on a change in pull request #149: SUBMARINE-335 Add a guide of yarn runtime with docker containers

[GitBox]

yuanzac commented on a change in pull request #149: SUBMARINE-335 Add a guide 
of yarn runtime with docker containers
URL: https://github.com/apache/submarine/pull/149#discussion_r365636688
 
 

 ##########
 File path: dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh
 ##########
 @@ -33,27 +33,37 @@ else
   JAVA_CMD="java"
 fi
 
-while getopts 'd:' OPT; do
+while getopts 'd:c' OPT; do
   case $OPT in
     d)
       DATA_URL="$OPTARG";;
+    c)
+      USE_DOCKER=1;;
   esac
 done
 shift $(($OPTIND - 1))
 
 if [[ -n "$DATA_URL" ]]; then
-  WORKER_CMD="myvenv.zip/venv/bin/python mnist_distributed.py --steps 2 
--data_dir /tmp/data --working_dir /tmp/mode --mnist_data_url ${DATA_URL}"
+  WORKER_CMD="venv/bin/python mnist_distributed.py --steps 2 --data_dir 
/tmp/data --working_dir /tmp/mode --mnist_data_url ${DATA_URL}"
 else
-  WORKER_CMD="myvenv.zip/venv/bin/python mnist_distributed.py --steps 2 
--data_dir /tmp/data --working_dir /tmp/mode"
-fi 
+  WORKER_CMD="venv/bin/python mnist_distributed.py --steps 2 --data_dir 
/tmp/data --working_dir /tmp/mode"
+fi
+
+if [[ -n "$USE_DOCKER" ]]; then
+  WORKER_CMD="/opt/$WORKER_CMD"
+  # tony-mnist-tf-1.13.1:0.0.1 is built from the 
Dockerfile.tony.tf.mnist.tf_1.13.1 under docs/helper/docker/tensorflow/mnist
+  DOCKER_CONF="--conf tony.docker.containers.image=tony-mnist-tf-1.13.1:0.0.1 
--conf tony.docker.enabled=true"
+else
+  WORKER_CMD="myvenv.zip/$WORKER_CMD"
+fi
 
 SUBMARINE_VERSION=0.3.0-SNAPSHOT
-HADOOP_VERSION=2.9
-SUBMARINE_PATH=/opt/submarine-current
-HADOOP_CONF_PATH=/usr/local/hadoop/etc/hadoop
-MNIST_PATH=/home/yarn/submarine
+SUBMARINE_HADOOP_VERSION=2.9
+SUBMARINE_PATH=/home/pi/apache/submarine/submarine-all/target
 
 Review comment:
   Mini-submarine is docker image which is created by the script of 
build_mini-submarine.sh. Users can tests submarine job with it. Mini-submarine 
has a one node hadoop cluster and submarine libs. The script of 
run_submarine_mnist_tony.sh is supposed to be executed in it for users' 
convenience. So SUBMARINE_PATH, HADOOP_CONF_PATH, MNIST_PATH are set according 
to the mini-submarine configuration.
   It's better for us not to change their values. Or we need changes the values 
both in run_submarine_mnist_tony.sh and build_mini-submarine.sh.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org