[jira] [Commented] (SUBMARINE-45) Can't specify queue by using the parameter --queue
[ https://issues.apache.org/jira/browse/SUBMARINE-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907839#comment-16907839 ] Hudson commented on SUBMARINE-45: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17128 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17128/]) SUBMARINE-45. Can't specify queue by using the parameter --queue. (ztang: rev 85a00fd2bbb8e8fee5691eff78477cdcb746a926) * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/AbstractServiceSpec.java * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/yarnservice/ParamBuilderForTest.java * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/yarnservice/TestYarnServiceRunJobCliCommons.java * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/yarnservice/TestYarnServiceRunJobCli.java > Can't specify queue by using the parameter --queue > --- > > Key: SUBMARINE-45 > URL: https://issues.apache.org/jira/browse/SUBMARINE-45 > Project: Hadoop Submarine > Issue Type: Bug >Reporter: Zac Zhou >Assignee: Ayush Saxena >Priority: Major > Fix For: 0.3.0 > > Attachments: SUBMARINE-45-03.patch, SUBMARINE-45.001.patch, > SUBMARINE-45.002.patch > > > We have the parameter "–queue" for users to specify the queue. But the > parameter "–queue" doesn't take effect -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (SUBMARINE-107) Reduce the scope of mockito-core in submarine to test
[ https://issues.apache.org/jira/browse/SUBMARINE-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907475#comment-16907475 ] Hudson commented on SUBMARINE-107: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17125 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17125/]) SUBMARINE-107. Reduce the scope of mockito-core in submarine to test. (ayushsaxena: rev c7204417d7f75fd9a7eb07bdc19d303ebdaade80) * (edit) hadoop-submarine/pom.xml > Reduce the scope of mockito-core in submarine to test > - > > Key: SUBMARINE-107 > URL: https://issues.apache.org/jira/browse/SUBMARINE-107 > Project: Hadoop Submarine > Issue Type: Improvement >Reporter: kevin su >Assignee: kevin su >Priority: Minor > Fix For: 0.3.0 > > Attachments: SUBMARINE-107.001.patch > > > should add test scope to mockito-core in submarine pom.xml -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (SUBMARINE-57) Add more elaborate message if submarine command is not recognized
[
https://issues.apache.org/jira/browse/SUBMARINE-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903799#comment-16903799
]
Hudson commented on SUBMARINE-57:
-
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17073 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17073/])
SUBMARINE-57. Add more elaborate message if submarine command is not (snemeth:
rev e5f4cd0fdae7e689789dd74bfbcfa6c52895f037)
* (edit)
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/Cli.java
> Add more elaborate message if submarine command is not recognized
> -
>
> Key: SUBMARINE-57
> URL: https://issues.apache.org/jira/browse/SUBMARINE-57
> Project: Hadoop Submarine
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Adam Antal
>Priority: Major
> Fix For: 0.2.1, 0.3.0
>
> Attachments: SUBMARINE-57.001.patch, SUBMARINE-57.001.patch
>
>
> In {{org.apache.hadoop.yarn.submarine.client.cli.Cli#main}}, we have this
> error handling:
> {code:java}
> if (args[0].equals("job")) {
> String subCmd = args[1];
> if (subCmd.equals(CliConstants.RUN)) {
> new RunJobCli(clientContext).run(moduleArgs);
> } else if (subCmd.equals(CliConstants.SHOW)) {
> new ShowJobCli(clientContext).run(moduleArgs);
> } else {
> printHelp();
> throw new IllegalArgumentException("Unknown option for job");
> }
> } else {
> printHelp();
> throw new IllegalArgumentException("Bad parameters ");
> }
> {code}
> "Bad parameters " need to be replaced with someting making more sense.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (SUBMARINE-72) Kill and destroy the job through the submarine client
[ https://issues.apache.org/jira/browse/SUBMARINE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902927#comment-16902927 ] Hudson commented on SUBMARINE-72: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17062 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17062/]) SUBMARINE-72. Kill and destroy the job through the submarine client (weichiu: rev 00b5a27864e0e5af7f55503a1d85dfa347f9ec0c) * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/ParametersHolder.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/Cli.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/KillJobParameters.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/KillJobCli.java * (add) hadoop-submarine/hadoop-submarine-core/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/TestKillJobCliParsing.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/CliConstants.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/Command.java > Kill and destroy the job through the submarine client > - > > Key: SUBMARINE-72 > URL: https://issues.apache.org/jira/browse/SUBMARINE-72 > Project: Hadoop Submarine > Issue Type: Improvement >Reporter: Xun Liu >Assignee: kevin su >Priority: Major > Labels: pull-request-available > Fix For: 0.3.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > The submarine client needs to be able to support the kill and destroy job. > Instead of executing it through yarn. > This way, the third-party system does not need to install the runtime > environment of yarn. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (SUBMARINE-62) PS_LAUNCH_CMD CLI description is wrong in RunJobCli
[ https://issues.apache.org/jira/browse/SUBMARINE-62?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885022#comment-16885022 ] Hudson commented on SUBMARINE-62: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16914 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16914/]) SUBMARINE-62. PS_LAUNCH_CMD CLI description is wrong in RunJobCli. (snemeth: rev be784de2d4c8d7ae2724cf348925a0fbdbe0c503) * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/runjob/RunJobCli.java > PS_LAUNCH_CMD CLI description is wrong in RunJobCli > --- > > Key: SUBMARINE-62 > URL: https://issues.apache.org/jira/browse/SUBMARINE-62 > Project: Hadoop Submarine > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Adam Antal >Priority: Major > Fix For: 0.3.0 > > Attachments: SUBMARINE-62.001.patch, SUBMARINE-62.002.patch, > SUBMARINE-62.002.patch > > > See: > [https://github.com/apache/hadoop/blob/trunk/hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/RunJobCli.java#L118-L120] > Currently, the description is: > "Commandline of worker, arguments will be directly used to launch the PS" > Shouldn't the description start with "Commandline of PS..."? > The rest of it can remain intact. > [~tangzhankun]: What's your opinion? -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (SUBMARINE-94) Support different docker images for job types in Tony Runtime
[ https://issues.apache.org/jira/browse/SUBMARINE-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878997#comment-16878997 ] Hudson commented on SUBMARINE-94: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16864 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16864/]) SUBMARINE-94. Support different docker images for (ztang: rev 928edb2c47e2d776daa35833897fac398cdeecd7) * (edit) hadoop-submarine/hadoop-submarine-tony-runtime/pom.xml > Support different docker images for job types in Tony Runtime > -- > > Key: SUBMARINE-94 > URL: https://issues.apache.org/jira/browse/SUBMARINE-94 > Project: Hadoop Submarine > Issue Type: Bug >Reporter: Keqiu Hu >Priority: Critical > Labels: submarine > Fix For: 0.2.1, 0.3.0 > > Attachments: SUBMARINE-94.001.patch > > > Bump TonY version to support different docker images for different job types. > Details in [https://github.com/linkedin/TonY/issues/311] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SUBMARINE-88) rat.sh regex pattern not working issue while using lower version of grep in ubuntu16.04
[ https://issues.apache.org/jira/browse/SUBMARINE-88?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856274#comment-16856274 ] Hudson commented on SUBMARINE-88: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16674 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16674/]) SUBMARINE-88. rat.sh regex pattern not working issue while using lower (sunilg: rev 96d818a2f7a0cd62f252234ba1a27c8c70bfed1f) * (edit) hadoop-submarine/dev-support/checks/rat.sh > rat.sh regex pattern not working issue while using lower version of grep in > ubuntu16.04 > --- > > Key: SUBMARINE-88 > URL: https://issues.apache.org/jira/browse/SUBMARINE-88 > Project: Hadoop Submarine > Issue Type: Sub-task >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Fix For: 0.2.0 > > Attachments: SUBMARINE-88.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SUBMARINE-86) Bump TonY version to pick up a docker related fix & Update documentation
[ https://issues.apache.org/jira/browse/SUBMARINE-86?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851672#comment-16851672 ] Hudson commented on SUBMARINE-86: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16628 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16628/]) SUBMARINE-86. Bump TonY version to pick up a docker related fix & Update (ztang: rev 2b303e9d5f2108fee621dd0c6e5582a861c401e4) * (edit) hadoop-submarine/hadoop-submarine-tony-runtime/pom.xml * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/TonYRuntimeGuide.md > Bump TonY version to pick up a docker related fix & Update documentation > > > Key: SUBMARINE-86 > URL: https://issues.apache.org/jira/browse/SUBMARINE-86 > Project: Hadoop Submarine > Issue Type: Improvement >Reporter: Keqiu Hu >Priority: Blocker > Fix For: 0.2.0 > > Attachments: SUBMARINE-86.001.patch, SUBMARINE-86.002.patch, > SUBMARINE-86.003.patch, SUBMARINE-86.004.patch > > > Bump TonY to 0.3.9 and update code to fix the following issues: > * Docker env was not properly set. > * Support renaming files after uploading w/ Localization flag. > * Fix an issue that security token defaults to use hostname (changed to use > ip address) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SUBMARINE-56) Update documentation to describe single-node PyTorch integration
[ https://issues.apache.org/jira/browse/SUBMARINE-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840977#comment-16840977 ] Hudson commented on SUBMARINE-56: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16557 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16557/]) SUBMARINE-56. Update documentation to describe single-node PyTorch (sunilg: rev de01422c2e13747686028e5c0e3a06306e9e5f08) * (add) hadoop-submarine/hadoop-submarine-core/src/site/markdown/RunningSingleNodeCifar10PTJobs.md * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/DeveloperGuide.md * (add) hadoop-submarine/hadoop-submarine-core/src/site/markdown/WriteDockerfilePT.md * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/InstallationGuideChineseVersion.md * (add) hadoop-submarine/hadoop-submarine-core/src/site/markdown/WriteDockerfileTF.md * (delete) hadoop-submarine/hadoop-submarine-core/src/site/markdown/WriteDockerfile.md * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/QuickStart.md * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/Index.md * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/InstallationGuide.md * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/Examples.md * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/RunningDistributedCifar10TFJobs.md > Update documentation to describe single-node PyTorch integration > > > Key: SUBMARINE-56 > URL: https://issues.apache.org/jira/browse/SUBMARINE-56 > Project: Hadoop Submarine > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Fix For: 0.2.0 > > Attachments: SUBMARINE-56.001.patch, SUBMARINE-56.002.patch, > SUBMARINE-56.003.patch, SUBMARINE-56.004.patch > > > We should include trying out and experimenting with PyTorch on a real cluster > to document it properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SUBMARINE-52) Generate Service spec + launch script for single-node PyTorch learning job
[ https://issues.apache.org/jira/browse/SUBMARINE-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837408#comment-16837408 ] Hudson commented on SUBMARINE-52: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16538 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16538/]) SUBMARINE-52. [SUBMARINE-14] Generate Service spec + launch script for (ztang: rev 36267b6f7c89c2ade05d97606cc5ac8685d22628) * (delete) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/RunJobParameters.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Spec.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/pytorch/with-cifar10-models/ubuntu-16.04/Dockerfile.gpu.pytorch_latest * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/tensorflow/base/ubuntu-16.04/Dockerfile.cpu.tf_1.13.1 * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/pytorch/command/package-info.java * (add) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjob-pytorch-yaml/valid-config-with-overrides.yaml * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/Dockerfile.cpu.tf_1.13.1 * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/tensorflow/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/generate_cifar10_tfrecords.py * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/FileSystemOperations.java * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TensorFlowLaunchCommand.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/pytorch/build-all.sh * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/TestLaunchCommandFactory.java * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/zeppelin-notebook-example/run_container.sh * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/component/TestTensorBoardComponent.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/ShowJobCli.java * (add) hadoop-submarine/hadoop-submarine-core/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/runjob/TestRunJobCliParsingCommonYaml.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/tensorflow/zeppelin-notebook-example/run_container.sh * (add) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjob-common-yaml/wrong-property-name.yaml * (delete) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/tensorboard-dockerimage-is-missing.yaml * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/LaunchScriptBuilder.java * (delete) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/AbstractLaunchCommandTestHelper.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/pytorch/component/PyTorchWorkerComponent.java * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TestTensorFlowLaunchCommand.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/tensorflow/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/cifar10_model.py * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/common/api/PyTorchRole.java * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/zeppelin-notebook-example/Dockerfile.gpu * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/ConfigType.java * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/LaunchCommandFactory.java * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TestTensorBoardLaunchCommand.java * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/yarnservice/TestYarnServiceRunJobCli.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/Cli.java * (edit)
[jira] [Commented] (SUBMARINE-64) Improve TonY runtime's document
[ https://issues.apache.org/jira/browse/SUBMARINE-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829929#comment-16829929 ] Hudson commented on SUBMARINE-64: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16479 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16479/]) SUBMARINE-64. Improve TonY runtime's document. Contributed by Keqiu Hu. (ztang: rev 24f218aef8e2eb4bf72e39fc030c2d8be0b9ac92) * (edit) hadoop-submarine/hadoop-submarine-tony-runtime/src/site/markdown/QuickStart.md > Improve TonY runtime's document > --- > > Key: SUBMARINE-64 > URL: https://issues.apache.org/jira/browse/SUBMARINE-64 > Project: Hadoop Submarine > Issue Type: Improvement >Reporter: Zhankun Tang >Priority: Major > Fix For: 0.2.0 > > Attachments: SUBMARINE-64.001.patch, tony.jpg > > > Found some minor issues when trying to run a job with TonY runtime. We should > document them in TonY's QuickStart.md > 1. There's seems no command flag "–container_resources" supported, we should > remove it in the section "Get the training examples". > 2. It should mention where can the user get the tony CLI, it seems there's > no 0.3.2 release yet so the user may build it by themselves. > 3. After the TonY's CLI jar is built, it complains with > "NoSuchMethodError:com.google.common.base.Preconditions.checkArgument" when > running which fails the AM directly. After I changed the guava version to the > same with Hadoop (27.0-jre), the job can succeed. > 4. There's no PyTorch example command there. We should add it. > Could you please help with this? [~oliverhuh...@gmail.com] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SUBMARINE-44) Upgrade Docker image's tensorflow version to 1.13.1
[ https://issues.apache.org/jira/browse/SUBMARINE-44?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825938#comment-16825938 ] Hudson commented on SUBMARINE-44: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16465 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16465/]) SUBMARINE-44. Upgrade Docker image's tensorflow version to 1.13.1. (ztang: rev b5dcf64f979f0225a394b5c928f92cb5f539bc6d) * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/WriteDockerfile.md * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.8.0/generate_cifar10_tfrecords.py * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/Dockerfile.cpu.tf_1.13.1 * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/model_base.py * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/base/ubuntu-16.04/Dockerfile.gpu.tf_1.13.1 * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0 * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/cifar10_main.py * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/InstallationGuideChineseVersion.md * (edit) hadoop-submarine/hadoop-submarine-core/src/main/docker/build-all.sh * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.8.0/cifar10_main.py * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.8.0/cifar10_utils.py * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/InstallationGuide.md * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.8.0/cifar10.py * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.8.0/cifar10_model.py * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0 * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.8.0/model_base.py * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/cifar10.py * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/cifar10_model.py * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/QuickStart.md * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/base/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0 * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/base/ubuntu-16.04/Dockerfile.cpu.tf_1.13.1 * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/base/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0 * (delete) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.8.0/README.md * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/cifar10_utils.py * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/generate_cifar10_tfrecords.py * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/RunningDistributedCifar10TFJobs.md * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/Dockerfile.gpu.tf_1.13.1 * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/TestAndTroubleshooting.md * (add) hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/README.md > Upgrade Docker image's tensorflow version to 1.13.1 > --- > > Key: SUBMARINE-44 > URL: https://issues.apache.org/jira/browse/SUBMARINE-44 > Project: Hadoop Submarine > Issue Type: Task >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Major > Fix For: 0.2.0 > > Attachments: SUBMARINE-44.001.patch, SUBMARINE-44.002.patch, > SUBMARINE-44.003.patch, SUBMARINE-44.004.patch, SUBMARINE-44.005.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SUBMARINE-54) Add test coverage for YarnServiceJobSubmitter and make it ready for extension for PyTorch
[ https://issues.apache.org/jira/browse/SUBMARINE-54?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825752#comment-16825752 ] Hudson commented on SUBMARINE-54: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16464 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16464/]) SUBMARINE-54. Add test coverage for YarnServiceJobSubmitter and make it (ztang: rev 0b3d41bdeef68afdde0fa4dca27fb582c83424c4) * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/YarnServiceUtils.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/utils/TestClassPathUtilities.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TensorFlowLaunchCommand.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/RunJobParameters.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/package-info.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/package-info.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TestTensorFlowLaunchCommand.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TestTensorBoardLaunchCommand.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/utils/package-info.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/component/TestTensorBoardComponent.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/AbstractLaunchCommand.java * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/TestTFConfigGenerator.java * (edit) hadoop-submarine/hadoop-submarine-core/src/test/java/org/apache/hadoop/yarn/submarine/common/fs/MockRemoteDirectoryManager.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/FileUtilitiesForTests.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/yarnservice/TestYarnServiceRunJobCliCommons.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/component/TestTensorFlowWorkerComponent.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/FileSystemOperations.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TensorBoardLaunchCommand.java * (edit) hadoop-submarine/hadoop-submarine-core/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/TestRunJobCliParsing.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/TestServiceWrapper.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/TensorFlowServiceSpec.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/AbstractLaunchCommandTestHelper.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/LaunchCommandFactory.java * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/YarnServiceJobSubmitter.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/package-info.java * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/pom.xml * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/TensorFlowCommons.java * (add) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TensorFlowWorkerLaunchCommand.java * (add)
[jira] [Commented] (SUBMARINE-40) Add TonY runtime to Submarine
[ https://issues.apache.org/jira/browse/SUBMARINE-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823802#comment-16823802 ] Hudson commented on SUBMARINE-40: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16451 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16451/]) SUBMARINE-40. Add TonY runtime to Submarine. Contributed by Keqiu Hu. (ztang: rev e79a9c12c14dcecc674048ed182e74d8690e663a) * (add) hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/buider/JobStatusBuilder.java * (add) hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/buider/package-info.java * (add) hadoop-submarine/hadoop-submarine-tony-runtime/src/site/resources/css/site.css * (add) hadoop-submarine/hadoop-submarine-tony-runtime/README.md * (add) hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/TonyJobSubmitter.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/RunJobParameters.java * (add) hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/package-info.java * (edit) hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/YarnServiceJobSubmitter.java * (add) hadoop-submarine/hadoop-submarine-tony-runtime/src/test/java/TestTonyUtils.java * (edit) hadoop-submarine/hadoop-submarine-core/src/site/markdown/QuickStart.md * (add) hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/TonyJobMonitor.java * (edit) hadoop-submarine/pom.xml * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/CliConstants.java * (add) hadoop-submarine/hadoop-submarine-tony-runtime/src/site/markdown/QuickStart.md * (add) hadoop-submarine/hadoop-submarine-tony-runtime/pom.xml * (add) hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/TonyUtils.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/RunJobCli.java * (add) hadoop-submarine/hadoop-submarine-tony-runtime/src/site/site.xml * (add) hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/TonyRuntimeFactory.java > Add TonY runtime to Submarine > - > > Key: SUBMARINE-40 > URL: https://issues.apache.org/jira/browse/SUBMARINE-40 > Project: Hadoop Submarine > Issue Type: New Feature >Affects Versions: 0.2.0 >Reporter: Keqiu Hu >Assignee: Keqiu Hu >Priority: Major > Labels: submarine, tony > Fix For: 0.2.0 > > Attachments: SUBMARINE-40.003.patch, SUBMARINE-40.004.patch, > SUBMARINE-40.005.patch, SUBMARINE-40.006.patch, SUBMARINE-40.007.patch, > SUBMARINE-40.008.patch, SUBMARINE-40.009.patch, SUBMARINE-40.010.patch, > SUBMARINE-40.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > Currently Submarine depends on YARN native service, we need another runtime > to support Hadoop versions w/o native service. Plan is to add a TonY runtime > into Submarine eco system. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SUBMARINE-47) Provide an implementation to parse configuration values from a YAML file for submarine run CLI
[ https://issues.apache.org/jira/browse/SUBMARINE-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811414#comment-16811414 ] Hudson commented on SUBMARINE-47: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16356 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16356/]) SUBMARINE-47. Provide an implementation to parse configuration values (sunilg: rev 1b9ba0ebb20952d848d205269f4dffe7985c5107) * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/ParametersHolder.java * (add) hadoop-submarine/hadoop-submarine-core/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/TestRunJobCliParsingYaml.java * (add) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/tensorboard-dockerimage-is-missing.yaml * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Configs.java * (edit) hadoop-submarine/hadoop-submarine-core/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/TestRunJobCliParsing.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/RunJobParameters.java * (add) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/security-principal-is-missing.yaml * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/YamlConfigFile.java * (add) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/test-false-values.yaml * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Spec.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/PsRole.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/TensorBoard.java * (add) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/envs-are-missing.yaml * (add) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/missing-configs.yaml * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Scheduling.java * (add) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/some-sections-missing.yaml * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Role.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/package-info.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/RunParameters.java * (edit) hadoop-submarine/hadoop-submarine-core/pom.xml * (add) hadoop-submarine/hadoop-submarine-core/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/TestRunJobCliParsingYamlStandalone.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Security.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/WorkerRole.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/ShowJobCli.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/YamlParseException.java * (add) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/valid-config.yaml * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/CliConstants.java * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/BaseParameters.java * (add) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Roles.java * (add) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/wrong-property-name.yaml * (edit) hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/RunJobCli.java * (add) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/valid-config-with-overrides.yaml * (add) hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/wrong-indentation.yaml * (add) hadoop-submarine/hadoop-submarine-core/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/YamlConfigTestUtils.java > Provide an implementation to parse configuration values from a YAML file for > submarine run CLI > -- > > Key: SUBMARINE-47 > URL: https://issues.apache.org/jira/browse/SUBMARINE-47 >
[jira] [Commented] (SUBMARINE-41) Fix ASF warnings related to submarine target files
[
https://issues.apache.org/jira/browse/SUBMARINE-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786392#comment-16786392
]
Hudson commented on SUBMARINE-41:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16150 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16150/])
SUBMARINE-41. Fix ASF warnings related to submarine target files. (sunilg: rev
09a9938db77e5c3a23c27b0d96088db9b15b46b9)
* (edit) pom.xml
> Fix ASF warnings related to submarine target files
> --
>
> Key: SUBMARINE-41
> URL: https://issues.apache.org/jira/browse/SUBMARINE-41
> Project: Hadoop Submarine
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.2.0
>
> Attachments: SUBMARINE-41.00.patch, SUBMARINE-41.01.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header:
> !? hadoop-submarine/hadoop-submarine-core/target/.plxarc
> !? hadoop-submarine/hadoop-submarine-core/target/antrun/build-main.xml
> !?
> hadoop-submarine/hadoop-submarine-core/target/surefire-reports/org.apache.hadoop.yarn.submarine.client.cli.TestRunJobCliParsing.txt
> !?
> hadoop-submarine/hadoop-submarine-core/target/surefire-reports/org.apache.hadoop.yarn.submarine.client.cli.TestShowJobCliParsing-output.txt
> !?
> hadoop-submarine/hadoop-submarine-core/target/surefire-reports/TEST-org.apache.hadoop.yarn.submarine.client.cli.TestShowJobCliParsing.xml
> !?
> hadoop-submarine/hadoop-submarine-core/target/surefire-reports/org.apache.hadoop.yarn.submarine.client.cli.TestRunJobCliParsing-output.txt
> !?
> hadoop-submarine/hadoop-submarine-core/target/surefire-reports/org.apache.hadoop.yarn.submarine.runtimes.common.TestFSBasedSubmarineStorage.txt
> !?
> hadoop-submarine/hadoop-submarine-core/target/surefire-reports/TEST-org.apache.hadoop.yarn.submarine.client.cli.TestRunJobCliParsing.xml
> !?
> hadoop-submarine/hadoop-submarine-core/target/surefire-reports/org.apache.hadoop.yarn.submarine.runtimes.common.TestFSBasedSubmarineStorage-output.txt
> !?
> hadoop-submarine/hadoop-submarine-core/target/surefire-reports/TEST-org.apache.hadoop.yarn.submarine.runtimes.common.TestFSBasedSubmarineStorage.xml
> !?
> hadoop-submarine/hadoop-submarine-core/target/surefire-reports/org.apache.hadoop.yarn.submarine.client.cli.TestShowJobCliParsing.txt
> {code}
> https://builds.apache.org/job/PreCommit-YARN-Build/23492/artifact/out/patch-asflicense-problems.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
