[GitHub] [submarine] tangzhankun commented on a change in pull request #175: SUBMRINE-323. Add how to release guide and scripts.
tangzhankun commented on a change in pull request #175: SUBMRINE-323. Add how
to release guide and scripts.
URL: https://github.com/apache/submarine/pull/175#discussion_r376195917
##
File path: dev-support/cicd/create_release.sh
##
@@ -0,0 +1,110 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# The script helps making a release.
+# You need to specify release version and branch|tag name.
+#
+# Here are some helpful documents for the release.
+# http://www.apache.org/dev/release.html
+# http://www.apache.org/dev/release-publishing
+# http://www.apache.org/dev/release-signing.html
+
+set -e
+
+BASEDIR="$(dirname "$0")"
+. "${BASEDIR}/common_release.sh"
+echo "${BASEDIR}/common_release.sh"
+
+if [[ $# -ne 2 ]]; then
+ usage
+fi
+
+for var in GPG_PASSPHRASE; do
+ if [[ -z "${!var}" ]]; then
+echo "You need ${var} variable set"
+exit 1
+ fi
+done
+
+RELEASE_VERSION="$1"
+GIT_TAG="$2"
+
+function compile_src_and_bin() {
+ cd ${WORKING_DIR}/submarine
+ echo "git submodule update --init --recursive"
+ git submodule update --init --recursive
+ echo "git submodule update --recursive"
+ git submodule update --recursive
Review comment:
Thanks for reminding this. Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-371) Remove tony submodule
[ https://issues.apache.org/jira/browse/SUBMARINE-371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zac Zhou resolved SUBMARINE-371. Fix Version/s: 0.4.0 Resolution: Fixed Issue resolved by pull request 173 [https://github.com/apache/submarine/pull/173] > Remove tony submodule > - > > Key: SUBMARINE-371 > URL: https://issues.apache.org/jira/browse/SUBMARINE-371 > Project: Apache Submarine > Issue Type: Improvement > Components: K8s Submitter >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > To work with tony community more closely, we need to remove tony submodule > and use tony dependency directly. We should not maintain tony code by > ourselves. If a patch to tony is needed, we should submit a PR to tony > repository -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine] asfgit closed pull request #173: SUBMARINE-371. Remove tony submodule.
asfgit closed pull request #173: SUBMARINE-371. Remove tony submodule. URL: https://github.com/apache/submarine/pull/173 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Work logged] (SUBMARINE-371) Remove tony submodule
[ https://issues.apache.org/jira/browse/SUBMARINE-371?focusedWorklogId=383318&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-383318 ] ASF GitHub Bot logged work on SUBMARINE-371: Author: ASF GitHub Bot Created on: 07/Feb/20 02:24 Start Date: 07/Feb/20 02:24 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #173: SUBMARINE-371. Remove tony submodule. URL: https://github.com/apache/submarine/pull/173 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 383318) Time Spent: 20m (was: 10m) > Remove tony submodule > - > > Key: SUBMARINE-371 > URL: https://issues.apache.org/jira/browse/SUBMARINE-371 > Project: Apache Submarine > Issue Type: Improvement > Components: K8s Submitter >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > To work with tony community more closely, we need to remove tony submodule > and use tony dependency directly. We should not maintain tony code by > ourselves. If a patch to tony is needed, we should submit a PR to tony > repository -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[submarine] branch master updated: SUBMARINE-371. Remove tony submodule.
This is an automated email from the ASF dual-hosted git repository.
zhouquan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 82fae82 SUBMARINE-371. Remove tony submodule.
82fae82 is described below
commit 82fae8227fb18e4be5137360d16d06f3a61bda8a
Author: Quan zhou
AuthorDate: Mon Feb 3 16:33:10 2020 +0800
SUBMARINE-371. Remove tony submodule.
### What is this PR for?
To work with tony community more closely, we need to remove tony submodule
and use tony dependency directly. We should not maintain tony code by
ourselves. If a patch to tony is needed, we should submit a PR to tony
repository
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-371
### How should this be tested?
https://travis-ci.org/yuanzac/hadoop-submarine/builds/646704729?utm_source=github_status&utm_medium=notification
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
Author: Quan zhou
Closes #173 from yuanzac/topic/SUBMARINE-371 and squashes the following
commits:
e045f11 [Quan zhou] SUBMARINE-371. Remove tony submodule.
---
.gitmodules| 4
.travis.yml| 3 +--
dev-support/docker-images/submarine/build.sh | 6 -
dev-support/mini-submarine/build_mini-submarine.sh | 7 --
docs/development/BuildFromCode.md | 21 ++---
pom.xml| 3 +--
.../server-submitter/submitter-yarn/pom.xml| 26 ++
.../submarine/server/submitter/yarn/YarnUtils.java | 5 ++---
.../src/test/java/YarnUtilsTest.java | 6 +++--
submodules/tony| 1 -
10 files changed, 36 insertions(+), 46 deletions(-)
diff --git a/.gitmodules b/.gitmodules
index adf143c..e69de29 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,4 +0,0 @@
-[submodule "submodules/tony"]
- path = submodules/tony
- url = https://github.com/hadoopsubmarine/linkedin-tony.git
- branch = submarine
diff --git a/.travis.yml b/.travis.yml
index 58cf9dc..b0b82b0 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -46,7 +46,6 @@ env:
-
EXCLUDE_SUBMARINE="!submarine-all,!submarine-client,!submarine-commons,!submarine-commons/commons-runtime,!submarine-dist,!submarine-server/server-submitter/submitter-yarn,!submarine-server/server-core,!submarine-server/server-rpc"
-
EXCLUDE_WORKBENCH="!submarine-workbench,!submarine-workbench/workbench-web,!submarine-workbench/workbench-web-ng"
-
EXCLUDE_INTERPRETER="!submarine-workbench/interpreter,!submarine-workbench/interpreter/interpreter-engine,!submarine-workbench/interpreter/python-interpreter,!submarine-workbench/interpreter/spark-interpreter"
--
EXCLUDE_SUBMODULE_TONY="!submodules/tony,!submodules/tony/tony-mini,!submodules/tony/tony-core,!submodules/tony/tony-proxy,!submodules/tony/tony-portal,!submodules/tony/tony-azkaban,!submodules/tony/tony-cli"
- EXCLUDE_K8S="!submarine-server/server-submitter/submitter-k8s"
- EXCLUDE_COMMON_RPC="!submarine-commons/commons-rpc"
- MOZ_HEADLESS=1
@@ -176,7 +175,7 @@ matrix:
language: java
jdk: "openjdk8"
dist: xenial
- env: PYTHON="3" PROFILE="-Phadoop-2.9" BUILD_FLAG="clean package install
-DskipTests" TEST_FLAG="test -DskipRat -am" MODULES="-pl
${EXCLUDE_K8S},${EXCLUDE_SUBMARINE},${EXCLUDE_WORKBENCH},${EXCLUDE_SUBMODULE_TONY}"
TEST_MODULES="-pl $(echo ${EXCLUDE_INTERPRETER} | sed 's/!//g')"
TEST_PROJECTS=""
+ env: PYTHON="3" PROFILE="-Phadoop-2.9" BUILD_FLAG="clean package install
-DskipTests" TEST_FLAG="test -DskipRat -am" MODULES="-pl
${EXCLUDE_K8S},${EXCLUDE_SUBMARINE},${EXCLUDE_WORKBENCH}" TEST_MODULES="-pl
$(echo ${EXCLUDE_INTERPRETER} | sed 's/!//g')" TEST_PROJECTS=""
- name: Test submarine-sdk
language: python
diff --git a/dev-support/docker-images/submarine/build.sh
b/dev-support/docker-images/submarine/build.sh
index 831b3e2..691af90 100755
--- a/dev-support/docker-images/submarine/build.sh
+++ b/dev-support/docker-images/submarine/build.sh
@@ -33,12 +33,6 @@ fi
submarine_dist_exists=$(find -L "${SUBMARINE_HOME}/submarine-dist/target"
-name "submarine-dist-${SUBMARINE_VERSION}*.tar.gz")
# Build source code if the package doesn't exist.
if [[ -z "${submarine_dist_exists}" ]]; then
- # update tony code
- if is_empty_dir "${SUBMARINE_HOME}/submodules/tony" ]; then
-git submodule update --init --recursive
- else
-git submodule update --recursive
- fi
cd "${SUBMARINE_HOME}"
mvn clean package -DskipTests
fi
diff --git a/dev-
[GitHub] [submarine] yuanzac commented on issue #174: SUBMARINE-202. submarine core need to support MXNet
yuanzac commented on issue #174: SUBMARINE-202. submarine core need to support MXNet URL: https://github.com/apache/submarine/pull/174#issuecomment-583202932 @lowc1012, Thanks for the contributions. Can you take a look at the travis error? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
Re: [ANNOUNCE] Apache Submarine 0.3.0 release!
Thanks to all contributors! Let's continue to develop better features for submarine!! On Thu, Feb 6, 2020 at 7:25 PM Sunil Govindan wrote: > Awesome! Hearty Congratulations to all of you for the first release of > Submarine as Apache TLP! > Kudos to all who helped in this release! > > - Sunil > > On Thu, Feb 6, 2020 at 4:36 PM Zhankun Tang wrote: > > > Hi folks, > > > > It's a great honor for me to announce that the Apache Submarine community > > has released Apache Submarine 0.3.0! > > > > Apache Submarine 0.3.0 is the first release after the spin-off from > Apache > > Hadoop. It includes 196 patches since the prior version. > > The highlighted features are: > > - Mini-submarine (YARN) > > - Basic Tensorflow job submission to k8s through submarine-server RESTful > > API > > - Job submission on YARN through submarine-server RPC protocol > > > > Tons of thanks to our contributors and community! Let's keep fighting! > > > > *Apache Submarine 0.3.0 released*: > > http://submarine.apache.org/releases/submarine-release-0.3.0.html > > *Changelog*: https://s.apache.org/4ezw1 > > > > BR, > > Zhankun > > >
[GitHub] [submarine] liuxunorg commented on a change in pull request #175: SUBMRINE-323. Add how to release guide and scripts.
liuxunorg commented on a change in pull request #175: SUBMRINE-323. Add how to
release guide and scripts.
URL: https://github.com/apache/submarine/pull/175#discussion_r375798881
##
File path: dev-support/cicd/publish_release.sh
##
@@ -0,0 +1,182 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# The script helps publishing release to maven.
+# You need to specify release version and branch|tag name.
+#
+# Here's some helpful documents for the release.
+# http://www.apache.org/dev/publishing-maven-artifacts.html
+set -e
+BASEDIR="$(dirname "$0")"
+. "${BASEDIR}/common_release.sh"
+
+if [[ $# -ne 2 ]]; then
+ usage
+fi
+
+for var in GPG_PASSPHRASE ASF_USERID ASF_PASSWORD; do
+ if [[ -z "${!var}" ]]; then
+echo "You need ${var} variable set"
+exit 1
+ fi
+done
+
+export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512m"
+RED='\033[0;31m'
+NC='\033[0m' # No Color
+
+RELEASE_VERSION="$1"
+GIT_TAG="$2"
+if [[ $RELEASE_VERSION == *"SNAPSHOT"* ]]; then
+ DO_SNAPSHOT="yes"
+fi
+
+NEXUS_STAGING="https://repository.apache.org/service/local/staging";
+# Get this from apache infra admin
+NEXUS_PROFILE="2525cde13fad2a"
+
+function cleanup() {
+ echo "Remove working directory and maven local repository"
+ rm -rf ${WORKING_DIR}
+}
+
+function curl_error() {
+ ret=${1}
+ if [[ $ret -ne 0 ]]; then
+echo "curl response code is: ($ret)"
+echo "See https://curl.haxx.se/libcurl/c/libcurl-errors.html to know the
detailed cause of error."
+echo -e "${RED}Failed to publish maven artifact to staging repository."
+echo -e "IMPORTANT: You will have to re-run publish_release.sh to complete
maven artifact publish.${NC}"
+cleanup
+exit 1
+ fi
+}
+
+
+#
+# Publishing Apache Submarine artifact to Apache snapshot repository.
+#
+function publish_snapshot_to_maven() {
+ cd "${WORKING_DIR}/submarine"
+ echo "Deploying Apache Submarine $RELEASE_VERSION version to snapshot
repository."
+
+ if [[ ! $RELEASE_VERSION == *"SNAPSHOT"* ]]; then
+echo "ERROR: Snapshots must have a version containing 'SNAPSHOT'"
+echo "ERROR: You gave version '$RELEASE_VERSION'"
+exit 1
+ fi
+
+ tmp_repo="$(mktemp -d /tmp/submarine-repo-X)"
+ echo "git submodule update --init --recursive"
+ git submodule update --init --recursive
+ echo "git submodule update --recursive"
+ git submodule update --recursive
Review comment:
submodule tony will be delete. https://github.com/apache/submarine/pull/173
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #175: SUBMRINE-323. Add how to release guide and scripts.
liuxunorg commented on a change in pull request #175: SUBMRINE-323. Add how to
release guide and scripts.
URL: https://github.com/apache/submarine/pull/175#discussion_r375798199
##
File path: dev-support/cicd/create_release.sh
##
@@ -0,0 +1,110 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# The script helps making a release.
+# You need to specify release version and branch|tag name.
+#
+# Here are some helpful documents for the release.
+# http://www.apache.org/dev/release.html
+# http://www.apache.org/dev/release-publishing
+# http://www.apache.org/dev/release-signing.html
+
+set -e
+
+BASEDIR="$(dirname "$0")"
+. "${BASEDIR}/common_release.sh"
+echo "${BASEDIR}/common_release.sh"
+
+if [[ $# -ne 2 ]]; then
+ usage
+fi
+
+for var in GPG_PASSPHRASE; do
+ if [[ -z "${!var}" ]]; then
+echo "You need ${var} variable set"
+exit 1
+ fi
+done
+
+RELEASE_VERSION="$1"
+GIT_TAG="$2"
+
+function compile_src_and_bin() {
+ cd ${WORKING_DIR}/submarine
+ echo "git submodule update --init --recursive"
+ git submodule update --init --recursive
+ echo "git submodule update --recursive"
+ git submodule update --recursive
Review comment:
submodule tony will be delete. https://github.com/apache/submarine/pull/173
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #175: SUBMRINE-323. Add how to release guide and scripts.
liuxunorg commented on a change in pull request #175: SUBMRINE-323. Add how to
release guide and scripts.
URL: https://github.com/apache/submarine/pull/175#discussion_r375799064
##
File path: dev-support/cicd/publish_release.sh
##
@@ -0,0 +1,182 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# The script helps publishing release to maven.
+# You need to specify release version and branch|tag name.
+#
+# Here's some helpful documents for the release.
+# http://www.apache.org/dev/publishing-maven-artifacts.html
+set -e
+BASEDIR="$(dirname "$0")"
+. "${BASEDIR}/common_release.sh"
+
+if [[ $# -ne 2 ]]; then
+ usage
+fi
+
+for var in GPG_PASSPHRASE ASF_USERID ASF_PASSWORD; do
+ if [[ -z "${!var}" ]]; then
+echo "You need ${var} variable set"
+exit 1
+ fi
+done
+
+export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512m"
+RED='\033[0;31m'
+NC='\033[0m' # No Color
+
+RELEASE_VERSION="$1"
+GIT_TAG="$2"
+if [[ $RELEASE_VERSION == *"SNAPSHOT"* ]]; then
+ DO_SNAPSHOT="yes"
+fi
+
+NEXUS_STAGING="https://repository.apache.org/service/local/staging";
+# Get this from apache infra admin
+NEXUS_PROFILE="2525cde13fad2a"
+
+function cleanup() {
+ echo "Remove working directory and maven local repository"
+ rm -rf ${WORKING_DIR}
+}
+
+function curl_error() {
+ ret=${1}
+ if [[ $ret -ne 0 ]]; then
+echo "curl response code is: ($ret)"
+echo "See https://curl.haxx.se/libcurl/c/libcurl-errors.html to know the
detailed cause of error."
+echo -e "${RED}Failed to publish maven artifact to staging repository."
+echo -e "IMPORTANT: You will have to re-run publish_release.sh to complete
maven artifact publish.${NC}"
+cleanup
+exit 1
+ fi
+}
+
+
+#
+# Publishing Apache Submarine artifact to Apache snapshot repository.
+#
+function publish_snapshot_to_maven() {
+ cd "${WORKING_DIR}/submarine"
+ echo "Deploying Apache Submarine $RELEASE_VERSION version to snapshot
repository."
+
+ if [[ ! $RELEASE_VERSION == *"SNAPSHOT"* ]]; then
+echo "ERROR: Snapshots must have a version containing 'SNAPSHOT'"
+echo "ERROR: You gave version '$RELEASE_VERSION'"
+exit 1
+ fi
+
+ tmp_repo="$(mktemp -d /tmp/submarine-repo-X)"
+ echo "git submodule update --init --recursive"
+ git submodule update --init --recursive
+ echo "git submodule update --recursive"
+ git submodule update --recursive
+ mvn versions:set -DnewVersion=$RELEASE_VERSION
+ tmp_settings="tmp-settings.xml"
+ echo "" > $tmp_settings
+ echo "apache.snapshots.https$ASF_USERID" >>
$tmp_settings
+ echo "$ASF_PASSWORD" >> $tmp_settings
+ echo "" >> $tmp_settings
+
+ #Disable TonY mvn deploy
+ #tony_pom="./submodules/tony/pom.xml"
+ mvn --settings $tmp_settings -Dmaven.repo.local="${tmp_repo}" -DskipTests
deploy
Review comment:
submodule tony will be delete. https://github.com/apache/submarine/pull/173
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine] pingsutw commented on issue #174: SUBMARINE-202. submarine core need to support MXNet
pingsutw commented on issue #174: SUBMARINE-202. submarine core need to support MXNet URL: https://github.com/apache/submarine/pull/174#issuecomment-582865062 @lowc1012 Thanks for the contribution. - We also need to add `RunJobCliParsingMxnetTest.java`, `RunJobCliParsingMxnetYamlTest.java` like `submarine/submarine-client/src/test/java/org/apache/submarine/client/cli/runjob/pytorch/` https://github.com/apache/submarine/tree/master/submarine-client/src/test/java/org/apache/submarine/client/cli/runjob/pytorch - This PR looks like we only support MXnet run on TonY runtime, we need to make sure it can also support RPC runtime and restful runtime, you could file another JIRA ticket This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine] tangzhankun opened a new pull request #175: SUBMRINE-323. Add how to release guide and scripts.
tangzhankun opened a new pull request #175: SUBMRINE-323. Add how to release guide and scripts. URL: https://github.com/apache/submarine/pull/175 ### What is this PR for? Submarine release scripts and guide ### What type of PR is it? Feature ### Todos * [1] - Automate steps in cicd images ### What is the Jira issue? https://issues.apache.org/jira/browse/SUBMARINE-323 ### How should this be tested? When a committer is preparing a new release, this can be verified. Zhankun has gone through the guide with the scripts. No big issue found yet. ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
Re: [ANNOUNCE] Apache Submarine 0.3.0 release!
Awesome! Hearty Congratulations to all of you for the first release of Submarine as Apache TLP! Kudos to all who helped in this release! - Sunil On Thu, Feb 6, 2020 at 4:36 PM Zhankun Tang wrote: > Hi folks, > > It's a great honor for me to announce that the Apache Submarine community > has released Apache Submarine 0.3.0! > > Apache Submarine 0.3.0 is the first release after the spin-off from Apache > Hadoop. It includes 196 patches since the prior version. > The highlighted features are: > - Mini-submarine (YARN) > - Basic Tensorflow job submission to k8s through submarine-server RESTful > API > - Job submission on YARN through submarine-server RPC protocol > > Tons of thanks to our contributors and community! Let's keep fighting! > > *Apache Submarine 0.3.0 released*: > http://submarine.apache.org/releases/submarine-release-0.3.0.html > *Changelog*: https://s.apache.org/4ezw1 > > BR, > Zhankun >
[ANNOUNCE] Apache Submarine 0.3.0 release!
Hi folks, It's a great honor for me to announce that the Apache Submarine community has released Apache Submarine 0.3.0! Apache Submarine 0.3.0 is the first release after the spin-off from Apache Hadoop. It includes 196 patches since the prior version. The highlighted features are: - Mini-submarine (YARN) - Basic Tensorflow job submission to k8s through submarine-server RESTful API - Job submission on YARN through submarine-server RPC protocol Tons of thanks to our contributors and community! Let's keep fighting! *Apache Submarine 0.3.0 released*: http://submarine.apache.org/releases/submarine-release-0.3.0.html *Changelog*: https://s.apache.org/4ezw1 BR, Zhankun
[jira] [Updated] (SUBMARINE-202) submarine core need to support MXNet
[ https://issues.apache.org/jira/browse/SUBMARINE-202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Lo updated SUBMARINE-202: -- Status: Patch Available (was: Reopened) Issue resolved by pull request 174 [https://github.com/apache/submarine/pull/174] > submarine core need to support MXNet > > > Key: SUBMARINE-202 > URL: https://issues.apache.org/jira/browse/SUBMARINE-202 > Project: Apache Submarine > Issue Type: Bug >Reporter: Zac Zhou >Assignee: Ryan Lo >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Submarine core checks the machine learning framework types which must be > tensorflow or PyTorch, MXNet is supported by tony. Submarine core should > enable MXNet as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Work logged] (SUBMARINE-202) submarine core need to support MXNet
[ https://issues.apache.org/jira/browse/SUBMARINE-202?focusedWorklogId=382809&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382809 ] ASF GitHub Bot logged work on SUBMARINE-202: Author: ASF GitHub Bot Created on: 06/Feb/20 10:18 Start Date: 06/Feb/20 10:18 Worklog Time Spent: 10m Work Description: lowc1012 commented on pull request #174: SUBMARINE-202. submarine core need to support MXNet URL: https://github.com/apache/submarine/pull/174 ### What is this PR for? To support MXNet framework in Submarine ### What type of PR is it? [Improvement] ### Todos * [ ] - Task ### What is the Jira issue? [SUBMARINE-202](https://issues.apache.org/jira/projects/SUBMARINE/issues/SUBMARINE-202) ### How should this be tested? [passed CI](https://travis-ci.org/lowc1012/submarine/builds/646785076) ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382809) Remaining Estimate: 0h Time Spent: 10m > submarine core need to support MXNet > > > Key: SUBMARINE-202 > URL: https://issues.apache.org/jira/browse/SUBMARINE-202 > Project: Apache Submarine > Issue Type: Bug >Reporter: Zac Zhou >Assignee: Ryan Lo >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Submarine core checks the machine learning framework types which must be > tensorflow or PyTorch, MXNet is supported by tony. Submarine core should > enable MXNet as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-202) submarine core need to support MXNet
[ https://issues.apache.org/jira/browse/SUBMARINE-202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SUBMARINE-202: - Labels: pull-request-available (was: ) > submarine core need to support MXNet > > > Key: SUBMARINE-202 > URL: https://issues.apache.org/jira/browse/SUBMARINE-202 > Project: Apache Submarine > Issue Type: Bug >Reporter: Zac Zhou >Assignee: Ryan Lo >Priority: Major > Labels: pull-request-available > > Submarine core checks the machine learning framework types which must be > tensorflow or PyTorch, MXNet is supported by tony. Submarine core should > enable MXNet as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine] lowc1012 opened a new pull request #174: SUBMARINE-202. submarine core need to support MXNet
lowc1012 opened a new pull request #174: SUBMARINE-202. submarine core need to support MXNet URL: https://github.com/apache/submarine/pull/174 ### What is this PR for? To support MXNet framework in Submarine ### What type of PR is it? [Improvement] ### Todos * [ ] - Task ### What is the Jira issue? [SUBMARINE-202](https://issues.apache.org/jira/projects/SUBMARINE/issues/SUBMARINE-202) ### How should this be tested? [passed CI](https://travis-ci.org/lowc1012/submarine/builds/646785076) ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine-site] liuxunorg merged pull request #12: delete contributors in submarine release note
liuxunorg merged pull request #12: delete contributors in submarine release note URL: https://github.com/apache/submarine-site/pull/12 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine-site] liuxunorg merged pull request #13: delete contributors in 0.3.0 release note html
liuxunorg merged pull request #13: delete contributors in 0.3.0 release note html URL: https://github.com/apache/submarine-site/pull/13 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine-site] tangzhankun opened a new pull request #13: delete contributors in 0.3.0 release note html
tangzhankun opened a new pull request #13: delete contributors in 0.3.0 release note html URL: https://github.com/apache/submarine-site/pull/13 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine-site] tangzhankun opened a new pull request #12: delete contributors in submarine release note
tangzhankun opened a new pull request #12: delete contributors in submarine release note URL: https://github.com/apache/submarine-site/pull/12 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-377) [SDK] Submit a job by pysubamrine
Kevin Su created SUBMARINE-377: -- Summary: [SDK] Submit a job by pysubamrine Key: SUBMARINE-377 URL: https://issues.apache.org/jira/browse/SUBMARINE-377 Project: Apache Submarine Issue Type: Sub-task Reporter: Kevin Su Assignee: Kevin Su Fix For: 0.4.0 Use SDK to submit a submarine job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-376) [Umbrealla] Submarine python client API
Kevin Su created SUBMARINE-376: -- Summary: [Umbrealla] Submarine python client API Key: SUBMARINE-376 URL: https://issues.apache.org/jira/browse/SUBMARINE-376 Project: Apache Submarine Issue Type: New Feature Components: SDK Reporter: Kevin Su Assignee: Kevin Su Fix For: 0.4.0 Currently, we use the command line to submit a Submarine job, it will be helpful to submit a job from python API. On the server-side, we start a submarine RPC server. O the client-side, we use *pip* to install python package so that we can submit/list/delete submarine jobs by gprc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
Re: [VOTE] Submarine-0.3.0-RC0 is ready for a vote! Re: [Submarine Release Plan] Submarine-0.3.0
Hi folks, Thanks for the vote. We've got 4 +1(binding) and 1 +1 (non-binding). The vote has passed! I will announce the release soon. BR, Zhankun On Mon, 3 Feb 2020 at 20:45, Sunil Govindan wrote: > Thanks Zhankun for clarifying. > > I am converting my vote to +1 (binding) > Appreciate all of your efforts for this release > > - Sunil > > On Mon, Feb 3, 2020 at 1:44 PM Zhankun Tang wrote: > > > Hi Sunil, > > > > Thanks for the help in verifying it! > > > > For the 404-Not Found of the maven staging repo, it's because I've > clicked > > the release button for it. We've got 3 binding +1 and 1 non-binding +1. > I'm > > already doing the publishing. :) > > For the submarine docker images, it's stored under the apache/submarine > > repo created by "apache" user(maintained by Apache INFRA) in Docker hub. > > And the submarine images are maintained by the docker hub users owned by > > submarine PMCs. See here for more details. > > https://issues.apache.org/jira/browse/INFRA-19650 > > > > BR, > > Zhankun > > > > > > On Mon, 3 Feb 2020 at 15:08, Sunil Govindan wrote: > > > > > Generally the tests looks good to me. > > > I tried to run locally and basic setup seems fine. > > > > > > Some doubts in the below parts. > > > 1. > > > > > > https://repository.apache.org/content/repositories/orgapachesubmarine-1001 > > > is > > > coming as 404-Not Found for me ? > > > 2. Could we use apache docker repo? or is this maintained by Submarine, > > if > > > so, how many images we can maintain here? > > > > > > Thanks > > > Sunil > > > > > > On Mon, Feb 3, 2020 at 9:18 AM Zac Zhou wrote: > > > > > > > +1 > > > > > > > > Zhankun, thanks for your proposal. > > > > > > > > kevin su 于2020年1月25日周六 下午8:18写道： > > > > > > > > > +1, thanks Zhankun > > > > > > > > > > Xun Liu 於 2020年1月25日 週六 下午7:48寫道： > > > > > > > > > > > +1. Thanks, Xun Liu. > > > > > > > > > > > > On Thu, Jan 23, 2020 at 12:08 AM Xun Liu > > wrote: > > > > > > > > > > > > > Thanks Zhankun & all the contributors for release 0.3.0 > > > > > > > > > > > > > > On Wed, Jan 22, 2020 at 6:44 PM Zhankun Tang > > > > > wrote: > > > > > > > > > > > > > >> Hi folks, > > > > > > >> > > > > > > >> Thanks to everyone's help on this release. Special thanks to > > > > Wanqiang, > > > > > > >> Zac, Kevin, Xun, and Jeff! > > > > > > >> > > > > > > >> I've created a release candidate (RC0) for submarine 0.3.0. > The > > > > > > >> highlighted features are as follows: > > > > > > >> - Mini-submarine > > > > > > >> - Basic Tensorflow job submission to k8s through > > submarine-server > > > > > > RESTful > > > > > > >> API > > > > > > >> - Job submission on YARN through submarine-server RPC protocol > > > > > > >> > > > > > > >> The RC tag in git is here: > > > > > > >> https://github.com/apache/submarine/tree/release-0.3.0-RC0 > > > > > > >> > > > > > > >> The RC release artifacts are available at: > > > > > > >> http://home.apache.org/~ztang/submarine-0.3.0-RC0/ > > > > > > >> > > > > > > >> The mini-submarine image is here: > > > > > > >> docker pull apache/submarine:mini-0.3.0-RC0 > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://hub.docker.com/layers/apache/submarine/mini-0.3.0-RC0/images/sha256-3dd49054bf8a91521f5743c675278d626a5fa568e91651c67867b8ba6ceba340 > > > > > > >> > > > > > > >> The Maven staging repository is here: > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapachesubmarine-1001 > > > > > > >> > > > > > > >> My public key is here: > > > > > > >> https://dist.apache.org/repos/dist/release/submarine/KEYS > > > > > > >> > > > > > > >> *This vote will run for 10 days, ending on Feb. 1st, 2020 at > > 11:59 > > > > pm > > > > > > >> PST.* > > > > > > >> > > > > > > >> For the testing, I have verified the build from source, run > the > > > > mnist > > > > > > >> example with mini-submarine(both local and remote mode) and > > > verified > > > > > > >> posting a mnist TFJob to K8s through the submarine. > > > > > > >> > > > > > > >> Please follow the document to test these features. > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/submarine/tree/master/dev-support/mini-submarine > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/submarine/blob/master/docs/submarine-server/README.md > > > > > > >> > > > > > > >> My +1 (binding) to start. Thanks! > > > > > > >> > > > > > > >> BR, > > > > > > >> Zhankun > > > > > > >> > > > > > > >> On Thu, 16 Jan 2020 at 18:11, yuan.zac.zq < > > yuan.zac...@gmail.com> > > > > > > wrote: > > > > > > >> > > > > > > >>> Thanks Zhankun & all the contributors for release 0.3.0 > > > > > > >>> > > > > > > >>> Zac Zhou > > > > > > >>> On 1/16/2020 18:08，Xun Liu < > > liu...@apache.org > > > > > > > > > > >>> wrote： > > > > > > >>> > > > > > > >>> Submarine 0.3.0 is release, which is really good news! > > > > > > >>> > > > > > > >>> On Thu, Jan 16, 2020 at 6:04 PM Rohith Shar
