This is an automated email from the ASF dual-hosted git repository.
pingsutw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 63e5f5c SUBMARINE-1071. Fix pysubmarine test of tensorflow2
63e5f5c is described below
commit 63e5f5c199841bd7cfb3b103252ae93cd3ebbde1
Author: featherchen <[email protected]>
AuthorDate: Tue Nov 2 16:54:41 2021 +0800
SUBMARINE-1071. Fix pysubmarine test of tensorflow2
### What is this PR for?
<!-- A few sentences describing the overall goals of the pull request's
commits.
First time? Check out the contributing guide -
https://submarine.apache.org/contribution/contributions.html
-->
Due to package version problem, we need to downgrade tensorflow-estimator
to 2.6.0 and upgrade numpy to 1.19.2.
Also I have refactored the workflow of python-sdk test to make it more
clear.
### What type of PR is it?
Bug Fix
### Todos
* [x] - Fix test
### What is the Jira issue?
<!-- * Open an issue on Jira
https://issues.apache.org/jira/browse/SUBMARINE/
* Put link here, and add [SUBMARINE-*Jira number*] in PR title, eg.
`SUBMARINE-23. PR title`
-->
https://issues.apache.org/jira/projects/SUBMARINE/issues/SUBMARINE-1071?filter=allopenissues
### How should this be tested?
<!--
* First time? Setup Travis CI as described on
https://submarine.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed
behavior
* Outline any manual steps to test the PR here.
-->
By Github action
### Screenshots (if appropriate)
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? No
Author: featherchen <[email protected]>
Signed-off-by: Kevin <[email protected]>
Closes #792 from featherchen/SUBMARINE-1071 and squashes the following
commits:
68fb4606 [featherchen] change job name
7d231ae8 [featherchen] refactor
3df75c20 [featherchen] refactor
2d4dc795 [featherchen] refactor
25c77769 [featherchen] refactor
e231e510 [featherchen] add -e
e1348d56 [featherchen] echo
01afc4ce [featherchen] refactor python workflow of CI/CD
c556f434 [featherchen] refactor python workflow of CI/CD
0108c249 [featherchen] refactor python workflow of CI/CD
bdc7545c [featherchen] refactor python workflow of CI/CD
e342e00b [featherchen] refactor python workflow of CI/CD
bb316cda [featherchen] refactor python workflow of CI/CD
a9f307b7 [featherchen] refactor python workflow of CI/CD
042ae7c4 [featherchen] downgrade tensorflow-estimator to 2.6.0
4185c5e5 [featherchen] downgrade tensorflow-estimator to 2.6.0
1ad8e8be [featherchen] downgrade tensorflow-estimator to 2.6.0
---
.github/workflows/python.yml | 34 ++++++++++++++--------
submarine-sdk/pysubmarine/setup.py | 11 +++++--
.../ml/tensorflow/model/test_base_tf_model.py | 4 +--
.../tests/ml/tensorflow/model/test_ccpm.py | 4 +--
.../tests/ml/tensorflow/model/test_deepfm.py | 5 ++--
.../tests/ml/tensorflow/model/test_fm.py | 4 +--
.../tests/ml/tensorflow/model/test_nfm.py | 4 +--
.../tests/ml/tensorflow/test_optimizer.py | 4 +--
.../ml/tensorflow_v2/model/test_base_tf_model.py | 4 +--
.../tests/ml/tensorflow_v2/model/test_ccpm.py | 4 +--
.../tests/ml/tensorflow_v2/model/test_deepfm.py | 4 +--
.../tests/ml/tensorflow_v2/model/test_fm.py | 5 ++--
.../tests/ml/tensorflow_v2/model/test_nfm.py | 4 +--
.../tests/ml/tensorflow_v2/test_optimizer.py | 5 ++--
14 files changed, 57 insertions(+), 39 deletions(-)
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
index f5b6b64..137e200 100644
--- a/.github/workflows/python.yml
+++ b/.github/workflows/python.yml
@@ -19,13 +19,26 @@ name: python-sdk
on: [push, pull_request]
jobs:
+ check-style:
+ runs-on: ubuntu-latest
+ timeout-minutes: 30
+ steps:
+ - uses: actions/checkout@v2
+ - name: Install dependencies
+ run: |
+ pip install --upgrade pip
+ pip install -r ./dev-support/style-check/python/lint-requirements.txt
+ pip install -r ./dev-support/style-check/python/mypy-requirements.txt
+ - name: Check python sdk code style
+ run: ./dev-support/style-check/python/lint.sh
+
unit:
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
matrix:
python-version: [3.6, 3.7]
- tf-version: [1.14.0, 1.15.0, 2.6.0]
+ tf-version: [1.15.0, 2.6.0]
fail-fast: false
steps:
- uses: actions/checkout@v2
@@ -33,19 +46,16 @@ jobs:
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- - name: Install dependencies
+ - name: Install requirements
run: |
- pip install --upgrade pip
- pip install --no-cache-dir tensorflow==${{ matrix.tf-version }}
- pip install --no-cache-dir torch==1.5.0
- pip install --no-cache-dir tensorflow-addons
- pip install --no-cache-dir tf_slim
- pip install --no-cache-dir ./submarine-sdk/pysubmarine/.
+ pip install --upgrade pip
pip install -r
./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
- pip install -r ./dev-support/style-check/python/lint-requirements.txt
- pip install -r ./dev-support/style-check/python/mypy-requirements.txt
- - name: Check python sdk code style
- run: ./dev-support/style-check/python/lint.sh
+ - name: Install pysubmarine with tf1 and pytorch
+ if: matrix.tf-version == '1.15.0'
+ run: pip install --no-cache-dir -e
./submarine-sdk/pysubmarine/.[tf,pytorch]
+ - name: Install pysubmarine with tf2 and pytorch
+ if: matrix.tf-version == '2.6.0'
+ run: pip install --no-cache-dir -e
./submarine-sdk/pysubmarine/.[tf2,pytorch]
- name: Run unit test
run: pytest --cov=submarine -vs -m "not e2e"
diff --git a/submarine-sdk/pysubmarine/setup.py
b/submarine-sdk/pysubmarine/setup.py
index eb2a687..1935c01 100644
--- a/submarine-sdk/pysubmarine/setup.py
+++ b/submarine-sdk/pysubmarine/setup.py
@@ -28,7 +28,7 @@ setup(
packages=find_packages(exclude=["tests", "tests.*"]),
install_requires=[
"six>=1.10.0",
- "numpy==1.18.5",
+ "numpy==1.19.2",
"pandas",
"sqlalchemy>=1.4.0",
"sqlparse",
@@ -42,8 +42,13 @@ setup(
"boto3>=1.17.58",
],
extras_require={
- "tf": ["tensorflow>=1.14.0,<2.0.0"],
- "tf2": ["tensorflow==2.6.0", "tf_slim==1.1.0",
"tensorflow-addons==0.14.0"],
+ "tf": ["tensorflow==1.15.0"],
+ "tf2": [
+ "tensorflow==2.6.0",
+ "tf_slim==1.1.0",
+ "tensorflow-addons==0.14.0",
+ "tensorflow-estimator==2.6.0",
+ ],
"pytorch": ["torch>=1.5.0", "torchvision>=0.6.0"],
},
classifiers=[
diff --git
a/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_base_tf_model.py
b/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_base_tf_model.py
index 8bf7915..17978e2 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_base_tf_model.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_base_tf_model.py
@@ -16,11 +16,11 @@
import pytest
import tensorflow as tf
-from submarine.ml.tensorflow.model.base_tf_model import BaseTFModel
-
@pytest.mark.skipif(tf.__version__ >= "2.0.0", reason="requires tf1")
def test_create_base_tf_model():
+ from submarine.ml.tensorflow.model.base_tf_model import BaseTFModel
+
params = {"learning rate": 0.05}
with pytest.raises(AssertionError, match="Does not define any input
parameters"):
BaseTFModel(params)
diff --git a/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_ccpm.py
b/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_ccpm.py
index bc0fa87..b3422c1 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_ccpm.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_ccpm.py
@@ -16,11 +16,11 @@
import pytest
import tensorflow as tf
-from submarine.ml.tensorflow.model import CCPM
-
@pytest.mark.skipif(tf.__version__ >= "2.0.0", reason="requires tf1")
def test_run_ccpm(get_model_param):
+ from submarine.ml.tensorflow.model import CCPM
+
params = get_model_param
model = CCPM(model_params=params)
diff --git a/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_deepfm.py
b/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_deepfm.py
index 2a6bd8d..e9aff62 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_deepfm.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_deepfm.py
@@ -16,11 +16,12 @@
import pytest
import tensorflow as tf
-from submarine.ml.tensorflow.model import DeepFM
-
@pytest.mark.skipif(tf.__version__ >= "2.0.0", reason="requires tf1")
def test_run_deepfm(get_model_param):
+
+ from submarine.ml.tensorflow.model import DeepFM
+
params = get_model_param
model = DeepFM(model_params=params)
diff --git a/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_fm.py
b/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_fm.py
index fb5abb8..200c0a7 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_fm.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_fm.py
@@ -16,11 +16,11 @@
import pytest
import tensorflow as tf
-from submarine.ml.tensorflow.model import FM
-
@pytest.mark.skipif(tf.__version__ >= "2.0.0", reason="requires tf1")
def test_run_fm(get_model_param):
+ from submarine.ml.tensorflow.model import FM
+
params = get_model_param
model = FM(model_params=params)
diff --git a/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_nfm.py
b/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_nfm.py
index 5000810..980aa44 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_nfm.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow/model/test_nfm.py
@@ -16,11 +16,11 @@
import pytest
import tensorflow as tf
-from submarine.ml.tensorflow.model import NFM
-
@pytest.mark.skipif(tf.__version__ >= "2.0.0", reason="requires tf1")
def test_run_nfm(get_model_param):
+ from submarine.ml.tensorflow.model import NFM
+
params = get_model_param
model = NFM(model_params=params)
diff --git a/submarine-sdk/pysubmarine/tests/ml/tensorflow/test_optimizer.py
b/submarine-sdk/pysubmarine/tests/ml/tensorflow/test_optimizer.py
index 776ed49..4841dcd 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow/test_optimizer.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow/test_optimizer.py
@@ -16,11 +16,11 @@
import pytest
import tensorflow as tf
-from submarine.ml.tensorflow.optimizer import get_optimizer
-
@pytest.mark.skipif(tf.__version__ >= "2.0.0", reason="requires tf1")
def test_get_optimizer():
+ from submarine.ml.tensorflow.optimizer import get_optimizer
+
optimizer_keys = ["adam", "adagrad", "momentum", "ftrl"]
invalid_optimizer_keys = ["adddam"]
diff --git
a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_base_tf_model.py
b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_base_tf_model.py
index e706abb..74556ac 100644
---
a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_base_tf_model.py
+++
b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_base_tf_model.py
@@ -16,11 +16,11 @@
import pytest
import tensorflow as tf
-from submarine.ml.tensorflow_v2.model.base_tf_model import BaseTFModel
-
@pytest.mark.skipif(tf.__version__ < "2.0.0", reason="requires tf2")
def test_create_base_tf_model():
+ from submarine.ml.tensorflow_v2.model.base_tf_model import BaseTFModel
+
params = {"learning rate": 0.05}
with pytest.raises(AssertionError, match="Does not define any input
parameters"):
BaseTFModel(params)
diff --git
a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_ccpm.py
b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_ccpm.py
index 858b6bb..99d78df 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_ccpm.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_ccpm.py
@@ -16,11 +16,11 @@
import pytest
import tensorflow as tf
-from submarine.ml.tensorflow_v2.model import CCPM
-
@pytest.mark.skipif(tf.__version__ < "2.0.0", reason="requires tf2")
def test_run_ccpm(get_model_param):
+ from submarine.ml.tensorflow_v2.model import CCPM
+
params = get_model_param
model = CCPM(model_params=params)
diff --git
a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_deepfm.py
b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_deepfm.py
index c6f2d4d..bcaecf2 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_deepfm.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_deepfm.py
@@ -16,11 +16,11 @@
import pytest
import tensorflow as tf
-from submarine.ml.tensorflow_v2.model import DeepFM
-
@pytest.mark.skipif(tf.__version__ < "2.0.0", reason="requires tf2")
def test_run_deepfm(get_model_param):
+ from submarine.ml.tensorflow_v2.model import DeepFM
+
params = get_model_param
model = DeepFM(model_params=params)
diff --git a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_fm.py
b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_fm.py
index 1d95304..e8e5717 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_fm.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_fm.py
@@ -16,11 +16,12 @@
import pytest
import tensorflow as tf
-from submarine.ml.tensorflow_v2.model import FM
-
@pytest.mark.skipif(tf.__version__ < "2.0.0", reason="requires tf2")
def test_run_fm(get_model_param):
+
+ from submarine.ml.tensorflow_v2.model import FM
+
params = get_model_param
model = FM(model_params=params)
diff --git a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_nfm.py
b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_nfm.py
index 9de05f3..301c255 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_nfm.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/model/test_nfm.py
@@ -16,11 +16,11 @@
import pytest
import tensorflow as tf
-from submarine.ml.tensorflow_v2.model import NFM
-
@pytest.mark.skipif(tf.__version__ < "2.0.0", reason="requires tf2")
def test_run_nfm(get_model_param):
+ from submarine.ml.tensorflow_v2.model import NFM
+
params = get_model_param
model = NFM(model_params=params)
diff --git a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/test_optimizer.py
b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/test_optimizer.py
index 1ef8fa1..e539812 100644
--- a/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/test_optimizer.py
+++ b/submarine-sdk/pysubmarine/tests/ml/tensorflow_v2/test_optimizer.py
@@ -16,11 +16,12 @@
import pytest
import tensorflow as tf
-from submarine.ml.tensorflow_v2.optimizer import get_optimizer
-
@pytest.mark.skipif(tf.__version__ < "2.0.0", reason="requires tf2")
def test_get_optimizer():
+
+ from submarine.ml.tensorflow_v2.optimizer import get_optimizer
+
optimizer_keys = ["adam", "adagrad", "momentum", "ftrl"]
invalid_optimizer_keys = ["adddam"]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
