This is an automated email from the ASF dual-hosted git repository.
cdmikechen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new ca25b9de SUBMARINE-1329. Improve test coverage of submarine.artifacts
module
ca25b9de is described below
commit ca25b9de97e730fd0ff4441eee198cda3e779e4d
Author: huang06 <[email protected]>
AuthorDate: Fri Sep 23 16:10:38 2022 +0800
SUBMARINE-1329. Improve test coverage of submarine.artifacts module
### What is this PR for?
We lack tests in the `submodule.artifacts` module. We should add tests and
aim for at least 90% test coverage for this module.
**Additionally**, this PR also fix SUBMARINE-922 issue since #994
accidentally modify the code.
### What type of PR is it?
Improvement
### Todos
### What is the Jira issue?
<https://issues.apache.org/jira/browse/SUBMARINE-1329>
### How should this be tested?
```bash
cd submarine-sdk/pysubmarine
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip setuptools wheel
pip install -e .[tf2,pytorch]
pip install -r github-actions/test-requirements.txt
```
```bash
$ pytest -m 'not e2e'
...
---------- coverage: platform linux, python 3.7.14-final-0 -----------
Name Stmts Miss
Cover Missing
-----------------------------------------------------------------------------------------
submarine/__init__.py 9 0
100%
submarine/artifacts/__init__.py 2 0
100%
submarine/artifacts/repository.py 30 0
100%
submarine/cli/__init__.py 2 0
100%
...
-----------------------------------------------------------------------------------------
TOTAL 4725 2621
45%
============================================================================ 42
passed, 7 skipped, 31 deselected, 10 warnings in 23.88s
============================================================================
```
### Screenshots (if appropriate)
### Questions:
* Do the license files need updating? Yes/No
* Are there breaking changes for older versions? Yes/No
* Does this need new documentation? Yes/No
Author: huang06 <[email protected]>
Signed-off-by: cdmikechen <[email protected]>
Closes #998 from huang06/test/pytest and squashes the following commits:
42df4353 [huang06] ci: undo codecov
8e665772 [huang06] docs: apply Apache License
1b5eb129 [huang06] feat: use pytest.ini as configuration file
7bbb66b1 [huang06] ci: update
2d1e335b [huang06] ci: restore
eb4eb128 [huang06] test: update
78667fcd [huang06] feat: ignore .mypy_cache/
6ce63719 [huang06] test: update
1fa6b50f [huang06] refactor: remove unused options
bc8b7e12 [huang06] ci: configure pytest options
01cd21f9 [huang06] style: black
9433a96c [huang06] ci: restore
481d5525 [huang06] feat: avoid GPL dependency
837a8557 [huang06] test: add tests for repository
e9f2c34e [huang06] test: add moto package
e3895fe0 [huang06] ci: test
ccc81713 [huang06] ci: test
6b3a4d67 [huang06] ci: generate pytest report with line numbers
---
.github/workflows/master.yml | 2 +-
.github/workflows/python.yml | 4 +-
.gitignore | 3 +
pyproject.toml | 4 -
.../github-actions/test-requirements.txt | 3 +-
submarine-sdk/pysubmarine/pytest.ini | 21 +++++
submarine-sdk/pysubmarine/setup.py | 2 +-
.../pysubmarine/tests/artifacts/__init__.py | 16 ++++
.../pysubmarine/tests/artifacts/test_repository.py | 96 ++++++++++++++++++++++
9 files changed, 142 insertions(+), 9 deletions(-)
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index 1f3ca70d..3c625cd3 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -560,7 +560,7 @@ jobs:
with:
path: |
./submarine-server/server-submitter/target/jacoco.exec
- key: ${{ runner.os }}-docker-${{ github.sha }}
+ key: ${{ runner.os }}-docker-${{ github.sha }}
- name: Cache submitter-k8s jacoco.exec
uses: actions/cache@v2
with:
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
index a159f59c..e118955b 100644
--- a/.github/workflows/python.yml
+++ b/.github/workflows/python.yml
@@ -78,7 +78,7 @@ jobs:
- name: List installed packages
run: pip list
- name: Run unit test
- run: pytest --cov=submarine -vs -m "not e2e"
+ run: pytest -m "not e2e"
integration:
runs-on: ubuntu-latest
@@ -168,7 +168,7 @@ jobs:
submarine config list
- name: Run integration test
working-directory: ./submarine-sdk/pysubmarine
- run: pytest --cov=submarine -vs -m "e2e"
+ run: pytest -m "e2e"
- name: Failure status
run: |
kubectl describe nodes
diff --git a/.gitignore b/.gitignore
index 312d4aa7..f45e20ed 100644
--- a/.gitignore
+++ b/.gitignore
@@ -105,3 +105,6 @@ dev-support/docker-images/operator/tmp/
# installation binary
submarine-serve/installation/istioctl
+
+# mypy
+.mypy_cache/
diff --git a/pyproject.toml b/pyproject.toml
index 26c7bd76..f1d80f5e 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -26,7 +26,3 @@ skip-string-normalization = true
[tool.mypy]
plugins = "sqlalchemy.ext.mypy.plugin"
ignore_missing_imports = true # suppress `Cannot find implementation or
library stub` error
-
-[tool.pytest.ini_options]
-addopts = "--strict-markers"
-markers = ["e2e"]
diff --git a/submarine-sdk/pysubmarine/github-actions/test-requirements.txt
b/submarine-sdk/pysubmarine/github-actions/test-requirements.txt
index 1f38d655..7319debb 100644
--- a/submarine-sdk/pysubmarine/github-actions/test-requirements.txt
+++ b/submarine-sdk/pysubmarine/github-actions/test-requirements.txt
@@ -15,6 +15,7 @@
certifi>=14.05.14
freezegun==1.2.2
+moto[s3]==4.0.5
pandas>=1.2.0,<=1.3.5 # 1.4.x does not support cp37
pylint==2.15.2
PyMySQL==1.0.2
@@ -22,6 +23,6 @@ pytest==7.1.3
pytest-cov==3.0.0
pytest-localserver==0.7.0
python_dateutil>=2.5.3
-scikit-learn>=0.24.2,<=1.0.2 # 1.1.x doest not support cp37
+scikit-learn>=0.24.2,<=1.0.2 # 1.1.x does not support cp37
setuptools>=21.0.0
urllib3>=1.15.1
diff --git a/submarine-sdk/pysubmarine/pytest.ini
b/submarine-sdk/pysubmarine/pytest.ini
new file mode 100644
index 00000000..fe68557e
--- /dev/null
+++ b/submarine-sdk/pysubmarine/pytest.ini
@@ -0,0 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[pytest]
+markers =
+ e2e
+addopts = -vs --strict-markers --cov submarine --cov-report term-missing
diff --git a/submarine-sdk/pysubmarine/setup.py
b/submarine-sdk/pysubmarine/setup.py
index f30c2000..5a74c74f 100644
--- a/submarine-sdk/pysubmarine/setup.py
+++ b/submarine-sdk/pysubmarine/setup.py
@@ -33,7 +33,7 @@ setup(
"sqlalchemy>=1.4.0",
"sqlparse",
"pymysql",
- "requests",
+ "requests>=2.26.0", # SUBMARINE-922. avoid GPL dependency.
"urllib3>=1.15.1",
"certifi>=14.05.14",
"python-dateutil>=2.5.3",
diff --git a/submarine-sdk/pysubmarine/tests/artifacts/__init__.py
b/submarine-sdk/pysubmarine/tests/artifacts/__init__.py
new file mode 100644
index 00000000..13a83393
--- /dev/null
+++ b/submarine-sdk/pysubmarine/tests/artifacts/__init__.py
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/submarine-sdk/pysubmarine/tests/artifacts/test_repository.py
b/submarine-sdk/pysubmarine/tests/artifacts/test_repository.py
new file mode 100644
index 00000000..0e421bbd
--- /dev/null
+++ b/submarine-sdk/pysubmarine/tests/artifacts/test_repository.py
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pathlib
+import shutil
+
+import boto3
+from moto import mock_s3
+
+from submarine.artifacts import Repository
+
+
+@mock_s3
+def test_log_artifact():
+ s3 = boto3.resource("s3")
+ s3.create_bucket(Bucket="submarine")
+
+ local_file = pathlib.Path(__file__).parent / "text.txt"
+ with local_file.open("w", encoding="utf-8") as file:
+ file.write("test")
+
+ repo = Repository()
+ dest_path = "folder01/subfolder01"
+ repo.log_artifact(dest_path=dest_path, local_file=str(local_file))
+ local_file.unlink()
+
+ common_prefixes = repo.list_artifact_subfolder("folder01")
+ assert common_prefixes == [{"Prefix": "folder01/subfolder01/"}]
+
+
+@mock_s3
+def test_log_artifacts():
+ s3 = boto3.resource("s3")
+ s3.create_bucket(Bucket="submarine")
+
+ # create the following directory tree:
+ # data/
+ # ├── subdir-00
+ # │ └── subdir-10
+ # │ └── text1.txt
+ # └── subdir-01
+ # └── subdir-10
+ # └── text2.txt
+ local_dir = pathlib.Path(__file__).parent / "data"
+ (local_dir / "subdir-00" / "subdir-10").mkdir(parents=True, exist_ok=True)
+ (local_dir / "subdir-01" / "subdir-10").mkdir(parents=True, exist_ok=True)
+ local_file1 = local_dir / "subdir-00" / "subdir-10" / "text1.txt"
+ with local_file1.open("w", encoding="utf-8") as file:
+ file.write("test")
+ local_file2 = local_dir / "subdir-01" / "subdir-10" / "text2.txt"
+ with local_file2.open("w", encoding="utf-8") as file:
+ file.write("test")
+
+ repo = Repository()
+ s3_folder_name = repo.log_artifacts(dest_path="data",
local_dir=str(local_dir))
+
+ shutil.rmtree(local_dir)
+
+ assert s3_folder_name == "s3://submarine/data"
+
+ common_prefixes = repo.list_artifact_subfolder("data")
+ assert common_prefixes == [{'Prefix': 'data/subdir-00/'}, {'Prefix':
'data/subdir-01/'}]
+
+
+@mock_s3
+def test_delete_folder():
+ s3 = boto3.resource("s3")
+ s3.create_bucket(Bucket="submarine")
+
+ local_file = pathlib.Path(__file__).parent / "text.txt"
+ with local_file.open("w", encoding="utf-8") as file:
+ file.write("test")
+
+ s3.meta.client.upload_file(str(local_file), "submarine",
"folder01/subfolder01/text.txt")
+ s3.meta.client.upload_file(str(local_file), "submarine",
"folder01/subfolder02/text.txt")
+ local_file.unlink()
+
+ repo = Repository()
+ repo.delete_folder("folder01/subfolder01")
+
+ common_prefixes = repo.list_artifact_subfolder("folder01")
+ assert common_prefixes == [{"Prefix": "folder01/subfolder02/"}]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
