This is an automated email from the ASF dual-hosted git repository.
pingsutw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 5cd74b9 SUBMARINE-1009. Support mypy for Submarine SDK
5cd74b9 is described below
commit 5cd74b996fc8ca3e1870e159fcd81e516b31b496
Author: KUAN-HSUN-LI <[email protected]>
AuthorDate: Tue Sep 7 15:55:23 2021 +0800
SUBMARINE-1009. Support mypy for Submarine SDK
### What is this PR for?
[mypy](https://github.com/python/mypy) is an optional static type checker
for Python.
With `mypy` we can find bugs in our programs without even running them.
### What type of PR is it?
[Improvement]
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-1009
### How should this be tested?
ci pass
### Screenshots (if appropriate)
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? Yes
Author: KUAN-HSUN-LI <[email protected]>
Signed-off-by: Kevin <[email protected]>
Closes #739 from KUAN-HSUN-LI/SUBMARINE-1009 and squashes the following
commits:
6203323a [KUAN-HSUN-LI] fix ci
950dbdce [KUAN-HSUN-LI] SUBMARINE-1009. update development docs
e0821715 [KUAN-HSUN-LI] SUBMARINE-1009. fix files incompatible with mypy
e36f5347 [KUAN-HSUN-LI] SUBMARINE-1009. mypy requirements
---
.github/workflows/python.yml | 1 +
dev-support/style-check/python/lint.sh | 2 ++
.../python/{lint.sh => mypy-requirements.txt} | 20 +++++---------------
.../pysubmarine/submarine/experiment/api_client.py | 2 +-
.../submarine/ml/pytorch/input/libsvm_dataset.py | 11 +++++++----
.../pysubmarine/submarine/store/database/models.py | 4 +++-
.../pysubmarine/submarine/tracking/fluent.py | 1 -
.../submarine-sdk/pysubmarine/development.md | 6 +++++-
.../submarine-sdk/pysubmarine/development.md | 6 +++++-
9 files changed, 29 insertions(+), 24 deletions(-)
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
index 9ffdb4d..a741316 100644
--- a/.github/workflows/python.yml
+++ b/.github/workflows/python.yml
@@ -41,6 +41,7 @@ jobs:
pip install --no-cache-dir ./submarine-sdk/pysubmarine/.
pip install -r
./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
pip install -r ./dev-support/style-check/python/lint-requirements.txt
+ pip install -r ./dev-support/style-check/python/mypy-requirements.txt
- name: Check python sdk code style
run: ./dev-support/style-check/python/lint.sh
- name: Run unit test
diff --git a/dev-support/style-check/python/lint.sh
b/dev-support/style-check/python/lint.sh
index db1f3af..596e6c5 100755
--- a/dev-support/style-check/python/lint.sh
+++ b/dev-support/style-check/python/lint.sh
@@ -26,5 +26,7 @@ isort -c submarine-sdk/ dev-support/ website/
black submarine-sdk/ dev-support/ website/ --check
--experimental-string-processing
# Check lint: Not checking the code in website
flake8 submarine-sdk/ dev-support/
+# Check static typing for Python
+mypy --ignore-missing-imports submarine-sdk/pysubmarine/submarine/
set +euxo pipefail
diff --git a/dev-support/style-check/python/lint.sh
b/dev-support/style-check/python/mypy-requirements.txt
old mode 100755
new mode 100644
similarity index 67%
copy from dev-support/style-check/python/lint.sh
copy to dev-support/style-check/python/mypy-requirements.txt
index db1f3af..6a581e3
--- a/dev-support/style-check/python/lint.sh
+++ b/dev-support/style-check/python/mypy-requirements.txt
@@ -1,4 +1,3 @@
-#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,17 +13,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -euxo pipefail
-
-FWDIR="$(cd "$(dirname "$0")"; pwd)"
-cd "$FWDIR"
-cd ../../../
-
-# Check imports
-isort -c submarine-sdk/ dev-support/ website/
-# Check code format
-black submarine-sdk/ dev-support/ website/ --check
--experimental-string-processing
-# Check lint: Not checking the code in website
-flake8 submarine-sdk/ dev-support/
-
-set +euxo pipefail
+mypy==0.910
+types-requests==2.25.6
+types-certifi==2020.4.0
+types-six==1.16.1
+types-python-dateutil==2.8.0
diff --git a/submarine-sdk/pysubmarine/submarine/experiment/api_client.py
b/submarine-sdk/pysubmarine/submarine/experiment/api_client.py
index 54c18d3..2414243 100644
--- a/submarine-sdk/pysubmarine/submarine/experiment/api_client.py
+++ b/submarine-sdk/pysubmarine/submarine/experiment/api_client.py
@@ -71,7 +71,7 @@ class ApiClient(object):
PRIMITIVE_TYPES = (float, bool, bytes, six.text_type) + six.integer_types
NATIVE_TYPES_MAPPING = {
"int": int,
- "long": int if six.PY3 else long, # noqa: F821
+ "long": int,
"float": float,
"str": str,
"bool": bool,
diff --git
a/submarine-sdk/pysubmarine/submarine/ml/pytorch/input/libsvm_dataset.py
b/submarine-sdk/pysubmarine/submarine/ml/pytorch/input/libsvm_dataset.py
index 0ec3b0d..76efa3b 100644
--- a/submarine-sdk/pysubmarine/submarine/ml/pytorch/input/libsvm_dataset.py
+++ b/submarine-sdk/pysubmarine/submarine/ml/pytorch/input/libsvm_dataset.py
@@ -17,7 +17,7 @@ import functools
import itertools
import multiprocessing as mp
import os
-from typing import List, Tuple
+from typing import List, Optional, Tuple
import numpy as np
import torch
@@ -53,9 +53,12 @@ class LIBSVMDataset(Dataset):
return feature_idx, feature_value, int(label)
@classmethod
- def prepare_dataset(cls, data_uri: str, n_jobs: int = os.cpu_count()):
- sample_offset =
LIBSVMDataset._locate_sample_offsets(data_uri=data_uri, n_jobs=n_jobs)
- return LIBSVMDataset(data_uri=data_uri, sample_offset=sample_offset)
+ def prepare_dataset(cls, data_uri: str, n_jobs: Optional[int] =
os.cpu_count()):
+ if n_jobs is None:
+ raise Exception("No enough cpu!")
+ else:
+ sample_offset =
LIBSVMDataset._locate_sample_offsets(data_uri=data_uri, n_jobs=n_jobs)
+ return LIBSVMDataset(data_uri=data_uri,
sample_offset=sample_offset)
@classmethod
def _locate_sample_offsets(cls, data_uri: str, n_jobs: int) -> np.ndarray:
diff --git a/submarine-sdk/pysubmarine/submarine/store/database/models.py
b/submarine-sdk/pysubmarine/submarine/store/database/models.py
index 8b9752e..c4593d7 100644
--- a/submarine-sdk/pysubmarine/submarine/store/database/models.py
+++ b/submarine-sdk/pysubmarine/submarine/store/database/models.py
@@ -14,6 +14,7 @@
# limitations under the License.
import time
+from typing import Any
import sqlalchemy as sa
from sqlalchemy import BigInteger, Boolean, Column, PrimaryKeyConstraint,
String
@@ -21,7 +22,8 @@ from sqlalchemy.ext.declarative import declarative_base
from submarine.entities import Metric, Param
-Base = declarative_base()
+# Base class in sqlalchemy is a dynamic type
+Base: Any = declarative_base()
#
+--------------------+-------+-------------------+--------------+---------------+------+--------+
diff --git a/submarine-sdk/pysubmarine/submarine/tracking/fluent.py
b/submarine-sdk/pysubmarine/submarine/tracking/fluent.py
index 816569e..f7ee031 100644
--- a/submarine-sdk/pysubmarine/submarine/tracking/fluent.py
+++ b/submarine-sdk/pysubmarine/submarine/tracking/fluent.py
@@ -25,7 +25,6 @@ from submarine.tracking.client import SubmarineClient
from submarine.tracking.utils import get_job_id, get_worker_index
_RUN_ID_ENV_VAR = "SUBMARINE_RUN_ID"
-_active_run_stack = []
_logger = logging.getLogger(__name__)
diff --git a/website/docs/userDocs/submarine-sdk/pysubmarine/development.md
b/website/docs/userDocs/submarine-sdk/pysubmarine/development.md
index d2f18b6..537cf82 100644
--- a/website/docs/userDocs/submarine-sdk/pysubmarine/development.md
+++ b/website/docs/userDocs/submarine-sdk/pysubmarine/development.md
@@ -30,6 +30,9 @@ conda activate submarine-dev
# Install auto-format and lints (lint-requirements.txt is in
./dev-support/style-check/python)
pip install -r lint-requirements.txt
+# Install mypy (mypy-requirements.txt is in ./dev-support/style-check/python)
+pip install -r mypy-requirements.txt
+
# test-requirements.txt is in ./submarine-sdk/pysubmarine/github-actions
pip install -r test-requirements.txt
@@ -62,7 +65,8 @@ The script does the following things:
./dev-support/style-check/python/auto-format.sh
```
-- Use [flake8](https://github.com/PyCQA/flake8) to verify the linter, its'
configure is in `.flake8`
+- Use [flake8](https://github.com/PyCQA/flake8) to verify the linter, its'
configure is in `.flake8`.
+- Also, we are using [mypy](https://github.com/python/mypy) to check the
static type in `submarine-sdk/pysubmarine/submarine`.
- Verify linter pass before submitting a pull request by running:
```bash
diff --git
a/website/versioned_docs/version-0.6.0/userDocs/submarine-sdk/pysubmarine/development.md
b/website/versioned_docs/version-0.6.0/userDocs/submarine-sdk/pysubmarine/development.md
index d2f18b6..537cf82 100644
---
a/website/versioned_docs/version-0.6.0/userDocs/submarine-sdk/pysubmarine/development.md
+++
b/website/versioned_docs/version-0.6.0/userDocs/submarine-sdk/pysubmarine/development.md
@@ -30,6 +30,9 @@ conda activate submarine-dev
# Install auto-format and lints (lint-requirements.txt is in
./dev-support/style-check/python)
pip install -r lint-requirements.txt
+# Install mypy (mypy-requirements.txt is in ./dev-support/style-check/python)
+pip install -r mypy-requirements.txt
+
# test-requirements.txt is in ./submarine-sdk/pysubmarine/github-actions
pip install -r test-requirements.txt
@@ -62,7 +65,8 @@ The script does the following things:
./dev-support/style-check/python/auto-format.sh
```
-- Use [flake8](https://github.com/PyCQA/flake8) to verify the linter, its'
configure is in `.flake8`
+- Use [flake8](https://github.com/PyCQA/flake8) to verify the linter, its'
configure is in `.flake8`.
+- Also, we are using [mypy](https://github.com/python/mypy) to check the
static type in `submarine-sdk/pysubmarine/submarine`.
- Verify linter pass before submitting a pull request by running:
```bash
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
