[jira] [Commented] (SPARK-32797) Install mypy on the Jenkins CI workers
[
https://issues.apache.org/jira/browse/SPARK-32797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17386524#comment-17386524
]
Shane Knapp commented on SPARK-32797:
-
i've temporarily disabled mypy on the workers:
for x in `cat ~/txt/jenkins-workers.txt`; do ssh jenkins@${x} "mv
/home/jenkins/anaconda2/envs/py36/bin/mypy
/home/jenkins/anaconda2/envs/py36/bin/mypy.bak"; done
the files that need fixing are here:
{noformat}
starting mypy test...
mypy checks failed:
python/pyspark/pandas/data_type_ops/categorical_ops.py:128: error: Function is
missing a type annotation
python/pyspark/mllib/tree.pyi:29: error: Overloaded function signatures 1 and 2
overlap with incompatible return types
python/pyspark/mllib/tree.pyi:38: error: Overloaded function signatures 1 and 2
overlap with incompatible return types
python/pyspark/mllib/feature.pyi:34: error: Overloaded function signatures 1
and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:42: error: Overloaded function signatures 1
and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:48: error: Overloaded function signatures 1
and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:54: error: Overloaded function signatures 1
and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:76: error: Overloaded function signatures 1
and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:124: error: Overloaded function signatures 1
and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:165: error: Overloaded function signatures 1
and 2 overlap with incompatible return types
python/pyspark/mllib/clustering.pyi:45: error: Overloaded function signatures 1
and 2 overlap with incompatible return types
python/pyspark/mllib/clustering.pyi:72: error: Overloaded function signatures 1
and 2 overlap with incompatible return types
python/pyspark/mllib/classification.pyi:39: error: Overloaded function
signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/classification.pyi:52: error: Overloaded function
signatures 1 and 2 overlap with incompatible return types
{noformat}
could someone create a PR that addresses these, and once that's done i'll
re-enable mypy on the workers and we'll coordinate a quick test.
> Install mypy on the Jenkins CI workers
> --
>
> Key: SPARK-32797
> URL: https://issues.apache.org/jira/browse/SPARK-32797
> Project: Spark
> Issue Type: Improvement
> Components: jenkins, PySpark
>Affects Versions: 3.1.0
>Reporter: Fokko Driesprong
>Assignee: Shane Knapp
>Priority: Major
> Fix For: 3.3.0
>
>
> We want to check the types of the PySpark code. This requires mypy to be
> installed on the CI. Can you do this [~shaneknapp]?
> Related PR: [https://github.com/apache/spark/pull/29180]
> You can install this using pip: [https://pypi.org/project/mypy/] Should be
> similar to flake8 and sphinx. The latest version is ok! Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32797) Install mypy on the Jenkins CI workers
[ https://issues.apache.org/jira/browse/SPARK-32797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17386522#comment-17386522 ] Shane Knapp commented on SPARK-32797: - [~hyukjin.kwon] this is now causing all builds to fail (properly, i assume, but fail nonetheless): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141578/console > Install mypy on the Jenkins CI workers > -- > > Key: SPARK-32797 > URL: https://issues.apache.org/jira/browse/SPARK-32797 > Project: Spark > Issue Type: Improvement > Components: jenkins, PySpark >Affects Versions: 3.1.0 >Reporter: Fokko Driesprong >Assignee: Shane Knapp >Priority: Major > Fix For: 3.3.0 > > > We want to check the types of the PySpark code. This requires mypy to be > installed on the CI. Can you do this [~shaneknapp]? > Related PR: [https://github.com/apache/spark/pull/29180] > You can install this using pip: [https://pypi.org/project/mypy/] Should be > similar to flake8 and sphinx. The latest version is ok! Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32797) Install mypy on the Jenkins CI workers
[ https://issues.apache.org/jira/browse/SPARK-32797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17386515#comment-17386515 ] Apache Spark commented on SPARK-32797: -- User 'shaneknapp' has created a pull request for this issue: https://github.com/apache/spark/pull/33503 > Install mypy on the Jenkins CI workers > -- > > Key: SPARK-32797 > URL: https://issues.apache.org/jira/browse/SPARK-32797 > Project: Spark > Issue Type: Improvement > Components: jenkins, PySpark >Affects Versions: 3.1.0 >Reporter: Fokko Driesprong >Assignee: Shane Knapp >Priority: Major > Fix For: 3.3.0 > > > We want to check the types of the PySpark code. This requires mypy to be > installed on the CI. Can you do this [~shaneknapp]? > Related PR: [https://github.com/apache/spark/pull/29180] > You can install this using pip: [https://pypi.org/project/mypy/] Should be > similar to flake8 and sphinx. The latest version is ok! Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32797) Install mypy on the Jenkins CI workers
[ https://issues.apache.org/jira/browse/SPARK-32797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17386514#comment-17386514 ] Apache Spark commented on SPARK-32797: -- User 'shaneknapp' has created a pull request for this issue: https://github.com/apache/spark/pull/33503 > Install mypy on the Jenkins CI workers > -- > > Key: SPARK-32797 > URL: https://issues.apache.org/jira/browse/SPARK-32797 > Project: Spark > Issue Type: Improvement > Components: jenkins, PySpark >Affects Versions: 3.1.0 >Reporter: Fokko Driesprong >Assignee: Shane Knapp >Priority: Major > Fix For: 3.3.0 > > > We want to check the types of the PySpark code. This requires mypy to be > installed on the CI. Can you do this [~shaneknapp]? > Related PR: [https://github.com/apache/spark/pull/29180] > You can install this using pip: [https://pypi.org/project/mypy/] Should be > similar to flake8 and sphinx. The latest version is ok! Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32797) Install mypy on the Jenkins CI workers
[ https://issues.apache.org/jira/browse/SPARK-32797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385124#comment-17385124 ] Apache Spark commented on SPARK-32797: -- User 'shaneknapp' has created a pull request for this issue: https://github.com/apache/spark/pull/33469 > Install mypy on the Jenkins CI workers > -- > > Key: SPARK-32797 > URL: https://issues.apache.org/jira/browse/SPARK-32797 > Project: Spark > Issue Type: Improvement > Components: jenkins, PySpark >Affects Versions: 3.1.0 >Reporter: Fokko Driesprong >Assignee: Shane Knapp >Priority: Major > > We want to check the types of the PySpark code. This requires mypy to be > installed on the CI. Can you do this [~shaneknapp]? > Related PR: [https://github.com/apache/spark/pull/29180] > You can install this using pip: [https://pypi.org/project/mypy/] Should be > similar to flake8 and sphinx. The latest version is ok! Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32797) Install mypy on the Jenkins CI workers
[ https://issues.apache.org/jira/browse/SPARK-32797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385123#comment-17385123 ] Apache Spark commented on SPARK-32797: -- User 'shaneknapp' has created a pull request for this issue: https://github.com/apache/spark/pull/33469 > Install mypy on the Jenkins CI workers > -- > > Key: SPARK-32797 > URL: https://issues.apache.org/jira/browse/SPARK-32797 > Project: Spark > Issue Type: Improvement > Components: jenkins, PySpark >Affects Versions: 3.1.0 >Reporter: Fokko Driesprong >Assignee: Shane Knapp >Priority: Major > > We want to check the types of the PySpark code. This requires mypy to be > installed on the CI. Can you do this [~shaneknapp]? > Related PR: [https://github.com/apache/spark/pull/29180] > You can install this using pip: [https://pypi.org/project/mypy/] Should be > similar to flake8 and sphinx. The latest version is ok! Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32797) Install mypy on the Jenkins CI workers
[ https://issues.apache.org/jira/browse/SPARK-32797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385038#comment-17385038 ] Shane Knapp commented on SPARK-32797: - ill roll this out (and other python package updates) later today/this week. > Install mypy on the Jenkins CI workers > -- > > Key: SPARK-32797 > URL: https://issues.apache.org/jira/browse/SPARK-32797 > Project: Spark > Issue Type: Improvement > Components: jenkins, PySpark >Affects Versions: 3.1.0 >Reporter: Fokko Driesprong >Assignee: Shane Knapp >Priority: Major > > We want to check the types of the PySpark code. This requires mypy to be > installed on the CI. Can you do this [~shaneknapp]? > Related PR: [https://github.com/apache/spark/pull/29180] > You can install this using pip: [https://pypi.org/project/mypy/] Should be > similar to flake8 and sphinx. The latest version is ok! Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32797) Install mypy on the Jenkins CI workers
[ https://issues.apache.org/jira/browse/SPARK-32797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246209#comment-17246209 ] Hyukjin Kwon commented on SPARK-32797: -- We need this for Python 3 in master and branch-3.1 > Install mypy on the Jenkins CI workers > -- > > Key: SPARK-32797 > URL: https://issues.apache.org/jira/browse/SPARK-32797 > Project: Spark > Issue Type: Improvement > Components: jenkins, PySpark >Affects Versions: 3.1.0 >Reporter: Fokko Driesprong >Assignee: Shane Knapp >Priority: Major > > We want to check the types of the PySpark code. This requires mypy to be > installed on the CI. Can you do this [~shaneknapp]? > Related PR: [https://github.com/apache/spark/pull/29180] > You can install this using pip: [https://pypi.org/project/mypy/] Should be > similar to flake8 and sphinx. The latest version is ok! Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32797) Install mypy on the Jenkins CI workers
[ https://issues.apache.org/jira/browse/SPARK-32797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245395#comment-17245395 ] Shane Knapp commented on SPARK-32797: - should this be installed for both pyton2.7 and 3.6? > Install mypy on the Jenkins CI workers > -- > > Key: SPARK-32797 > URL: https://issues.apache.org/jira/browse/SPARK-32797 > Project: Spark > Issue Type: Improvement > Components: jenkins, PySpark >Affects Versions: 3.1.0 >Reporter: Fokko Driesprong >Assignee: Shane Knapp >Priority: Major > > We want to check the types of the PySpark code. This requires mypy to be > installed on the CI. Can you do this [~shaneknapp]? > Related PR: [https://github.com/apache/spark/pull/29180] > You can install this using pip: [https://pypi.org/project/mypy/] Should be > similar to flake8 and sphinx. The latest version is ok! Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
