[ https://issues.apache.org/jira/browse/SPARK-32797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17386524#comment-17386524 ]
Shane Knapp commented on SPARK-32797: ------------------------------------- i've temporarily disabled mypy on the workers: for x in `cat ~/txt/jenkins-workers.txt`; do ssh jenkins@${x} "mv /home/jenkins/anaconda2/envs/py36/bin/mypy /home/jenkins/anaconda2/envs/py36/bin/mypy.bak"; done the files that need fixing are here: {noformat} starting mypy test... mypy checks failed: python/pyspark/pandas/data_type_ops/categorical_ops.py:128: error: Function is missing a type annotation python/pyspark/mllib/tree.pyi:29: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/tree.pyi:38: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:34: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:42: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:48: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:54: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:76: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:124: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:165: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/clustering.pyi:45: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/clustering.pyi:72: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/classification.pyi:39: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/classification.pyi:52: error: Overloaded function signatures 1 and 2 overlap with incompatible return types {noformat} could someone create a PR that addresses these, and once that's done i'll re-enable mypy on the workers and we'll coordinate a quick test. > Install mypy on the Jenkins CI workers > -------------------------------------- > > Key: SPARK-32797 > URL: https://issues.apache.org/jira/browse/SPARK-32797 > Project: Spark > Issue Type: Improvement > Components: jenkins, PySpark > Affects Versions: 3.1.0 > Reporter: Fokko Driesprong > Assignee: Shane Knapp > Priority: Major > Fix For: 3.3.0 > > > We want to check the types of the PySpark code. This requires mypy to be > installed on the CI. Can you do this [~shaneknapp]? > Related PR: [https://github.com/apache/spark/pull/29180] > You can install this using pip: [https://pypi.org/project/mypy/] Should be > similar to flake8 and sphinx. The latest version is ok! Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org