[
https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427888#comment-17427888
]
Maciej Szymkiewicz commented on SPARK-36989:
--------------------------------------------
Currently I am working on [some
fixes|https://github.com/typeddjango/pytest-mypy-plugins/commits?author=zero323]
to
[typeddjango/pytest-mypy-plugins|https://github.com/typeddjango/pytest-mypy-plugins]
and I hope it will allow us to bring data test to Spark, without depending on
internal mypy testing suite (which, adding to being internal, requires rather
specific project layout).
> Migrate type hint data tests
> ----------------------------
>
> Key: SPARK-36989
> URL: https://issues.apache.org/jira/browse/SPARK-36989
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 3.3.0
> Reporter: Maciej Szymkiewicz
> Priority: Major
>
> Before the migration, {{pyspark-stubs}} contained a set of data tests,
> modeled after, and using internal test utilities, of mypy.
> These were omitted during the migration for a few reasons:
> * Simplicity.
> * Relative slowness.
> * Dependence on non public API.
>
> Data tests are useful for a number of reasons:
>
> * Improve test coverage for type hints.
> * Checking if type checkers infer expected types.
> * Checking if type checkers reject incorrect code.
> * Detecting unusual errors with code that otherwise type checks,
>
> Especially, the last two functions are not fulfilled by simple validation of
> existing codebase.
>
> Data tests are not required for all annotations and can be restricted to code
> that has high possibility of failure:
> * Complex overloaded signatures.
> * Complex generics.
> * Generic {{self}} annotations
> * Code containing {{type: ignore}}
> The biggest risk, is that output matchers have to be updated when signature
> changes and / or mypy output changes.
> Example of problem detected with data tests can be found in SPARK-36894 PR
> ([https://github.com/apache/spark/pull/34146]).
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
