Hi everyone, For the last few months I've been working on static type annotations for PySpark. For those of you, who are not familiar with the idea, typing hints have been introduced by PEP 484 (https://www.python.org/dev/peps/pep-0484/) and further extended with PEP 526 (https://www.python.org/dev/peps/pep-0526/) with the main goal of providing information required for static analysis. Right now there a few tools which support typing hints, including Mypy (https://github.com/python/mypy) and PyCharm (https://www.jetbrains.com/help/pycharm/2017.1/type-hinting-in-pycharm.html). Type hints can be added using function annotations (https://www.python.org/dev/peps/pep-3107/, Python 3 only), docstrings, or source independent stub files (https://www.python.org/dev/peps/pep-0484/#stub-files). Typing is optional, gradual and has no runtime impact.
At this moment I've annotated majority of the API, including majority of
pyspark.sql and pyspark.ml. At this moment project is still rough around
the edges, and may result in both false positive and false negatives,
but I think it become mature enough to be useful in practice.
The current version is compatible only with Python 3, but it is
possible, with some limitations, to backport it to Python 2 (though it
is not on my todo list).
There is a number of possible benefits for PySpark users and developers:
* Static analysis can detect a number of common mistakes to prevent
runtime failures. Generic self is still fairly limited, so it is
more useful with DataFrames, SS and ML than RDD, DStreams or RDD.
* Annotations can be used for documenting complex signatures
(https://git.io/v95JN) including dependencies on arguments and value
(https://git.io/v95JA).
* Detecting possible bugs in Spark (SPARK-20631) .
* Showing API inconsistencies.
Roadmap
* Update the project to reflect Spark 2.2.
* Refine existing annotations.
If there will be enough interest I am happy to contribute this back to
Spark or submit to Typeshed (https://github.com/python/typeshed - this
would require a formal ASF approval, and since Typeshed doesn't provide
versioning, is probably not the best option in our case).
Further inforamtion:
* https://github.com/zero323/pyspark-stubs - GitHub repository
*
https://speakerdeck.com/marcobonzanini/static-type-analysis-for-robust-data-products-at-pydata-london-2017
- interesting presentation by Marco Bonzanini
--
Best,
Maciej
signature.asc
Description: OpenPGP digital signature
