Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14180 Hello. Been a long time, it probably needs a full rework. Maybe we need to take a step back and have a talk between several person interested in this feature to see what is the more suitable for the Spark project. I work a lot on Python packaging nowdays, so I have a pretty good idea on different distribution solutions we have for python (anaconda, pip/virtualenv, now Pipfile), and not only barely generating a python package and throwing it in the wild, I mean ensuring my package work in the targetted environment: pyexecutable is also a solution eventhough it is more complex, wheelhouse + some tricks might also do the job for Spark. Ultimately, the goal is to have something cool and easy to use for PySpark users willing to distribute any kind of work without having to ask the IT guys to install this numpy version on the cluster.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org