[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/22963 re https://github.com/apache/spark/pull/22963#issuecomment-437133365 i checked, and the only one we can seemingly download independently is pycodestyle. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22963 I also agree with @srowen's (https://github.com/apache/spark/pull/22963#issuecomment-437133365) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22963 OMG, I don't know why I missed these comments. I will read it tomorrow (now it's 6 am and I could get sleep .. ) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/22963 > pydocstyle tests passed w/o issue btw: > https://amplab.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/134/consoleFull > > this is on ubuntu w/python 3.5, flake8 3.6.0, pydocstyle 3.0.0 and pycodestyle 2.4.0. > @shaneknapp - That is expected, as for now, dev/tox.ini has all checks in ignore section for pydocstyle. thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/22963 orthogonal to this PR, but just as an FYI: https://github.com/apache/spark/pull/22994 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/22963 > I don't have the context here to have a strong opinion, but, it seems like we should make the test env setup self-contained if possible, to avoid dependencies on and maintenance of the build env. After all others need to run these tests too. To that end, downloading and installing particular versions seems reasonable (if they're not already installed at the right version?) Is that hard? it *should* be reasonable to download a binary as we do for pycodestyle. i'll look around and see what i can find. > > Whatever makes this more robust and needs less maintenance from you sounds good. I think you're welcome to clean up the script as you like too. i will absolutely be taking a long, hard look at lint-python... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/22963 pydocstyle tests passed w/o issue btw: https://amplab.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/134/consoleFull this is on ubuntu w/python 3.5, flake8 3.6.0, pydocstyle 3.0.0 and pycodestyle 2.4.0. i also updated all of the centos jenkins workers to have flake8 3.6.0 and pycodestyle 2.4.0. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22963 I don't have the context here to have a strong opinion, but, it seems like we should make the test env setup self-contained if possible, to avoid dependencies on and maintenance of the build env. After all others need to run these tests too. To that end, downloading and installing particular versions seems reasonable (if they're not already installed at the right version?) Is that hard? Whatever makes this more robust and needs less maintenance from you sounds good. I think you're welcome to clean up the script as you like too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22963
interesting. from the flake8 webpage:
```
It is very important to install Flake8 on the correct version of Python for
your needs. If you want Flake8 to properly parse new language features in
Python 3.5 (for example), you need it to be installed on 3.5 for Flake8 to
understand those features. In many ways, Flake8 is tied to the version of
Python on which it runs.
```
we're testing flake8 w/python 3.4.5 on the centos workers, and soon to be
python 3.5 everywhere. this means that flake8 probably hasn't been behaving
properly since the get-go.
also, this PR sets the flake8 version to 3.6, which means we should be
testing against python3.6... i was only planning on bumping the python version
from 3.4 -> 3.5, however.
what's *also* confusing is that there are version of flake8 3.6.0 for
python2.7, 3.6 and 3.7.
i think we have some serious, and heretofore unknown, version
incompatibilities in our python testing environment.
what i'm going to do/test:
* pin flake8 to version 3.5.0, and the python2.7 testing environment.
* pin pycodestyle to 3.6.0, again in the python2.7 testing environment.
* leave sphinx-build in the python 3.{4,5} environment (as it always has
been)
* install pydocstyle in a test env and see what breaks
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/22963 ok, i missed that previous comment about flake8 and pycodestyle version incompatibility. since we're running flake8 3.50, i agree that this could be causing problems w/pycodestyle running properly: ``` flake8 3.5.0 has requirement pycodestyle<2.4.0,>=2.0.0, but you'll have pycodestyle 2.4.0 which is incompatible. ``` i'll bump flake8 to 3.6 on all of the workers, and then bump pycodestyle to 2.4.0 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/22963 sorry to jump in late on this, but i just wanted to check in on some of this stuff... `dev/lint-python` is a nightmare and i just wanted to discuss a couple of things: 1) the script downloads pycodestyle if it's not the right version. however, flake8, pycodestyle and sphinx are all installed and managed by the build system (see versions @ the end of this comment). i'll update all of the workers (and the ansible) to install the right version(s) of things as found in the script today. 1) in addition to (1), i'd like to do one of two things: either remove the pycodestyle download step, OR add installation steps for flake8, pydocstyle and sphinx. 2) we currently don't have pydocstyle installed on any of the workers. i am more than happy to install it immediately, but am concerned about build breakages as we've never tested this before. 3) holy crap `dev/lint-python` makes my eyes bleed! currently installed packages + versions: ``` -bash-4.1$ which pycodestyle && pycodestyle --version /home/anaconda/bin/pycodestyle 2.3.1 -bash-4.1$ which flake8 && flake8 --version /home/anaconda/bin/flake8 3.5.0 (mccabe: 0.6.1, pycodestyle: 2.3.1, pyflakes: 1.6.0) CPython 2.7.13 on Linux -bash-4.1$ which sphinx-build && sphinx-build --version /home/anaconda/envs/py3k/bin/sphinx-build Sphinx (sphinx-build) 1.2.3 ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22963 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22963 Thanks, @srowen and @dongjoon-hyun. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98549/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22963 **[Test build #98549 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98549/testReport)** for PR 22963 at commit [`802a9d4`](https://github.com/apache/spark/commit/802a9d4118e09dc255e65f023eab3f1fb5856e93). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22963 cc @srowen, @holdenk and @rekhajoshm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22963 **[Test build #98549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98549/testReport)** for PR 22963 at commit [`802a9d4`](https://github.com/apache/spark/commit/802a9d4118e09dc255e65f023eab3f1fb5856e93). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22963 **[Test build #98548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98548/testReport)** for PR 22963 at commit [`bef4190`](https://github.com/apache/spark/commit/bef419006459cf612c87ce60a582a58c581bc75a). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98548/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22963 **[Test build #98548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98548/testReport)** for PR 22963 at commit [`bef4190`](https://github.com/apache/spark/commit/bef419006459cf612c87ce60a582a58c581bc75a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4812/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22963 **[Test build #98547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98547/testReport)** for PR 22963 at commit [`a7192a3`](https://github.com/apache/spark/commit/a7192a3550110f3bfb64d4f559d909156fc72a44). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98547/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22963 **[Test build #98547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98547/testReport)** for PR 22963 at commit [`a7192a3`](https://github.com/apache/spark/commit/a7192a3550110f3bfb64d4f559d909156fc72a44). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4811/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22963 **[Test build #98546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98546/testReport)** for PR 22963 at commit [`b7f0e2f`](https://github.com/apache/spark/commit/b7f0e2f6bf4236a723756ca8a4d602b9a9221e88). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98546/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22963 **[Test build #98546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98546/testReport)** for PR 22963 at commit [`b7f0e2f`](https://github.com/apache/spark/commit/b7f0e2f6bf4236a723756ca8a4d602b9a9221e88). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4810/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98545/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22963 **[Test build #98545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98545/testReport)** for PR 22963 at commit [`e597243`](https://github.com/apache/spark/commit/e59724316ef5a37b0c9b8cb3311c4867d0e5e7af). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22963 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22963 **[Test build #98545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98545/testReport)** for PR 22963 at commit [`e597243`](https://github.com/apache/spark/commit/e59724316ef5a37b0c9b8cb3311c4867d0e5e7af). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
