[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user WeichenXu123 commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-222069163 @jkbradley --modules='pyspark-ml' will run a bunch of test in pyspark sub directory parallel, not single python file. and my purpose is add a way to debug single python file code. Because the python test code should launch with bin/pyspark so it is not convenient to directly debug in IDE. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221993639 To test a single subpackage, you can already use ```---modules='pyspark-ml'``` etc., so the ```---single-test``` arg may not be necessary. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user WeichenXu123 commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221210606 @davies How do you think about it ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221209434 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221209436 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59194/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221209336 **[Test build #59194 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59194/consoleFull)** for PR 13275 at commit [`52f1da9`](https://github.com/apache/spark/commit/52f1da93dc08fb831331436ac72163fb1458e0da). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221203888 **[Test build #59194 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59194/consoleFull)** for PR 13275 at commit [`52f1da9`](https://github.com/apache/spark/commit/52f1da93dc08fb831331436ac72163fb1458e0da). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user WeichenXu123 commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221203315 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221202318 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221202299 **[Test build #59193 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59193/consoleFull)** for PR 13275 at commit [`52f1da9`](https://github.com/apache/spark/commit/52f1da93dc08fb831331436ac72163fb1458e0da). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221202319 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59193/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221200497 **[Test build #59193 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59193/consoleFull)** for PR 13275 at commit [`52f1da9`](https://github.com/apache/spark/commit/52f1da93dc08fb831331436ac72163fb1458e0da). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221190940 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221190944 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59191/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221190936 **[Test build #59191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59191/consoleFull)** for PR 13275 at commit [`db1e37d`](https://github.com/apache/spark/commit/db1e37dccdd6570eceed2926ee3c7b391a21612c). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221190790 **[Test build #59191 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59191/consoleFull)** for PR 13275 at commit [`db1e37d`](https://github.com/apache/spark/commit/db1e37dccdd6570eceed2926ee3c7b391a21612c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221189017 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221189010 **[Test build #59190 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59190/consoleFull)** for PR 13275 at commit [`60e0e96`](https://github.com/apache/spark/commit/60e0e96bd7873b169854b361f90d1ef97378a22e). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221189020 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59190/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221188838 **[Test build #59190 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59190/consoleFull)** for PR 13275 at commit [`60e0e96`](https://github.com/apache/spark/commit/60e0e96bd7873b169854b361f90d1ef97378a22e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221187061 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221187063 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59189/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221186948 **[Test build #59189 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59189/consoleFull)** for PR 13275 at commit [`e694d10`](https://github.com/apache/spark/commit/e694d10f27d2da9607c6f52bb7b2a2e2fd50e0fd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13275#issuecomment-221187059 **[Test build #59189 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59189/consoleFull)** for PR 13275 at commit [`e694d10`](https://github.com/apache/spark/commit/e694d10f27d2da9607c6f52bb7b2a2e2fd50e0fd). * This patch **fails RAT tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15499][PySpark][Tests] Add python tests...
GitHub user WeichenXu123 opened a pull request: https://github.com/apache/spark/pull/13275 [SPARK-15499][PySpark][Tests] Add python testsuite with remote debug and single test parameter to help developer debug code easier ## What changes were proposed in this pull request? To python/run-tests.py script, I add the following parameters: --single-test=SINGLE_TEST specify a python module to run single python test. --debug-server=DEBUG_SERVER debug server host, only used in single test. --debug-port=DEBUG_PORT debug server port, only used in single test. Now, for example, I want to debug only pyspark.tests, I can use the following command: (first startup debug server in your python IDE such as pycharm or pydev, your IED on machine which has host MY_DEV_MACHINE and using debug port 5678, and make sure the machine running test can connect to MY_DEV_MACHINE) python/run-tests --python-executables=python2.7 --single-test=pyspark.tests --debug-server=MY_DEV_MACHINE --debug-port=5678 the parameter --single-test specify the single python testsuite you want to test, currently you can use the following value: ` pyspark.tests pyspark.ml.tests pyspark.mllib.tests pyspark.sql.tests pyspark.streaming.tests ` ## How was this patch tested? Existing test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/WeichenXu123/spark add_python_remote_debug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13275.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13275 commit 872ebdedc80a76b9bf064bda52af6074d16e7efc Author: WeichenXuDate: 2016-05-24T05:32:35Z add python test with single test and remote debug param commit e694d10f27d2da9607c6f52bb7b2a2e2fd50e0fd Author: WeichenXu Date: 2016-05-24T05:41:53Z remove useless code which used for debug by myself --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org