[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2016-01-05 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-169147628 > Merged into master and 1.6 branch, could you create a JIRA to clean this once the bugs are fixed in py4j (and released)? Created sub tasks in https://issues.a

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2016-01-05 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-169144547 Merged into master and 1.6 branch, could you create a JIRA to clean this once the bugs are fixed in py4j (and released)? --- If your project is set up for it, you can r

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2016-01-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10514 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2016-01-05 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-169141854 > do we need to merge this into 1.6? Yes since it affects all people using PySpark Streaming checkpoint. --- If your project is set up for it, you can reply to

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2016-01-05 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-169133324 LGTM, do we need to merge this into 1.6? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168139026 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168139002 **[Test build #48544 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48544/consoleFull)** for PR 10514 at commit [`340b034`](https://g

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168139028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168134339 **[Test build #48544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48544/consoleFull)** for PR 10514 at commit [`340b034`](https://gi

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168131624 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168114334 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168114335 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168114317 **[Test build #48512 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48512/consoleFull)** for PR 10514 at commit [`340b034`](https://g

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168107205 Also ping @tdas since you wrote the `getOrCreate` method. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168107024 I submitted a new commit to fix a similar issue when restarting from checkpoint. Unlike Scala, PySpark reads the checkpoint twice. Then the `PythonProxyHandle

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168106817 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168106813 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168106453 **[Test build #48523 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48523/consoleFull)** for PR 10514 at commit [`340b034`](https://g

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168100443 **[Test build #48523 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48523/consoleFull)** for PR 10514 at commit [`340b034`](https://gi

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168099470 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168091409 **[Test build #48512 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48512/consoleFull)** for PR 10514 at commit [`340b034`](https://gi

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168076649 Not sure when py4j will release the next version to fix this one. Looks py4j doesn't release frequently. --- If your project is set up for it, you can reply to this em

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168076075 Is this targeted for 1.6, or we just wait this to be fixed in py4j? The current change looks over complicated to me. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168075619 @davies could you take another look? Actually, `cls._transformerSerializer`'s fields need to be updated with the parameters. So my previous approach doesn't work. I up

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168069856 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168069774 **[Test build #48506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48506/consoleFull)** for PR 10514 at commit [`3c3e164`](https://g

[GitHub] spark pull request: [SPARK-12511][PySpark][Streaming]Make sure Pyt...

2015-12-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10514#issuecomment-168069855 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec