[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21427 cc @rxin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91134/ Test PASSed. ---

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 **[Test build #91134 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91134/testReport)** for PR 21366 at commit

[GitHub] spark pull request #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF shou...

2018-05-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21427#discussion_r190793613 --- Diff: python/pyspark/sql/tests.py --- @@ -4931,6 +4931,33 @@ def foo3(key, pdf): expected4 = udf3.func((), pdf)

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21426 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3574/

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21426 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3573/

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21426 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21426 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21427 Also, I really think we should mark this feature as experimental. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21426 **[Test build #91140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91140/testReport)** for PR 21426 at commit

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21427 Yea agree with not backporting and agree with configuration. Thing is, the configuration is inaccessible in worker.py side. That's why I was hesitant. The safest way is just to target 3.0.0 but

[GitHub] spark pull request #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/Pa...

2018-05-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21389#discussion_r190791115 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala --- @@ -0,0 +1,53 @@ +/* + * Licensed to the

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21426 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21426 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3572/

[GitHub] spark pull request #21331: [SPARK-24276][SQL] Order of literals in IN should...

2018-05-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21331#discussion_r190790891 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Canonicalize.scala --- @@ -85,6 +86,9 @@ object Canonicalize {

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21428 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91135/ Test PASSed. ---

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21428 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21410: [SPARK-24366][SQL] Improving of error messages for type ...

2018-05-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21410 LGTM except one minor comment. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21426 **[Test build #91139 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91139/testReport)** for PR 21426 at commit

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21428 **[Test build #91135 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91135/testReport)** for PR 21428 at commit

[GitHub] spark pull request #21410: [SPARK-24366][SQL] Improving of error messages fo...

2018-05-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21410#discussion_r190790254 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala --- @@ -309,6 +322,9 @@ object CatalystTypeConverters

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21426 I tested: submit with yarn client: .py local submit with yarn client: .py remote submit with standalone client: .py local submit with standalone client: .py remote

[GitHub] spark pull request #21415: [SPARK-24244][SPARK-24368][SQL] Passing only requ...

2018-05-24 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21415 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91133/ Test FAILed. ---

[GitHub] spark issue #21415: [SPARK-24244][SPARK-24368][SQL] Passing only required co...

2018-05-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21415 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 **[Test build #91133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91133/testReport)** for PR 21366 at commit

[GitHub] spark issue #21351: [SPARK-24002][SQL][BACKPORT-2.3] Task not serializable c...

2018-05-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21351 @imarios Please check the dev mailing list. It is being voted. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21427 Do not backport this to 2.3. This is a behavior change. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21427 How about making it configurable? Users can choose either resolve by names or resolve by positions. It is hard to say which one is right. If the names do not match when users want to resolve by

[GitHub] spark issue #21346: [SPARK-6237][NETWORK] Network-layer changes to allow str...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21346 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3571/

[GitHub] spark issue #21346: [SPARK-6237][NETWORK] Network-layer changes to allow str...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21346 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21346: [SPARK-6237][NETWORK] Network-layer changes to allow str...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21346 **[Test build #91138 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91138/testReport)** for PR 21346 at commit

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21428 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91132/ Test FAILed. ---

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21428 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21428 **[Test build #91132 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91132/testReport)** for PR 21428 at commit

[GitHub] spark pull request #21420: [SPARK-24377][Spark Submit] make --py-files work ...

2018-05-24 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/21420#discussion_r190783462 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -430,18 +430,15 @@ private[spark] class SparkSubmit extends Logging {

[GitHub] spark pull request #21420: [SPARK-24377][Spark Submit] make --py-files work ...

2018-05-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21420#discussion_r190783213 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -430,18 +430,15 @@ private[spark] class SparkSubmit extends Logging {

[GitHub] spark issue #21400: [SPARK-24351][SS]offsetLog/commitLog purge thresholdBatc...

2018-05-24 Thread ivoson
Github user ivoson commented on the issue: https://github.com/apache/spark/pull/21400 @jose-torres thanks for reply. I will try to add a unit test for this. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #21411: [SPARK-24367][SQL]Parquet: use JOB_SUMMARY_LEVEL ...

2018-05-24 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21411 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21426 I haven't tried yet but I believe it has since It downloads into local. It has the assumption that the file is local within deploy.PythonRunner side too. Will check for doubly sure. ---

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21428 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91131/ Test FAILed. ---

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21428 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21428 **[Test build #91131 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91131/testReport)** for PR 21428 at commit

[GitHub] spark issue #21411: [SPARK-24367][SQL]Parquet: use JOB_SUMMARY_LEVEL instead...

2018-05-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21411 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-24 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21426 Did you try remote py files, does it have similar issue? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #21411: [SPARK-24367][SQL]Parquet: use JOB_SUMMARY_LEVEL instead...

2018-05-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21411 I remember this summary file is disabled by default anyway. I think it's fine to just get rid of warnings. --- - To

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91130/ Test PASSed. ---

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21427 Just for clarification, I am okay @BryanCutler if you feel in this way too. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-24 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/21427 I do think the current default behavior might be confusing to users and hard to debug. I have also received similar complaints. I think at the very least, we should make sure when

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 **[Test build #91130 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91130/testReport)** for PR 21366 at commit

[GitHub] spark issue #21422: [Spark-24376][doc]Summary:compiling spark with scala-2.1...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21422 **[Test build #91137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91137/testReport)** for PR 21422 at commit

[GitHub] spark issue #21422: [Spark-24376][doc]Summary:compiling spark with scala-2.1...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21422 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91137/ Test PASSed. ---

[GitHub] spark issue #21422: [Spark-24376][doc]Summary:compiling spark with scala-2.1...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21422 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21346: [SPARK-6237][NETWORK] Network-layer changes to allow str...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21346 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3570/

[GitHub] spark issue #21346: [SPARK-6237][NETWORK] Network-layer changes to allow str...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21346 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21346: [SPARK-6237][NETWORK] Network-layer changes to allow str...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21346 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91136/ Test FAILed. ---

[GitHub] spark issue #21346: [SPARK-6237][NETWORK] Network-layer changes to allow str...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21346 **[Test build #91136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91136/testReport)** for PR 21346 at commit

[GitHub] spark issue #21346: [SPARK-6237][NETWORK] Network-layer changes to allow str...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21346 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files...

2018-05-24 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/21426#discussion_r190778192 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -372,8 +376,27 @@ private[spark] class SparkSubmit extends Logging {

[GitHub] spark issue #21422: [Spark-24376][doc]Summary:compiling spark with scala-2.1...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21422 **[Test build #91137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91137/testReport)** for PR 21422 at commit

[GitHub] spark pull request #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files...

2018-05-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21426#discussion_r190778033 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -372,8 +376,27 @@ private[spark] class SparkSubmit extends Logging {

[GitHub] spark issue #21422: [Spark-24376][doc]Summary:compiling spark with scala-2.1...

2018-05-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21422 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21427 H .. I got that this one is more preferable and I think we haven't got a discussion for this so far if I remember this correctly. Do you feel strongly about this @icexelloss and

[GitHub] spark issue #21346: [SPARK-6237][NETWORK] Network-layer changes to allow str...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21346 **[Test build #91136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91136/testReport)** for PR 21346 at commit

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91129/ Test PASSed. ---

[GitHub] spark issue #21390: [SPARK-24340][Core] Clean up non-shuffle disk block mana...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21390 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21390: [SPARK-24340][Core] Clean up non-shuffle disk block mana...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21390 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91128/ Test PASSed. ---

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 **[Test build #91129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91129/testReport)** for PR 21366 at commit

[GitHub] spark issue #21390: [SPARK-24340][Core] Clean up non-shuffle disk block mana...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21390 **[Test build #91128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91128/testReport)** for PR 21390 at commit

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3446/ ---

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3446/ ---

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3569/

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21428 **[Test build #91135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91135/testReport)** for PR 21428 at commit

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3568/

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 **[Test build #91134 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91134/testReport)** for PR 21366 at commit

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-24 Thread mccheah
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21366 Alright, this should be good for review now, with all cleanups and appropriate test coverage in place. Please take a look. Thanks! ---

[GitHub] spark pull request #21366: [SPARK-24248][K8S][WIP] Use the Kubernetes API to...

2018-05-24 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r190769478 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsPollingEventSource.scala --- @@ -0,0 +1,58

[GitHub] spark issue #21366: [SPARK-24248][K8S][WIP] Use the Kubernetes API to popula...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 **[Test build #91133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91133/testReport)** for PR 21366 at commit

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21428 **[Test build #91132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91132/testReport)** for PR 21428 at commit

[GitHub] spark issue #21415: [SPARK-24244][SPARK-24368][SQL] Passing only required co...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21415 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21415: [SPARK-24244][SPARK-24368][SQL] Passing only required co...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21415 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91126/ Test PASSed. ---

[GitHub] spark issue #21415: [SPARK-24244][SPARK-24368][SQL] Passing only required co...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21415 **[Test build #91126 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91126/testReport)** for PR 21415 at commit

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21428 **[Test build #91131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91131/testReport)** for PR 21428 at commit

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle write RDD...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21428 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21428: [SPARK-24235][SS] Implement continuous shuffle wr...

2018-05-24 Thread jose-torres
GitHub user jose-torres opened a pull request: https://github.com/apache/spark/pull/21428 [SPARK-24235][SS] Implement continuous shuffle write RDD for single reader partition. ## What changes were proposed in this pull request? Implement continuous shuffle write RDD for a

[GitHub] spark pull request #21385: [SPARK-24234][SS] Support multiple row writers in...

2018-05-24 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21385 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-24 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/21427 I first glance, I thought this issue was slightly different than https://issues.apache.org/jira/browse/SPARK-23929, but yeah it seems to be the same. After reading through that discussion, I

[GitHub] spark pull request #21366: [SPARK-24248][K8S][WIP] Use the Kubernetes API to...

2018-05-24 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r190762965 --- Diff: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/DeterministicExecutorPodsEventQueue.scala --- @@ -0,0

[GitHub] spark pull request #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files...

2018-05-24 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21426#discussion_r190761869 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -372,8 +376,27 @@ private[spark] class SparkSubmit extends Logging {

[GitHub] spark issue #21366: [SPARK-24248][K8S][WIP] Use the Kubernetes API to popula...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21366: [SPARK-24248][K8S][WIP] Use the Kubernetes API to popula...

2018-05-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91123/ Test PASSed. ---

[GitHub] spark issue #21366: [SPARK-24248][K8S][WIP] Use the Kubernetes API to popula...

2018-05-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 **[Test build #91123 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91123/testReport)** for PR 21366 at commit

[GitHub] spark pull request #21383: [SPARK-23754][Python] Re-raising StopIteration in...

2018-05-24 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21383#discussion_r190761062 --- Diff: python/pyspark/sql/tests.py --- @@ -900,6 +900,17 @@ def __call__(self, x): self.assertEqual(f, f_.func)

[GitHub] spark pull request #21416: [SPARK-24371] [SQL] Added isinSet in DataFrame AP...

2018-05-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/21416#discussion_r190759741 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -220,6 +219,7 @@ object OptimizeIn extends

  1   2   3   4   5   6   >