[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-06-01 Thread ssuchter
Github user ssuchter commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r192447027 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala --- @@ -0,0

[GitHub] spark issue #21479: [SPARK-23903][SQL] Add support for date extract

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21479 **[Test build #91396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91396/testReport)** for PR 21479 at commit

[GitHub] spark pull request #21479: [SPARK-23903][SQL] Add support for date extract

2018-06-01 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/21479 [SPARK-23903][SQL] Add support for date extract ## What changes were proposed in this pull request? Add support for date `extract`, supported field same as

[GitHub] spark issue #21448: [SPARK-24408][SQL][DOC] Move abs, bitwiseNOT, isnan, nan...

2018-06-01 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/21448 So, I personally don't have strong feelings about which groups these functions should be in, but just a reminder we should unify with PySpark with whatever our decision is. In PySpark

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21467 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91395/ Test FAILed. ---

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21467 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21467 **[Test build #91395 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91395/testReport)** for PR 21467 at commit

[GitHub] spark pull request #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop ite...

2018-06-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21467#discussion_r192437788 --- Diff: python/pyspark/sql/tests.py --- @@ -901,20 +901,43 @@ def __call__(self, x): self.assertEqual(return_type, f_.returnType)

[GitHub] spark pull request #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop ite...

2018-06-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21467#discussion_r192437180 --- Diff: python/pyspark/worker.py --- @@ -140,15 +139,20 @@ def read_single_udf(pickleSer, infile, eval_type): else:

[GitHub] spark pull request #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop ite...

2018-06-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21467#discussion_r192436670 --- Diff: python/pyspark/sql/tests.py --- @@ -901,20 +901,43 @@ def __call__(self, x): self.assertEqual(return_type, f_.returnType)

[GitHub] spark pull request #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop ite...

2018-06-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21467#discussion_r192436233 --- Diff: python/pyspark/sql/tests.py --- @@ -901,20 +901,43 @@ def __call__(self, x): self.assertEqual(return_type, f_.returnType)

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21467 **[Test build #91395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91395/testReport)** for PR 21467 at commit

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21092 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3623/ ---

[GitHub] spark pull request #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP P...

2018-06-01 Thread DazhuangSu
Github user DazhuangSu commented on a diff in the pull request: https://github.com/apache/spark/pull/19691#discussion_r192425378 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -515,28 +515,66 @@ case class

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21092 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3623/ ---

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21092 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3761/

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21092 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop ite...

2018-06-01 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21467#discussion_r192424377 --- Diff: python/pyspark/tests.py --- @@ -1303,11 +1303,13 @@ def stopit(*x): self.assertRaises(Py4JJavaError,

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread e-dorigatti
Github user e-dorigatti commented on the issue: https://github.com/apache/spark/pull/21467 @viirya for RDDs, we cannot check for stopiterations in the executor because this bug is introduced in the driver by rewriting the methods. consider rdd.map: ``` def map(self, f,

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21092 **[Test build #91394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91394/testReport)** for PR 21092 at commit

[GitHub] spark pull request #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP P...

2018-06-01 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19691#discussion_r192416800 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -515,28 +515,66 @@ case class AlterTableRenamePartitionCommand(

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread e-dorigatti
Github user e-dorigatti commented on the issue: https://github.com/apache/spark/pull/21467 `PyArrow` again --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP PARTITIO...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19691 **[Test build #91393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91393/testReport)** for PR 19691 at commit

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21467 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91392/ Test FAILed. ---

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21467 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21467 **[Test build #91392 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91392/testReport)** for PR 21467 at commit

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21467 **[Test build #91392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91392/testReport)** for PR 21467 at commit

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91388/ Test PASSed. ---

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21477 **[Test build #91388 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91388/testReport)** for PR 21477 at commit

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21467 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91391/ Test FAILed. ---

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21467 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21467 **[Test build #91391 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91391/testReport)** for PR 21467 at commit

[GitHub] spark pull request #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP P...

2018-06-01 Thread DazhuangSu
Github user DazhuangSu commented on a diff in the pull request: https://github.com/apache/spark/pull/19691#discussion_r192399827 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -515,28 +515,66 @@ case class

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21467 **[Test build #91391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91391/testReport)** for PR 21467 at commit

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread e-dorigatti
Github user e-dorigatti commented on the issue: https://github.com/apache/spark/pull/21467 messed up, of course :) fixing --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21370: [SPARK-24215][PySpark] Implement _repr_html_ for datafra...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21370 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21370: [SPARK-24215][PySpark] Implement _repr_html_ for datafra...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21370 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91389/ Test PASSed. ---

[GitHub] spark issue #21370: [SPARK-24215][PySpark] Implement _repr_html_ for datafra...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21370 **[Test build #91389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91389/testReport)** for PR 21370 at commit

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91384/ Test FAILed. ---

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21477 **[Test build #91384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91384/testReport)** for PR 21477 at commit

[GitHub] spark pull request #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP P...

2018-06-01 Thread DazhuangSu
Github user DazhuangSu commented on a diff in the pull request: https://github.com/apache/spark/pull/19691#discussion_r192388998 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -515,28 +515,66 @@ case class

[GitHub] spark issue #21061: [SPARK-23914][SQL] Add array_union function

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21061 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91387/ Test PASSed. ---

[GitHub] spark issue #21061: [SPARK-23914][SQL] Add array_union function

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21061 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21061: [SPARK-23914][SQL] Add array_union function

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21061 **[Test build #91387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91387/testReport)** for PR 21061 at commit

[GitHub] spark issue #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit execution w...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21450 **[Test build #91390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91390/testReport)** for PR 21450 at commit

[GitHub] spark pull request #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit exec...

2018-06-01 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/21450#discussion_r192386271 --- Diff: launcher/src/test/java/org/apache/spark/launcher/SparkSubmitCommandBuilderSuite.java --- @@ -190,6 +194,23 @@ public void

[GitHub] spark pull request #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit exec...

2018-06-01 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/21450#discussion_r192385957 --- Diff: launcher/src/test/java/org/apache/spark/launcher/SparkSubmitCommandBuilderSuite.java --- @@ -344,8 +365,15 @@ private

[GitHub] spark pull request #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit exec...

2018-06-01 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/21450#discussion_r192385912 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java --- @@ -229,7 +238,7 @@ args.add(join(",",

[GitHub] spark pull request #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP P...

2018-06-01 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19691#discussion_r192382198 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -515,28 +515,66 @@ case class AlterTableRenamePartitionCommand(

[GitHub] spark pull request #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP P...

2018-06-01 Thread DazhuangSu
Github user DazhuangSu commented on a diff in the pull request: https://github.com/apache/spark/pull/19691#discussion_r192380851 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -515,28 +515,66 @@ case class

[GitHub] spark pull request #21472: [SPARK-24445][SQL] Schema in json format for from...

2018-06-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21472#discussion_r192380680 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -747,8 +748,13 @@ case class

[GitHub] spark pull request #21472: [SPARK-24445][SQL] Schema in json format for from...

2018-06-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21472#discussion_r192379878 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -747,8 +748,13 @@ case class

[GitHub] spark issue #21449: [SPARK-24385][SQL] Resolve self-join condition ambiguity...

2018-06-01 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21449 Thanks for your comment @cloud-fan. I understand your point. That is quite a tricky problem, since we should know probably also the "DAG" of the dataframes in order to take the right decision.

[GitHub] spark pull request #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP P...

2018-06-01 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19691#discussion_r192367934 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -515,28 +515,66 @@ case class AlterTableRenamePartitionCommand(

[GitHub] spark pull request #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP P...

2018-06-01 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19691#discussion_r192367608 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -515,28 +515,66 @@ case class AlterTableRenamePartitionCommand(

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-06-01 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/20697 @ssuchter I think minikube could run within docker with flag –vm-driver=none right? This way you dont need minikube on hosts. Just an idea havent tried might be wrong. ---

[GitHub] spark pull request #21472: [SPARK-24445][SQL] Schema in json format for from...

2018-06-01 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/21472#discussion_r192358871 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -747,8 +748,13 @@ case class StructsToJson(

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total size of states in HDFSBac...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21469 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91382/ Test PASSed. ---

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total size of states in HDFSBac...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21469 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total size of states in HDFSBac...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21469 **[Test build #91382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91382/testReport)** for PR 21469 at commit

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21467 Can we also move the stop iteration wrapper of rdd to executor? --- - To unsubscribe, e-mail:

[GitHub] spark issue #21370: [SPARK-24215][PySpark] Implement _repr_html_ for datafra...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21370 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3760/

[GitHub] spark issue #21370: [SPARK-24215][PySpark] Implement _repr_html_ for datafra...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21370 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3759/

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-01 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r192349637 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +352,62 @@ def show(self, n=20, truncate=True, vertical=False): else:

[GitHub] spark issue #21370: [SPARK-24215][PySpark] Implement _repr_html_ for datafra...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21370 **[Test build #91389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91389/testReport)** for PR 21370 at commit

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-01 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r192349210 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +352,70 @@ def show(self, n=20, truncate=True, vertical=False): else:

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-01 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r192349075 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -291,37 +310,30 @@ class Dataset[T] private[sql]( }

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-01 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r192349023 --- Diff: docs/configuration.md --- @@ -456,6 +456,29 @@ Apart from these, the following properties are also available, and may be useful

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-01 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r192349063 --- Diff: python/pyspark/sql/dataframe.py --- @@ -78,6 +78,7 @@ def __init__(self, jdf, sql_ctx): self.is_cached = False

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-01 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r192348972 --- Diff: docs/configuration.md --- @@ -456,6 +456,29 @@ Apart from these, the following properties are also available, and may be useful

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21477 **[Test build #91388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91388/testReport)** for PR 21477 at commit

[GitHub] spark pull request #21383: [SPARK-23754][Python] Re-raising StopIteration in...

2018-06-01 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21383#discussion_r192344759 --- Diff: python/pyspark/tests.py --- @@ -161,6 +161,37 @@ def gen_gs(N, step=1): self.assertEqual(k, len(vs))

[GitHub] spark issue #21061: [SPARK-23914][SQL] Add array_union function

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21061 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3758/

[GitHub] spark issue #21061: [SPARK-23914][SQL] Add array_union function

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21061 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21477 **[Test build #91386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91386/testReport)** for PR 21477 at commit

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91386/ Test FAILed. ---

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21472: [SPARK-24445][SQL] Schema in json format for from...

2018-06-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21472#discussion_r192342580 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -747,8 +748,13 @@ case class

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3757/

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21477 **[Test build #91385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91385/testReport)** for PR 21477 at commit

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91385/ Test FAILed. ---

[GitHub] spark issue #21061: [SPARK-23914][SQL] Add array_union function

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21061 **[Test build #91387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91387/testReport)** for PR 21061 at commit

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-06-01 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r192340073 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1882,3 +1882,311 @@ case class

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3756/

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21472: [SPARK-24445][SQL] Schema in json format for from...

2018-06-01 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/21472#discussion_r192339647 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -747,8 +748,13 @@ case class StructsToJson(

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21477 **[Test build #91386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91386/testReport)** for PR 21477 at commit

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21467 @e-dorigatti, can you maybe rebase and then start this with revert commit? That should make guys less confused. I would do squash commits first, revert it and then bring the squashed commit

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21477 **[Test build #91385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91385/testReport)** for PR 21477 at commit

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-06-01 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r192336364 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1882,3 +1882,311 @@ case class

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3755/

[GitHub] spark issue #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured Stream...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21477 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-01 Thread e-dorigatti
Github user e-dorigatti commented on the issue: https://github.com/apache/spark/pull/21467 @viirya we only want to reverd `udf.py` and the hack in `_get_argspec`. Did I miss anything there? --- - To unsubscribe,

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2018-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19602 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91383/ Test FAILed. ---

<    1   2   3   4   >