[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19813 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19752 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19813 **[Test build #84207 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84207/testReport)** for PR 19813 at commit [`9f848be`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19607 **[Test build #84204 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84204/testReport)** for PR 19607 at commit [`f92eae3`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19607 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19607 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84204/ Test FAILed. ---

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19813 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19752 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84206/ Test FAILed. ---

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19813 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84207/ Test FAILed. ---

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19607 **[Test build #84205 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84205/testReport)** for PR 19607 at commit [`40a9735`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19607 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19752 **[Test build #84206 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84206/testReport)** for PR 19752 at commit [`f4c7896`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11994 **[Test build #84209 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84209/testReport)** for PR 11994 at commit [`e94def7`](https://github.com/apache/spark/commit/e9

[GitHub] spark pull request #19823: [SPARK-22601][SQL] Data load is getting displayed...

2017-11-27 Thread gczsjdy
Github user gczsjdy commented on a diff in the pull request: https://github.com/apache/spark/pull/19823#discussion_r153127637 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -2624,7 +2624,13 @@ class SQLQuerySuite extends QueryTest with SharedSQLC

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19607 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19813 **[Test build #84208 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84208/testReport)** for PR 19813 at commit [`9f848be`](https://github.com/apache/spark/commit/9f

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19607 **[Test build #84210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84210/testReport)** for PR 19607 at commit [`40a9735`](https://github.com/apache/spark/commit/40

[GitHub] spark pull request #19792: [SPARK-22566][PYTHON] Better error message for `_...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19792#discussion_r153130303 --- Diff: python/pyspark/sql/types.py --- @@ -1108,19 +1109,23 @@ def _has_nulltype(dt): return isinstance(dt, NullType) -de

[GitHub] spark pull request #19757: [SPARK-22529] [SQL] Relation stats should be cons...

2017-11-27 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19757#discussion_r153130613 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala --- @@ -366,10 +366,16 @@ case class CatalogStatistics(

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19607 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84205/ Test FAILed. ---

[GitHub] spark pull request #19792: [SPARK-22566][PYTHON] Better error message for `_...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19792#discussion_r153130823 --- Diff: python/pyspark/sql/types.py --- @@ -1108,19 +1109,23 @@ def _has_nulltype(dt): return isinstance(dt, NullType) -de

[GitHub] spark pull request #19823: [SPARK-22601][SQL] Data load is getting displayed...

2017-11-27 Thread gczsjdy
Github user gczsjdy commented on a diff in the pull request: https://github.com/apache/spark/pull/19823#discussion_r153131202 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -341,6 +341,12 @@ case class LoadDataCommand( } else

[GitHub] spark pull request #19793: [SPARK-22574] [Mesos] [Submit] Check submission r...

2017-11-27 Thread Gschiavon
Github user Gschiavon commented on a diff in the pull request: https://github.com/apache/spark/pull/19793#discussion_r153131125 --- Diff: core/src/test/scala/org/apache/spark/deploy/rest/SubmitRestProtocolSuite.scala --- @@ -86,6 +86,8 @@ class SubmitRestProtocolSuite extends Spar

[GitHub] spark pull request #19792: [SPARK-22566][PYTHON] Better error message for `_...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19792#discussion_r153132348 --- Diff: python/pyspark/sql/types.py --- @@ -1108,19 +1109,23 @@ def _has_nulltype(dt): return isinstance(dt, NullType) -de

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19813 @cloud-fan @kiszk `ctx.currentVars` and `ctx.INPUT_ROW` are not the only sources for expression evaluation under wholestage codegen. There are also eliminated subexpressions and the input rows and va

[GitHub] spark pull request #19823: [SPARK-22601][SQL] Data load is getting displayed...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19823#discussion_r153135417 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -341,6 +341,12 @@ case class LoadDataCommand( }

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19813 **[Test build #84211 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84211/testReport)** for PR 19813 at commit [`57b1add`](https://github.com/apache/spark/commit/57

[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...

2017-11-27 Thread gaborgsomogyi
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19814 Seems like the job died because of jvm internal issue: > *** glibc detected *** /usr/java/jdk1.8.0_60/bin/java: double free or corruption (out): 0x000100038720 *** Is it p

[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19814 Yup, seems unrelated. It's fine. Let me back soon with another look. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.

[GitHub] spark pull request #19792: [SPARK-22566][PYTHON] Better error message for `_...

2017-11-27 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19792#discussion_r153137292 --- Diff: python/pyspark/sql/types.py --- @@ -1108,19 +1109,23 @@ def _has_nulltype(dt): return isinstance(dt, NullType) -def _me

[GitHub] spark pull request #19792: [SPARK-22566][PYTHON] Better error message for `_...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19792#discussion_r153138373 --- Diff: python/pyspark/sql/types.py --- @@ -1108,19 +1109,23 @@ def _has_nulltype(dt): return isinstance(dt, NullType) -de

[GitHub] spark pull request #19717: [SPARK-18278] [Submission] Spark on Kubernetes - ...

2017-11-27 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r153131921 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/steps/BaseDriverConfigurationStep.scala --- @@ -0,0 +1,161 @@

[GitHub] spark pull request #19717: [SPARK-18278] [Submission] Spark on Kubernetes - ...

2017-11-27 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r153123413 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigurationStepsOrchestrator.scala --- @@ -0,0 +1,84

[GitHub] spark pull request #19717: [SPARK-18278] [Submission] Spark on Kubernetes - ...

2017-11-27 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r153098283 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/Client.scala --- @@ -0,0 +1,219 @@ +/* + * Licensed t

[GitHub] spark pull request #19717: [SPARK-18278] [Submission] Spark on Kubernetes - ...

2017-11-27 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r153132562 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/steps/DriverKubernetesCredentialsStep.scala --- @@ -0,0 +1,24

[GitHub] spark pull request #19717: [SPARK-18278] [Submission] Spark on Kubernetes - ...

2017-11-27 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r153093334 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/Client.scala --- @@ -0,0 +1,219 @@ +/* + * Licensed t

[GitHub] spark pull request #19717: [SPARK-18278] [Submission] Spark on Kubernetes - ...

2017-11-27 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r153092633 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -702,6 +715,18 @@ object SparkSubmit extends CommandLineUtils with Logging {

[GitHub] spark pull request #19717: [SPARK-18278] [Submission] Spark on Kubernetes - ...

2017-11-27 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r153126963 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala --- @@ -0,0 +1,184 @@ +/*

[GitHub] spark pull request #19717: [SPARK-18278] [Submission] Spark on Kubernetes - ...

2017-11-27 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r153093170 --- Diff: resource-managers/kubernetes/core/pom.xml --- @@ -0,0 +1,102 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/

[GitHub] spark pull request #19717: [SPARK-18278] [Submission] Spark on Kubernetes - ...

2017-11-27 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r153122738 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -0,0 +1,173 @@ +/* + * Licensed to the A

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19752 **[Test build #84212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84212/testReport)** for PR 19752 at commit [`5adb513`](https://github.com/apache/spark/commit/5a

[GitHub] spark pull request #19821: [WIP][SPARK-22608][SQL] add new API to CodeGenera...

2017-11-27 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19821#discussion_r153140836 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -785,13 +785,36 @@ class CodegenContext {

[GitHub] spark issue #19388: [SPARK-22162] Executors and the driver should use consis...

2017-11-27 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/19388 I agree with @vanzin - this looks very complicated for a enforcing a fairly simple constraint. It would be easier to depend on a AtomicInteger in driver for the id - and propagate that to execut

[GitHub] spark issue #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset API

2017-11-27 Thread ferdonline
Github user ferdonline commented on the issue: https://github.com/apache/spark/pull/19805 Thanks for you review. I'm working on the changes --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For addit

[GitHub] spark issue #19757: [SPARK-22529] [SQL] Relation stats should be consistent ...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19757 **[Test build #84213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84213/testReport)** for PR 19757 at commit [`45ab60b`](https://github.com/apache/spark/commit/45

[GitHub] spark pull request #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset A...

2017-11-27 Thread ferdonline
Github user ferdonline commented on a diff in the pull request: https://github.com/apache/spark/pull/19805#discussion_r153145177 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -524,22 +524,41 @@ class Dataset[T] private[sql]( */ @Experimen

[GitHub] spark pull request #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset A...

2017-11-27 Thread ferdonline
Github user ferdonline commented on a diff in the pull request: https://github.com/apache/spark/pull/19805#discussion_r153147567 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -524,22 +524,41 @@ class Dataset[T] private[sql]( */ @Experimen

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19813 **[Test build #84208 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84208/testReport)** for PR 19813 at commit [`9f848be`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19813 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84208/ Test FAILed. ---

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19813 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #19792: [SPARK-22566][PYTHON] Better error message for `_...

2017-11-27 Thread gberger
Github user gberger commented on a diff in the pull request: https://github.com/apache/spark/pull/19792#discussion_r153160307 --- Diff: python/pyspark/sql/tests.py --- @@ -1722,6 +1723,83 @@ def test_infer_long_type(self): self.assertEqual(_infer_type(2**61), LongType()

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19813 **[Test build #84211 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84211/testReport)** for PR 19813 at commit [`57b1add`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19813 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84211/ Test FAILed. ---

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19813 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #19824: Revert "[SPARK-18905][STREAMING] Fix the issue of...

2017-11-27 Thread victor-wong
GitHub user victor-wong opened a pull request: https://github.com/apache/spark/pull/19824 Revert "[SPARK-18905][STREAMING] Fix the issue of removing a failed jobset from JobScheduler.jobSets" ## What changes were proposed in this pull request? The code changes in PR(https:/

[GitHub] spark issue #19824: Revert "[SPARK-18905][STREAMING] Fix the issue of removi...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19824 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11994 **[Test build #84209 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84209/testReport)** for PR 11994 at commit [`e94def7`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11994 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11994 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84209/ Test PASSed. ---

[GitHub] spark pull request #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of ti...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19607#discussion_r153168918 --- Diff: python/pyspark/sql/session.py --- @@ -444,11 +445,30 @@ def _get_numpy_record_dtype(self, rec): record_type_list.append((str(c

[GitHub] spark pull request #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of ti...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19607#discussion_r153167040 --- Diff: python/pyspark/sql/tests.py --- @@ -3683,6 +3808,47 @@ def check_records_per_batch(x): else: self.spark.

[GitHub] spark pull request #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of ti...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19607#discussion_r153142283 --- Diff: python/pyspark/sql/types.py --- @@ -1678,37 +1679,105 @@ def from_arrow_schema(arrow_schema): for field in arrow_schema])

[GitHub] spark pull request #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of ti...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19607#discussion_r153149080 --- Diff: python/pyspark/sql/tests.py --- @@ -3192,16 +3255,49 @@ def test_filtered_frame(self): self.assertEqual(pdf.columns[0], "i")

[GitHub] spark pull request #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of ti...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19607#discussion_r153142413 --- Diff: python/pyspark/sql/types.py --- @@ -1678,37 +1679,105 @@ def from_arrow_schema(arrow_schema): for field in arrow_schema])

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19607 **[Test build #84210 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84210/testReport)** for PR 19607 at commit [`40a9735`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19607 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19607 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84210/ Test PASSed. ---

[GitHub] spark pull request #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of ti...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19607#discussion_r153176050 --- Diff: python/pyspark/sql/session.py --- @@ -444,11 +445,30 @@ def _get_numpy_record_dtype(self, rec): record_type_list.append((str(c

[GitHub] spark pull request #19752: [SPARK-22520][SQL] Support code generation for la...

2017-11-27 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19752#discussion_r153177002 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -158,111 +178,86 @@ abstract class Ca

[GitHub] spark pull request #19752: [SPARK-22520][SQL] Support code generation for la...

2017-11-27 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19752#discussion_r153177401 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -158,111 +178,86 @@ abstract class Ca

[GitHub] spark pull request #19823: [WIP][SPARK-22601][SQL] Data load is getting disp...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19823#discussion_r153177645 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -380,6 +380,12 @@ case class LoadDataCommand(

[GitHub] spark pull request #19757: [SPARK-22529] [SQL] Relation stats should be cons...

2017-11-27 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19757#discussion_r153177909 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveExplainSuite.scala --- @@ -29,21 +30,30 @@ class HiveExplainSuite extends Query

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19752 **[Test build #84212 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84212/testReport)** for PR 19752 at commit [`5adb513`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19752 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset A...

2017-11-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19805#discussion_r153178787 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -524,22 +524,41 @@ class Dataset[T] private[sql]( */ @Experime

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19752 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84212/ Test PASSed. ---

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19813 `splitExpressions` is the most common way we use in the codegen framework to deal with large code. If we can't make it work with whole stage codegen, we are not making many values. ---

[GitHub] spark issue #19821: [WIP][SPARK-22608][SQL] add new API to CodeGeneration.sp...

2017-11-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19821 is it really worth? seems not used in many places and eventually the if-else will be removed after we make `splitExpression` work with whole stage codegen --- --

[GitHub] spark pull request #19752: [SPARK-22520][SQL] Support code generation for la...

2017-11-27 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19752#discussion_r153181270 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -158,111 +178,86 @@ abstract class Cas

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-11-27 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19390 @vanzin @srowen Can I get a merge pls? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #19823: [WIP][SPARK-22601][SQL] Data load is getting displayed s...

2017-11-27 Thread sujith71955
Github user sujith71955 commented on the issue: https://github.com/apache/spark/pull/19823 Thanks for the comments guys, i am working on it.,will update the PR based on comments. --- - To unsubscribe, e-mail: revie

[GitHub] spark issue #19823: [WIP][SPARK-22601][SQL] Data load is getting displayed s...

2017-11-27 Thread sujith71955
Github user sujith71955 commented on the issue: https://github.com/apache/spark/pull/19823 Basically this validation stands good for both cases where scheme can come as null and not null, i will update the logic as Sean told. Thanks --- --

[GitHub] spark pull request #19752: [SPARK-22520][SQL] Support code generation for la...

2017-11-27 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19752#discussion_r153183113 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -158,111 +178,86 @@ abstract class Cas

[GitHub] spark issue #19757: [SPARK-22529] [SQL] Relation stats should be consistent ...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19757 **[Test build #84213 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84213/testReport)** for PR 19757 at commit [`45ab60b`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #19757: [SPARK-22529] [SQL] Relation stats should be consistent ...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19757 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19757: [SPARK-22529] [SQL] Relation stats should be consistent ...

2017-11-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19757 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84213/ Test PASSed. ---

[GitHub] spark issue #19817: [SPARK-22603][SQL] Fix 64KB JVM bytecode limit problem w...

2017-11-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19817 LGTM, merging to master/2.2! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #19811: [SPARK-18016][SQL] Code Generation: Constant Pool Limit ...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19811 **[Test build #84214 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84214/testReport)** for PR 19811 at commit [`006b2fd`](https://github.com/apache/spark/commit/00

[GitHub] spark pull request #19817: [SPARK-22603][SQL] Fix 64KB JVM bytecode limit pr...

2017-11-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19817 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19757: [SPARK-22529] [SQL] Relation stats should be cons...

2017-11-27 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/19757#discussion_r153184543 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveExplainSuite.scala --- @@ -29,21 +30,30 @@ class HiveExplainSuite extends QueryTest

[GitHub] spark pull request #19797: [SPARK-22570][SQL] Avoid to create a lot of globa...

2017-11-27 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19797#discussion_r153185397 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -173,6 +173,23 @@ class CodegenContext

[GitHub] spark pull request #19797: [SPARK-22570][SQL] Avoid to create a lot of globa...

2017-11-27 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19797#discussion_r153185862 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -111,11 +110,12 @@ private [sql] object G

[GitHub] spark pull request #19601: [SPARK-22383][SQL] Generate code to directly get ...

2017-11-27 Thread kiszk
Github user kiszk closed the pull request at: https://github.com/apache/spark/pull/19601 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19601: [SPARK-22383][SQL] Generate code to directly get ...

2017-11-27 Thread kiszk
GitHub user kiszk reopened a pull request: https://github.com/apache/spark/pull/19601 [SPARK-22383][SQL] Generate code to directly get value of primitive type array from ColumnVector for table cache ## What changes were proposed in this pull request? This PR generates the J

[GitHub] spark pull request #19797: [SPARK-22570][SQL] Avoid to create a lot of globa...

2017-11-27 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19797#discussion_r153186151 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala --- @@ -334,7 +334,7 @@ case class RegExpReplace(

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-11-27 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19601 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: rev

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19601 **[Test build #84215 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84215/testReport)** for PR 19601 at commit [`9b6b890`](https://github.com/apache/spark/commit/9b

[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...

2017-11-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19814 **[Test build #3992 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3992/testReport)** for PR 19814 at commit [`821ee19`](https://github.com/apache/spark/commit/8

[GitHub] spark pull request #19825: [SPARK-22615][SQL] Handle more cases in Propagate...

2017-11-27 Thread gengliangwang
GitHub user gengliangwang opened a pull request: https://github.com/apache/spark/pull/19825 [SPARK-22615][SQL] Handle more cases in PropagateEmptyRelation ## What changes were proposed in this pull request? Currently, in the optimize rule `PropagateEmptyRelation`, the follow

[GitHub] spark pull request #19797: [SPARK-22570][SQL] Avoid to create a lot of globa...

2017-11-27 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19797#discussion_r153187131 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala --- @@ -845,4 +845,24 @@ class CastSuite extends SparkFun

  1   2   3   4   5   >