[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21389 **[Test build #92263 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92263/testReport)** for PR 21389 at commit [`a3c400d`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21389 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21594 **[Test build #92265 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92265/testReport)** for PR 21594 at commit [`bf42fdf`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21389 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92263/ Test PASSed. ---

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21594 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21594 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92265/ Test FAILed. ---

[GitHub] spark issue #21424: [SPARK-24379] BroadcastExchangeExec should catch SparkOu...

2018-06-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21424 I left the JIRA resolved as "Won't fix" given the discussion above. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.a

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-24 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197631699 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -109,6 +134,20 @@ class JDBCOptions( s"W

[GitHub] spark issue #21618: [SPARK-20408][SQL] Get the glob path in parallel to redu...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21618 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/443/

[GitHub] spark issue #21618: [SPARK-20408][SQL] Get the glob path in parallel to redu...

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21618 **[Test build #92266 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92266/testReport)** for PR 21618 at commit [`9b8a4b3`](https://github.com/apache/spark/commit/9b

[GitHub] spark issue #21618: [SPARK-20408][SQL] Get the glob path in parallel to redu...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21618 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21621: [SPARK-24633][SQL] Fix codegen when split is requ...

2018-06-24 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21621#discussion_r197633691 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -556,6 +556,17 @@ class DataFrameFunctionsSuite extends QueryTest

[GitHub] spark issue #21621: [SPARK-24633][SQL] Fix codegen when split is required fo...

2018-06-24 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21621 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread Fokko
Github user Fokko commented on the issue: https://github.com/apache/spark/pull/21596 @MaxGekk Any pointers on how to build this test-jar containing spark sql test-classes? --- - To unsubscribe, e-mail: reviews-unsub

[GitHub] spark pull request #21621: [SPARK-24633][SQL] Fix codegen when split is requ...

2018-06-24 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21621#discussion_r197635219 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -556,6 +556,17 @@ class DataFrameFunctionsSuite extends QueryTe

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21596 `mvn package -pl sql/core -DskipTests` should build such jar --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org F

[GitHub] spark issue #21618: [SPARK-20408][SQL] Get the glob path in parallel to redu...

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21618 **[Test build #92266 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92266/testReport)** for PR 21618 at commit [`9b8a4b3`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #21618: [SPARK-20408][SQL] Get the glob path in parallel to redu...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21618 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92266/ Test FAILed. ---

[GitHub] spark issue #21618: [SPARK-20408][SQL] Get the glob path in parallel to redu...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21618 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21626: [SPARK-24642][SQL] New function infers schema for...

2018-06-24 Thread MaxGekk
GitHub user MaxGekk opened a pull request: https://github.com/apache/spark/pull/21626 [SPARK-24642][SQL] New function infers schema for JSON column ## What changes were proposed in this pull request? In the PR, I propose new aggregate function - *infer_schema()*. The functi

[GitHub] spark issue #21626: [SPARK-24642][SQL] New function infers schema for JSON c...

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21626 **[Test build #92267 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92267/testReport)** for PR 21626 at commit [`333139d`](https://github.com/apache/spark/commit/33

[GitHub] spark issue #21626: [SPARK-24642][SQL] New function infers schema for JSON c...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21626 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21626: [SPARK-24642][SQL] New function infers schema for JSON c...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21626 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20949: [SPARK-19018][SQL] Add support for custom encodin...

2018-06-24 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/20949#discussion_r197643948 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala --- @@ -512,6 +512,43 @@ class CSVSuite extends QueryTest w

[GitHub] spark pull request #20949: [SPARK-19018][SQL] Add support for custom encodin...

2018-06-24 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/20949#discussion_r197644302 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala --- @@ -512,6 +512,43 @@ class CSVSuite extends QueryTest w

[GitHub] spark pull request #20949: [SPARK-19018][SQL] Add support for custom encodin...

2018-06-24 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/20949#discussion_r197644657 --- Diff: python/pyspark/sql/readwriter.py --- @@ -895,6 +895,8 @@ def csv(self, path, mode=None, compression=None, sep=None, quote=None, escape=No

[GitHub] spark pull request #20949: [SPARK-19018][SQL] Add support for custom encodin...

2018-06-24 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/20949#discussion_r197644850 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala --- @@ -146,7 +148,13 @@ private[csv] class CsvOutputW

[GitHub] spark pull request #21621: [SPARK-24633][SQL] Fix codegen when split is requ...

2018-06-24 Thread bersprockets
Github user bersprockets commented on a diff in the pull request: https://github.com/apache/spark/pull/21621#discussion_r197646450 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -556,6 +556,17 @@ class DataFrameFunctionsSuite extends Que

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21596 **[Test build #92268 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92268/testReport)** for PR 21596 at commit [`4dd812a`](https://github.com/apache/spark/commit/4d

[GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21320#discussion_r197646687 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/QueryPlanConstraints.scala --- @@ -99,27 +100,28 @@ trait ConstraintHe

[GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21320#discussion_r197629698 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala --- @@ -301,7 +301,6 @@ case class FileSourceScanExec(

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92269 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92269/testReport)** for PR 21073 at commit [`ce5541e`](https://github.com/apache/spark/commit/ce

[GitHub] spark issue #21626: [SPARK-24642][SQL] New function infers schema for JSON c...

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21626 **[Test build #92267 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92267/testReport)** for PR 21626 at commit [`333139d`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #21626: [SPARK-24642][SQL] New function infers schema for JSON c...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21626 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92267/ Test PASSed. ---

[GitHub] spark issue #21626: [SPARK-24642][SQL] New function infers schema for JSON c...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21626 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21596 **[Test build #92270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92270/testReport)** for PR 21596 at commit [`95840a5`](https://github.com/apache/spark/commit/95

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21596 **[Test build #92270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92270/testReport)** for PR 21596 at commit [`95840a5`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21596 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92270/ Test FAILed. ---

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21596 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread Fokko
Github user Fokko commented on the issue: https://github.com/apache/spark/pull/21596 Thanks @MaxGekk for pointing out. I've also added the command to the comments in the test for future reference. I've ran the benchmark. The inferring became a bit slower (`0.85x` relative), b

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21320 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21320 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/444/

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21596 **[Test build #92271 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92271/testReport)** for PR 21596 at commit [`92a78aa`](https://github.com/apache/spark/commit/92

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21320 **[Test build #92272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92272/testReport)** for PR 21320 at commit [`cb858f2`](https://github.com/apache/spark/commit/cb

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-06-24 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/21320 @gatorsmile I've removed the changes to the files as you requested. This removes support for schema pruning on filters of queries. I've pushed the previous revision to a new branch in our `spark-pub

[GitHub] spark pull request #21625: [SPARK-24206][SQL][FOLLOW-UP] Update DataSourceRe...

2018-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/21625#discussion_r197652431 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala --- @@ -39,9 +39,11 @@ import org.apache.spa

[GitHub] spark pull request #21627: [SPARK-24484][MLLIB]Power Iteration Clustering is...

2018-06-24 Thread shahidki31
GitHub user shahidki31 opened a pull request: https://github.com/apache/spark/pull/21627 [SPARK-24484][MLLIB]Power Iteration Clustering is giving incorrect clustering results when there are mutiple leading eigen values. ## What changes were proposed in this pull request? ![imag

[GitHub] spark issue #21627: [SPARK-24484][MLLIB]Power Iteration Clustering is giving...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21627 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21627: [SPARK-24484][MLLIB]Power Iteration Clustering is giving...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21627 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21627: [SPARK-24484][MLLIB]Power Iteration Clustering is giving...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21627 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21596 **[Test build #92268 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92268/testReport)** for PR 21596 at commit [`4dd812a`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21596 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92268/ Test PASSed. ---

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21596 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92269 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92269/testReport)** for PR 21073 at commit [`ce5541e`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92269/ Test PASSed. ---

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92273/testReport)** for PR 21073 at commit [`4893df9`](https://github.com/apache/spark/commit/48

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21594 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21594 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/445/

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21594 **[Test build #92274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92274/testReport)** for PR 21594 at commit [`bf42fdf`](https://github.com/apache/spark/commit/bf

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21320 **[Test build #92272 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92272/testReport)** for PR 21320 at commit [`cb858f2`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21320 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21320 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92272/ Test PASSed. ---

[GitHub] spark issue #21598: [SPARK-24605][SQL] size(null) returns null instead of -1

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21598 All the behavior changes need very careful reviews and discussions. Whenever we decide to make a behavior change, we should document it in the migration guide and provide a conf to revert it back

[GitHub] spark pull request #21618: [SPARK-20408][SQL] Get the glob path in parallel ...

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21618#discussion_r197657738 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -724,4 +726,35 @@ object DataSource extends Logging

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21596 **[Test build #92271 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92271/testReport)** for PR 21596 at commit [`92a78aa`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21596 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92271/ Test PASSed. ---

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21596 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21221: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21221 **[Test build #92275 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92275/testReport)** for PR 21221 at commit [`812fdcf`](https://github.com/apache/spark/commit/81

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21596 Can we target this to Spark 3.0, which should be the next release after Spark 2.4 release? --- - To unsubscribe, e-mail: rev

[GitHub] spark issue #21598: [SPARK-24605][SQL] size(null) returns null instead of -1

2018-06-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21598 > Based on my understanding, the decision is made case by case. I concur. --- - To unsubscribe, e-mail: reviews-unsu

[GitHub] spark pull request #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21596#discussion_r197661126 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonBenchmarks.scala --- @@ -25,8 +25,13 @@ import org.apache.spark.u

[GitHub] spark issue #21617: [SPARK-24634][SS] Add a new metric regarding number of r...

2018-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21617 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: rev

[GitHub] spark issue #21617: [SPARK-24634][SS] Add a new metric regarding number of r...

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21617 **[Test build #92276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92276/testReport)** for PR 21617 at commit [`ff1b895`](https://github.com/apache/spark/commit/ff

[GitHub] spark pull request #20949: [SPARK-19018][SQL] Add support for custom encodin...

2018-06-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20949#discussion_r197662012 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala --- @@ -512,6 +512,43 @@ class CSVSuite extends QueryTe

[GitHub] spark pull request #20949: [SPARK-19018][SQL] Add support for custom encodin...

2018-06-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20949#discussion_r197662087 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala --- @@ -513,6 +513,43 @@ class CSVSuite extends QueryTe

[GitHub] spark issue #20949: [SPARK-19018][SQL] Add support for custom encoding on cs...

2018-06-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20949 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@s

[GitHub] spark issue #20949: [SPARK-19018][SQL] Add support for custom encoding on cs...

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20949 **[Test build #92277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92277/testReport)** for PR 20949 at commit [`0d0addf`](https://github.com/apache/spark/commit/0d

[GitHub] spark pull request #21626: [SPARK-24642][SQL] New function infers schema for...

2018-06-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21626#discussion_r197662496 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/InferSchema.scala --- @@ -0,0 +1,162 @@ +/* + * Licen

[GitHub] spark pull request #21626: [SPARK-24642][SQL] New function infers schema for...

2018-06-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21626#discussion_r197662799 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/InferSchema.scala --- @@ -0,0 +1,162 @@ +/* + * Licen

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92273 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92273/testReport)** for PR 21073 at commit [`4893df9`](https://github.com/apache/spark/commit/4

[GitHub] spark pull request #21628: [SPARK-23776][DOC] Update instructions for runnin...

2018-06-24 Thread bersprockets
GitHub user bersprockets opened a pull request: https://github.com/apache/spark/pull/21628 [SPARK-23776][DOC] Update instructions for running PySpark after building with SBT ## What changes were proposed in this pull request? This update tells the reader how to build Spark

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92273/ Test PASSed. ---

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21628 **[Test build #92278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92278/testReport)** for PR 21628 at commit [`9fcd05d`](https://github.com/apache/spark/commit/9f

[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21628 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21628 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21629: Fix minor typo in docs/cloud-integration.md

2018-06-24 Thread jkleckner
GitHub user jkleckner opened a pull request: https://github.com/apache/spark/pull/21629 Fix minor typo in docs/cloud-integration.md ## What changes were proposed in this pull request? Minor typo in docs/cloud-integration.md ## How was this patch tested? Thi

[GitHub] spark issue #21629: Fix minor typo in docs/cloud-integration.md

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21629 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21629: Fix minor typo in docs/cloud-integration.md

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21629 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21629: Fix minor typo in docs/cloud-integration.md

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21629 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21628 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...

2018-06-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21628 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92278/ Test PASSed. ---

[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...

2018-06-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21628 **[Test build #92278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92278/testReport)** for PR 21628 at commit [`9fcd05d`](https://github.com/apache/spark/commit/9

[GitHub] spark pull request #21628: [SPARK-23776][DOC] Update instructions for runnin...

2018-06-24 Thread bersprockets
Github user bersprockets commented on a diff in the pull request: https://github.com/apache/spark/pull/21628#discussion_r197667457 --- Diff: docs/building-spark.md --- @@ -215,19 +215,23 @@ If you are building Spark for use in a Python environment and you wish to pip in

[GitHub] spark pull request #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-24 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21073#discussion_r197666062 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -475,6 +474,231 @@ case class MapEntries(c

[GitHub] spark pull request #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-24 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21073#discussion_r197666362 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -475,6 +474,231 @@ case class MapEntries(c

[GitHub] spark pull request #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-24 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21073#discussion_r197666255 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -475,6 +474,231 @@ case class MapEntries(c

[GitHub] spark pull request #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-24 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21073#discussion_r197665954 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -475,6 +474,231 @@ case class MapEntries(c

  1   2   >