[GitHub] spark issue #22960: [SPARK-25955][TEST] Porting JSON tests for CSV functions

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22960 Ur, maybe, I'm not clear to the point. The refactoring scope of this PR is limited to the new tests here. ``` test("from_csv uses DDL strings for defining a schema - java")

[GitHub] spark issue #20944: [SPARK-23831][SQL] Add org.apache.derby to IsolatedClien...

2018-11-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20944 Please describe manual tests and how it relates to actual usecase. --- - To unsubscribe, e-mail:

[GitHub] spark issue #22960: [SPARK-25955][TEST] Porting JSON tests for CSV functions

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22960 Yes. It would be great if we do that in this PR. When I did the similar thing for ORC (`port tests from Parquet to ORC`, `port from old ORC to new ORC`). I received the same comments.

[GitHub] spark pull request #22939: [SPARK-25446][R] Add schema_of_json() and schema_...

2018-11-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22939#discussion_r231404180 --- Diff: R/pkg/R/functions.R --- @@ -2230,6 +2237,32 @@ setMethod("from_json", signature(x = "Column", schema = "characterOrstructType")

[GitHub] spark pull request #22921: [SPARK-25908][CORE][SQL] Remove old deprecated it...

2018-11-06 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/22921#discussion_r231403827 --- Diff: R/pkg/R/functions.R --- @@ -319,6 +319,27 @@ setMethod("acos", column(jc) }) +#' @details +#'

[GitHub] spark pull request #22939: [SPARK-25446][R] Add schema_of_json() and schema_...

2018-11-06 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/22939#discussion_r231403096 --- Diff: R/pkg/R/functions.R --- @@ -2230,6 +2237,32 @@ setMethod("from_json", signature(x = "Column", schema = "characterOrstructType")

[GitHub] spark issue #22960: [SPARK-25955][TEST] Porting JSON tests for CSV functions

2018-11-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22960 **[Test build #98542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98542/testReport)** for PR 22960 at commit

[GitHub] spark issue #20944: [SPARK-23831][SQL] Add org.apache.derby to IsolatedClien...

2018-11-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20944 Sorry, why was this change required? I don't see https://github.com/apache/spark/pull/20944#issuecomment-379525776 is addressed Can you elaborate please? Why do we make `org.apache.derby` as

[GitHub] spark pull request #22954: [DO-NOT-MERGE][POC] Enables Arrow optimization fr...

2018-11-06 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/22954#discussion_r231402726 --- Diff: R/pkg/R/SQLContext.R --- @@ -147,6 +147,30 @@ getDefaultSqlSource <- function() { l[["spark.sql.sources.default"]] }

[GitHub] spark pull request #22954: [DO-NOT-MERGE][POC] Enables Arrow optimization fr...

2018-11-06 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/22954#discussion_r231402235 --- Diff: R/pkg/R/SQLContext.R --- @@ -147,6 +147,30 @@ getDefaultSqlSource <- function() { l[["spark.sql.sources.default"]] }

[GitHub] spark pull request #22954: [DO-NOT-MERGE][POC] Enables Arrow optimization fr...

2018-11-06 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/22954#discussion_r231402297 --- Diff: R/pkg/R/SQLContext.R --- @@ -172,15 +196,17 @@ getDefaultSqlSource <- function() { createDataFrame <- function(data, schema = NULL,

[GitHub] spark pull request #22954: [DO-NOT-MERGE][POC] Enables Arrow optimization fr...

2018-11-06 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/22954#discussion_r231402063 --- Diff: R/pkg/R/SQLContext.R --- @@ -147,6 +147,30 @@ getDefaultSqlSource <- function() { l[["spark.sql.sources.default"]] }

[GitHub] spark pull request #22954: [DO-NOT-MERGE][POC] Enables Arrow optimization fr...

2018-11-06 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/22954#discussion_r231401994 --- Diff: R/pkg/R/SQLContext.R --- @@ -147,6 +147,30 @@ getDefaultSqlSource <- function() { l[["spark.sql.sources.default"]] }

[GitHub] spark issue #22960: [SPARK-25955][TEST] Porting JSON tests for CSV functions

2018-11-06 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22960 > Sorry, but Porting seems to be not the best way to do this. I saw a bunch of common code in `Csv`/`JsonExpressionsSuite`, `Csv`/`JsonFunctionsSuite` and `Csv`/`JsonSuite`. I just didn't

[GitHub] spark pull request #22960: [SPARK-25955][TEST] Porting JSON tests for CSV fu...

2018-11-06 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/22960#discussion_r231399775 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala --- @@ -86,4 +86,82 @@ class CsvFunctionsSuite extends QueryTest with

[GitHub] spark issue #22951: [SPARK-25945][SQL] Support locale while parsing date/tim...

2018-11-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22951 **[Test build #98541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98541/testReport)** for PR 22951 at commit

[GitHub] spark issue #22958: [SPARK-25952][SQL] Passing actual schema to JacksonParse...

2018-11-06 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22958 @cloud-fan @HyukjinKwon May I ask you to have a look at this PR. --- - To unsubscribe, e-mail:

[GitHub] spark issue #22938: [SPARK-25935][SQL] Prevent null rows from JSON parser

2018-11-06 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22938 @HyukjinKwon Are you ok with the changes? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #15899: [SPARK-18466] added withFilter method to RDD

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15899 Since the issue is closed, this PR will be closed at the next infra clean ups. --- - To unsubscribe, e-mail:

[GitHub] spark issue #15899: [SPARK-18466] added withFilter method to RDD

2018-11-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15899 +1 for the decision and closing it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #15899: [SPARK-18466] added withFilter method to RDD

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15899 I see. Thank you for the clear decision, @rxin ! I'll close the issue as `Won't Fix`. And, could you close this PR, @reggert ? ---

[GitHub] spark issue #22818: [SPARK-25904][CORE] Allocate arrays smaller than Int.Max...

2018-11-06 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/22818 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #15899: [SPARK-18466] added withFilter method to RDD

2018-11-06 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15899 Thanks for the example. I didn't even know that was possible in earlier versions. I just looked it up: looks like Scala 2.11 rewrites for comprehensions into map, filter, and flatMap. That

[GitHub] spark issue #15899: [SPARK-18466] added withFilter method to RDD

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15899 Hi, @rxin , @srowen , @dbtsai , @felixcheung , @gatorsmile , @cloud-fan . I know this was not a recommended style, but there really exists users with this issue. And, from Spark

[GitHub] spark pull request #15899: [SPARK-18466] added withFilter method to RDD

2018-11-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15899#discussion_r231390266 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -387,6 +387,14 @@ abstract class RDD[T: ClassTag]( preservesPartitioning = true)

[GitHub] spark pull request #19796: [SPARK-22581][SQL] Catalog api does not allow to ...

2018-11-06 Thread timvw
Github user timvw closed the pull request at: https://github.com/apache/spark/pull/19796 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15899: [SPARK-18466] added withFilter method to RDD

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/15899#discussion_r231389555 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -387,6 +387,14 @@ abstract class RDD[T: ClassTag](

[GitHub] spark pull request #22089: [SPARK-25098][SQL]‘Cast’ will return NULL whe...

2018-11-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22089 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22943: [SPARK-25098][SQL] Trim the string when cast stri...

2018-11-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22943 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22943: [SPARK-25098][SQL] Trim the string when cast stringToTim...

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22943 Thank you, @wangyum and @cloud-fan . Merged to master. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19796: [SPARK-22581][SQL] Catalog api does not allow to ...

2018-11-06 Thread timvw
Github user timvw commented on a diff in the pull request: https://github.com/apache/spark/pull/19796#discussion_r231382828 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala --- @@ -411,7 +410,29 @@ abstract class Catalog { tableName: String,

[GitHub] spark pull request #22943: [SPARK-25098][SQL] Trim the string when cast stri...

2018-11-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22943#discussion_r231382309 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala --- @@ -140,16 +140,10 @@ class DateTimeUtilsSuite

[GitHub] spark issue #22932: [SPARK-25102][SQL] Write Spark version to ORC/Parquet fi...

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22932 Could you review this, @gatorsmile ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22943: [SPARK-25098][SQL] Trim the string when cast stri...

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22943#discussion_r231381218 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala --- @@ -140,16 +140,10 @@ class DateTimeUtilsSuite

[GitHub] spark pull request #22960: [SPARK-25955][TEST] Porting JSON tests for CSV fu...

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22960#discussion_r231380992 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala --- @@ -86,4 +86,82 @@ class CsvFunctionsSuite extends QueryTest with

[GitHub] spark pull request #22943: [SPARK-25098][SQL] Trim the string when cast stri...

2018-11-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22943#discussion_r231380552 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala --- @@ -140,16 +140,10 @@ class DateTimeUtilsSuite

[GitHub] spark pull request #22952: [SPARK-20568][SS] Rename files which are complete...

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22952#discussion_r231378889 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -530,6 +530,8 @@ Here are the details of all the sources in Spark.

[GitHub] spark issue #22951: [SPARK-25945][SQL] Support locale while parsing date/tim...

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22951 Could you rebase this once again, @MaxGekk ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22943: [SPARK-25098][SQL] Trim the string when cast stringToTim...

2018-11-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22943 Could you review this, @gatorsmile and @cloud-fan ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22867: [SPARK-25778] WriteAheadLogBackedBlockRDD in YARN Cluste...

2018-11-06 Thread gss2002
Github user gss2002 commented on the issue: https://github.com/apache/spark/pull/22867 @vanzin you are right! I appreciate the help with this one. I will cut a patch in the AM after testing on a large scale cluster job that is taking from IBM MQ and ETLing data and shipping off to

[GitHub] spark issue #22943: [SPARK-25098][SQL] Trim the string when cast stringToTim...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22943 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98540/ Test PASSed. ---

[GitHub] spark issue #22943: [SPARK-25098][SQL] Trim the string when cast stringToTim...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22943 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22943: [SPARK-25098][SQL] Trim the string when cast stringToTim...

2018-11-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22943 **[Test build #98540 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98540/testReport)** for PR 22943 at commit

[GitHub] spark issue #22921: [SPARK-25908][CORE][SQL] Remove old deprecated items in ...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22921 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98537/ Test FAILed. ---

[GitHub] spark issue #22921: [SPARK-25908][CORE][SQL] Remove old deprecated items in ...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22921 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22921: [SPARK-25908][CORE][SQL] Remove old deprecated items in ...

2018-11-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22921 **[Test build #98537 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98537/testReport)** for PR 22921 at commit

[GitHub] spark issue #22951: [SPARK-25945][SQL] Support locale while parsing date/tim...

2018-11-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22951 Looks good. I or someone else should take a closer look before getting this in. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22956: [SPARK-25950][SQL] from_csv should respect to spa...

2018-11-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22956 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22932: [SPARK-25102][SQL] Write Spark version to ORC/Parquet fi...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22932 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98539/ Test PASSed. ---

[GitHub] spark issue #22932: [SPARK-25102][SQL] Write Spark version to ORC/Parquet fi...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22932 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22932: [SPARK-25102][SQL] Write Spark version to ORC/Parquet fi...

2018-11-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22932 **[Test build #98539 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98539/testReport)** for PR 22932 at commit

[GitHub] spark issue #22956: [SPARK-25950][SQL] from_csv should respect to spark.sql....

2018-11-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22956 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22921: [SPARK-25908][CORE][SQL] Remove old deprecated items in ...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22921 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22921: [SPARK-25908][CORE][SQL] Remove old deprecated items in ...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22921 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98536/ Test FAILed. ---

[GitHub] spark pull request #22956: [SPARK-25950][SQL] from_csv should respect to spa...

2018-11-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22956#discussion_r231370599 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala --- @@ -92,8 +93,14 @@ case class CsvToStructs(

[GitHub] spark issue #22921: [SPARK-25908][CORE][SQL] Remove old deprecated items in ...

2018-11-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22921 **[Test build #98536 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98536/testReport)** for PR 22921 at commit

[GitHub] spark issue #22275: [SPARK-25274][PYTHON][SQL] In toPandas with Arrow send o...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22275 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98538/ Test PASSed. ---

[GitHub] spark issue #22275: [SPARK-25274][PYTHON][SQL] In toPandas with Arrow send o...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22275 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22275: [SPARK-25274][PYTHON][SQL] In toPandas with Arrow send o...

2018-11-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22275 **[Test build #98538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98538/testReport)** for PR 22275 at commit

[GitHub] spark issue #22937: [SPARK-25934] [Mesos] Don't propagate SPARK_CONF_DIR fro...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22937 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98535/ Test PASSed. ---

[GitHub] spark issue #22937: [SPARK-25934] [Mesos] Don't propagate SPARK_CONF_DIR fro...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22937 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22937: [SPARK-25934] [Mesos] Don't propagate SPARK_CONF_DIR fro...

2018-11-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22937 **[Test build #98535 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98535/testReport)** for PR 22937 at commit

[GitHub] spark issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsa...

2018-11-06 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22617 Retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-06 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r231359962 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -123,7

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-06 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r231359624 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark issue #22956: [SPARK-25950][SQL] from_csv should respect to spark.sql....

2018-11-06 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22956 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22956: [SPARK-25950][SQL] from_csv should respect to spa...

2018-11-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22956#discussion_r231359024 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala --- @@ -92,8 +93,14 @@ case class CsvToStructs(

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r231358749 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns {

[GitHub] spark issue #22165: [SPARK-25017][Core] Add test suite for ContextBarrierSta...

2018-11-06 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/22165 gental ping @jiangxb1987 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r231358690 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns {

[GitHub] spark issue #22961: [SPARK-25947][SQL] Reduce memory usage in ShuffleExchang...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22961 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22961: [SPARK-25947][SQL] Reduce memory usage in ShuffleExchang...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22961 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22961: [SPARK-25947][SQL] Reduce memory usage in ShuffleExchang...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22961 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22961: [SPARK-25947][SQL] Reduce memory usage in Shuffle...

2018-11-06 Thread mu5358271
GitHub user mu5358271 opened a pull request: https://github.com/apache/spark/pull/22961 [SPARK-25947][SQL] Reduce memory usage in ShuffleExchangeExec by selecting only the sort columns ## What changes were proposed in this pull request? When sorting rows,

[GitHub] spark issue #22855: [SPARK-25839] [Core] Implement use of KryoPool in KryoSe...

2018-11-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22855 **[Test build #4417 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4417/testReport)** for PR 22855 at commit

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-06 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r231350156 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark issue #22926: [SPARK-25917][Spark UI] memoryMetrics should be Json ign...

2018-11-06 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22926 Is this a problem in master at all? The data is serialized with `JacksonMessageWriter`, which seems to be configured properly: ``` private[v1] class JacksonMessageWriter extends

[GitHub] spark issue #22590: [SPARK-25574][SQL]Add an option `keepQuotes` for parsing...

2018-11-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22590 I wonder how important it is. I know `spark-csv` at Databricks supported different quote modes and that's gone when we ported that into Spark - the root cause was due to replacing the library

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r231348306 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -123,7

[GitHub] spark issue #22590: [SPARK-25574][SQL]Add an option `keepQuotes` for parsing...

2018-11-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22590 They should be documented in API doc like `DataFrameReader.scala`. For site, we should avoid doc duplication - It's a general issue to document options. ---

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Clie...

2018-11-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r231346067 --- Diff: docs/configuration.md --- @@ -266,6 +266,40 @@ of the most common options to set are: Only has effect in Spark standalone mode or Mesos

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Clie...

2018-11-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r231346390 --- Diff: docs/configuration.md --- @@ -266,6 +266,40 @@ of the most common options to set are: Only has effect in Spark standalone mode or Mesos

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Clie...

2018-11-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r231346161 --- Diff: docs/configuration.md --- @@ -266,6 +266,40 @@ of the most common options to set are: Only has effect in Spark standalone mode or Mesos

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Clie...

2018-11-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r231346117 --- Diff: docs/configuration.md --- @@ -266,6 +266,40 @@ of the most common options to set are: Only has effect in Spark standalone mode or Mesos

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Clie...

2018-11-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r231346507 --- Diff: docs/configuration.md --- @@ -266,6 +266,40 @@ of the most common options to set are: Only has effect in Spark standalone mode or Mesos

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Clie...

2018-11-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r231346593 --- Diff: docs/monitoring.md --- @@ -202,6 +202,28 @@ Security options for the Spark History Server are covered more detail in the applications

[GitHub] spark issue #22954: [DO-NOT-MERGE][POC] Enables Arrow optimization from R Da...

2018-11-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22954 So far, the regressions tests are passed and newly added test for R optimization is verified locally. Let me fix CRAN test and some nits. ---

[GitHub] spark issue #22956: [SPARK-25950][SQL] from_csv should respect to spark.sql....

2018-11-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22956 Looks good. I or someone else should take a closer look before getting this in. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-06 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r231344398 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -123,7

[GitHub] spark pull request #22960: [SPARK-25955][TEST] Porting JSON tests for CSV fu...

2018-11-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22960#discussion_r231344120 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala --- @@ -86,4 +86,82 @@ class CsvFunctionsSuite extends QueryTest with

[GitHub] spark issue #22894: [SPARK-25885][Core][Minor] HighlyCompressedMapStatus des...

2018-11-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22894 **[Test build #4416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4416/testReport)** for PR 22894 at commit

[GitHub] spark issue #22909: [SPARK-25897][k8s] Hook up k8s integration tests to sbt ...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22909 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22909: [SPARK-25897][k8s] Hook up k8s integration tests to sbt ...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22909 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98532/ Test FAILed. ---

[GitHub] spark issue #22943: [SPARK-25098][SQL] Trim the string when cast stringToTim...

2018-11-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22943 **[Test build #98540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98540/testReport)** for PR 22943 at commit

[GitHub] spark issue #22909: [SPARK-25897][k8s] Hook up k8s integration tests to sbt ...

2018-11-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22909 **[Test build #98532 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98532/testReport)** for PR 22909 at commit

[GitHub] spark issue #22943: [SPARK-25098][SQL] Trim the string when cast stringToTim...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22943 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22943: [SPARK-25098][SQL] Trim the string when cast stringToTim...

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22943 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22926: [SPARK-25917][Spark UI] memoryMetrics should be Json ign...

2018-11-06 Thread jianjianjiao
Github user jianjianjiao commented on the issue: https://github.com/apache/spark/pull/22926 @AmplabJenkins Could you please find someone to review this? I believe this is a bug in Spark UI. Thanks. --- - To

[GitHub] spark issue #22960: [SPARK-25955][TEST] Porting JSON tests for CSV functions

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22960 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22960: [SPARK-25955][TEST] Porting JSON tests for CSV functions

2018-11-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22960 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98531/ Test PASSed. ---

  1   2   3   4   >