[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159831366 --- Diff: core/src/test/scala/org/apache/spark/ThreadAudit.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159831631 --- Diff: core/src/test/scala/org/apache/spark/ThreadAudit.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159832598 --- Diff: core/src/test/scala/org/apache/spark/ThreadAudit.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159832729 --- Diff: core/src/test/scala/org/apache/spark/ThreadAudit.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159834709 --- Diff: core/src/test/scala/org/apache/spark/ThreadAudit.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159837804 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SessionStateSuite.scala --- @@ -39,6 +41,7 @@ class SessionStateSuite extends SparkFunSuite

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159838691 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/DataSourceAnalysisSuite.scala --- @@ -29,16 +29,24 @@ import org.apache.spark.sql.type

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20096 **[Test build #85720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85720/testReport)** for PR 20096 at commit [`be6c378`](https://github.com/apache/spark/commit/be

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159840754 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSessionStateSuite.scala --- @@ -28,14 +28,18 @@ import org.apache.spark.sql.hive.test

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159842007 --- Diff: core/src/test/scala/org/apache/spark/SparkFunSuite.scala --- @@ -31,15 +31,28 @@ import org.apache.spark.util.AccumulatorContext abstrac

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159841944 --- Diff: core/src/test/scala/org/apache/spark/SparkFunSuite.scala --- @@ -31,15 +31,28 @@ import org.apache.spark.util.AccumulatorContext abstrac

[GitHub] spark pull request #20164: [SPARK-22971][ML] OneVsRestModel should use tempo...

2018-01-05 Thread zhengruifeng
GitHub user zhengruifeng opened a pull request: https://github.com/apache/spark/pull/20164 [SPARK-22971][ML] OneVsRestModel should use temporary RawPredictionCol ## What changes were proposed in this pull request? use temporary RawPredictionCol in `OneVsRestModel#transform`

[GitHub] spark issue #20164: [SPARK-22971][ML] OneVsRestModel should use temporary Ra...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20164 **[Test build #85721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85721/testReport)** for PR 20164 at commit [`6c567ff`](https://github.com/apache/spark/commit/6c

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159844819 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -848,7 +853,7 @@ private[spark] class AppStatusListener(

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159835376 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -119,118 +121,115 @@ private class LiveTask( import LiveEnti

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159851476 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/SharedSQLContext.scala --- @@ -17,4 +17,17 @@ package org.apache.spark.sql.test

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #85722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85722/testReport)** for PR 19893 at commit [`0851ef2`](https://github.com/apache/spark/commit/08

[GitHub] spark issue #20164: [SPARK-22971][ML] OneVsRestModel should use temporary Ra...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20164 **[Test build #85721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85721/testReport)** for PR 20164 at commit [`6c567ff`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #20164: [SPARK-22971][ML] OneVsRestModel should use temporary Ra...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20164 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85721/ Test PASSed. ---

[GitHub] spark issue #20164: [SPARK-22971][ML] OneVsRestModel should use temporary Ra...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20164 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-05 Thread gengliangwang
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/20013 @vanzin I haven't run the code. I wonder which changes double the disk usage? The new indices or the cached quantiles? --- --

[GitHub] spark issue #20163: [SPARK-22966][PySpark] Spark SQL should handle Python UD...

2018-01-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20163 The problem here seems, `returnType` is mismatched to the value. In case of `DateType`, it needs an explicit conversion into integers: https://github.com/apache/spark/blob/1c9f95cb771ac

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20096 **[Test build #85720 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85720/testReport)** for PR 20096 at commit [`be6c378`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20096 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85720/ Test FAILed. ---

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20096 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic p...

2018-01-05 Thread gatorsmile
Github user gatorsmile closed the pull request at: https://github.com/apache/spark/pull/20162 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread xubo245
GitHub user xubo245 opened a pull request: https://github.com/apache/spark/pull/20165 [SPARK-22972] Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc ## What changes were proposed in this pull request? Fix the warning: Couldn't fi

[GitHub] spark issue #20155: [SPARK-22961][REGRESSION] Constant columns should genera...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20155 Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mai

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20165 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20155: [SPARK-22961][REGRESSION] Constant columns should...

2018-01-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20155 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20165 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@sp

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20165 **[Test build #85723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85723/testReport)** for PR 20165 at commit [`fa902d6`](https://github.com/apache/spark/commit/fa

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159877971 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/internal/HiveSerDe.scala --- @@ -72,7 +72,7 @@ object HiveSerDe { def sourceToSerDe(source:

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159878203 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala --- @@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends OrcSuit

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159878257 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala --- @@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends OrcSuit

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159878556 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala --- @@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends OrcSuit

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread xubo245
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159878683 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/internal/HiveSerDe.scala --- @@ -72,7 +72,7 @@ object HiveSerDe { def sourceToSerDe(source: St

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread xubo245
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159878689 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala --- @@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends OrcSuite w

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread xubo245
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159879115 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala --- @@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends OrcSuite w

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread xubo245
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159879314 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala --- @@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends OrcSuite w

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20165 **[Test build #85724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85724/testReport)** for PR 20165 at commit [`cf7cbce`](https://github.com/apache/spark/commit/cf

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20165 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20165 **[Test build #85724 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85724/testReport)** for PR 20165 at commit [`cf7cbce`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20165 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85724/ Test FAILed. ---

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20165 **[Test build #85725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85725/testReport)** for PR 20165 at commit [`b64ce53`](https://github.com/apache/spark/commit/b6

[GitHub] spark issue #20080: [SPARK-22870][CORE] Dynamic allocation should allow 0 id...

2018-01-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20080 Good point @felixcheung though the polling is hard-coded to 100ms, so will be close enough to "immediately". Allowing a value of 0 on both timeouts seems OK, but yeah not clear whether it does have t

[GitHub] spark pull request #20076: [SPARK-21786][SQL] When acquiring 'compressionCod...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20076#discussion_r159885686 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompressionCodecPrecedenceSuite.scala --- @@ -0,0 +1,117 @@

[GitHub] spark pull request #20076: [SPARK-21786][SQL] When acquiring 'compressionCod...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20076#discussion_r159889119 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -323,11 +323,13 @@ object SQLConf { .createWithDefault(

[GitHub] spark pull request #19247: [Spark-21996][SQL] read files with space in name ...

2018-01-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19247#discussion_r159893158 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala --- @@ -233,7 +233,7 @@ class FileStreamSource(

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #85722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85722/testReport)** for PR 19893 at commit [`0851ef2`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85722/ Test PASSed. ---

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159900236 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala --- @@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends OrcSuit

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159900297 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala --- @@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends OrcSuit

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159900404 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala --- @@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends OrcSuit

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20165 **[Test build #85723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85723/testReport)** for PR 20165 at commit [`fa902d6`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20165 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20165 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85723/ Test FAILed. ---

[GitHub] spark issue #20163: [SPARK-22966][PySpark] Spark SQL should handle Python UD...

2018-01-05 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/20163 I ran some experiments: ``` py_date = udf(datetime.date, DateType()) py_timestamp = udf(datetime.datetime, TimestampType()) ``` This works correctly ``` spark.range(1).s

[GitHub] spark pull request #20166: [SPARK-22973][SQL] Fix incorrect results of Casti...

2018-01-05 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/20166 [SPARK-22973][SQL] Fix incorrect results of Casting Map to String ## What changes were proposed in this pull request? This pr fixed the issue when casting maps into strings; ``` scala> Se

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20166 **[Test build #85726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85726/testReport)** for PR 20166 at commit [`efc28e3`](https://github.com/apache/spark/commit/ef

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20166 **[Test build #85727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85727/testReport)** for PR 20166 at commit [`be04e64`](https://github.com/apache/spark/commit/be

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20165 **[Test build #85725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85725/testReport)** for PR 20165 at commit [`b64ce53`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20165 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85725/ Test FAILed. ---

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20165 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #20167: Allow providing Mesos principal & secret via file...

2018-01-05 Thread rvesse
GitHub user rvesse opened a pull request: https://github.com/apache/spark/pull/20167 Allow providing Mesos principal & secret via files (SPARK-16501) ## What changes were proposed in this pull request? This commit modifies the Mesos submission client to allow the principal

[GitHub] spark issue #20167: Allow providing Mesos principal & secret via files (SPAR...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20167 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20154: [SPARK-22960][k8s] Make build-push-docker-images.sh more...

2018-01-05 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20154 ARGs can have default values, so we could do that if we decide to use the Docker Hub infra. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #20155: [SPARK-22961][REGRESSION] Constant columns should genera...

2018-01-05 Thread gengliangwang
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/20155 A late LGTM! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark pull request #20147: [SPARK-22940][SQL] HiveExternalCatalogVersionsSui...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20147#discussion_r159939270 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala --- @@ -85,6 +93,34 @@ class HiveExternalCatalogVersionsS

[GitHub] spark pull request #20147: [SPARK-22940][SQL] HiveExternalCatalogVersionsSui...

2018-01-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20147 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-05 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20013 > I wonder which changes double the disk usage? It's the new indices, more explicitly the values, not the keys. I tried changing the disk layout to write all the indices in a new namespace w

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20096 **[Test build #85728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85728/testReport)** for PR 20096 at commit [`1e5c7a9`](https://github.com/apache/spark/commit/1e

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159943385 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -119,118 +121,115 @@ private class LiveTask( import LiveEntityHelpe

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20166 **[Test build #85726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85726/testReport)** for PR 20166 at commit [`efc28e3`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20166 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20166 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85726/ Test FAILed. ---

[GitHub] spark issue #20168: SPARK-22730 Add ImageSchema support for non-integer imag...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20168 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20168: SPARK-22730 Add ImageSchema support for non-integ...

2018-01-05 Thread tomasatdatabricks
GitHub user tomasatdatabricks opened a pull request: https://github.com/apache/spark/pull/20168 SPARK-22730 Add ImageSchema support for non-integer image formats ## What changes were proposed in this pull request? Added functionality to handle all OpenCV modes to ImageSchema:

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20166 **[Test build #85727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85727/testReport)** for PR 20166 at commit [`be04e64`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20166 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20166 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85727/ Test PASSed. ---

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159955132 --- Diff: core/src/test/scala/org/apache/spark/ThreadAudit.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159954589 --- Diff: core/src/test/scala/org/apache/spark/SparkFunSuite.scala --- @@ -27,19 +27,55 @@ import org.apache.spark.util.AccumulatorContext /**

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159954825 --- Diff: core/src/test/scala/org/apache/spark/SparkFunSuite.scala --- @@ -27,19 +27,55 @@ import org.apache.spark.util.AccumulatorContext /**

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159955747 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/SharedSQLContext.scala --- @@ -17,4 +17,22 @@ package org.apache.spark.sql.test

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2018-01-05 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19848 Done. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spa

[GitHub] spark issue #20132: [SPARK-13030][ML] Follow-up cleanups for OneHotEncoderEs...

2018-01-05 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/20132 Thanks! Merging with master and branch-2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional c

[GitHub] spark pull request #20132: [SPARK-13030][ML] Follow-up cleanups for OneHotEn...

2018-01-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20132 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20096 **[Test build #85728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85728/testReport)** for PR 20096 at commit [`1e5c7a9`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20096 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20096 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85728/ Test PASSed. ---

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159958467 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -33,15 +40,21 @@ class MicroBatchExecution(

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159960120 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -236,14 +264,27 @@ class MicroBatchExecution(

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159980863 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala --- @@ -35,6 +35,16 @@ case class DataSourceV2Relat

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159981274 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamSourceV2.scala --- @@ -28,17 +28,38 @@ import org.json4s.jackson.

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159961186 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -357,31 +400,39 @@ class MicroBatchExecution(

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159978319 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -392,6 +443,21 @@ class MicroBatchExecution(

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/20142 I added the test. @gatorsmile do you have to take a look or let me know who should I ping for review? --- - To unsubscribe, e

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159980158 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -33,15 +40,21 @@ class MicroBatchExecution(

  1   2   3   >