[GitHub] spark issue #22503: [SPARK-25493][SQL] Use auto-detection for CRLF in CSV da...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22503 Seems fine but I or someone else should take a closer look before getting this in. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22503: [SPARK-25493][SQL] Use auto-detection for CRLF in...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22503#discussion_r220410193 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala --- @@ -212,6 +212,8 @@ class CSVOptions(

[GitHub] spark issue #22549: [SPARK-25533][CORE][WEBUI]AppSummary should hold the inf...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22549 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96580/ Test FAILed. ---

[GitHub] spark issue #22549: [SPARK-25533][CORE][WEBUI]AppSummary should hold the inf...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22549 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22548: [SPARK-25534][SQL] Make `SQLHelper` trait

2018-09-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22548 Thank you! I'll merge this to the master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22520: [SPARK-25509][Core]Windows doesn't support POSIX permiss...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22520 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22458: [SPARK-25459] Add viewOriginalText back to Catalo...

2018-09-25 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/22458#discussion_r220410022 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -2348,4 +2348,17 @@ class HiveDDLSuite }

[GitHub] spark issue #22520: [SPARK-25509][Core]Windows doesn't support POSIX permiss...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22520 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96575/ Test PASSed. ---

[GitHub] spark issue #22549: [SPARK-25533][CORE][WEBUI]AppSummary should hold the inf...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22549 **[Test build #96580 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96580/testReport)** for PR 22549 at commit

[GitHub] spark pull request #22543: [SPARK-23715][SQL] improve document for from/to_u...

2018-09-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22543#discussion_r220409796 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1018,9 +1018,20 @@ case class

[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...

2018-09-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22524 Oh, I see. May the code template make some confusion. I'd change it. The downstream code is wrapped inside an if block. I don't clearly show how downstream operators work in codegen. Let me update

[GitHub] spark issue #22520: [SPARK-25509][Core]Windows doesn't support POSIX permiss...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22520 **[Test build #96575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96575/testReport)** for PR 22520 at commit

[GitHub] spark issue #22548: [SPARK-25534][SQL] Make `SQLHelper` trait

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22548 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22548: [SPARK-25534][SQL] Make `SQLHelper` trait

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22548 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96579/ Test PASSed. ---

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-09-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r220409552 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala --- @@ -167,6 +170,56 @@ package object util {

[GitHub] spark pull request #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22379#discussion_r220409483 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala --- @@ -520,7 +520,10 @@ object FunctionRegistry {

[GitHub] spark issue #22548: [SPARK-25534][SQL] Make `SQLHelper` trait

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22548 **[Test build #96579 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96579/testReport)** for PR 22548 at commit

[GitHub] spark pull request #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.wr...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22453#discussion_r220409331 --- Diff: docs/sql-programming-guide.md --- @@ -1002,6 +1002,21 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession

[GitHub] spark pull request #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-25 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22379#discussion_r220408893 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala --- @@ -520,7 +520,10 @@ object FunctionRegistry {

[GitHub] spark pull request #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.wr...

2018-09-25 Thread seancxmao
Github user seancxmao commented on a diff in the pull request: https://github.com/apache/spark/pull/22453#discussion_r220407692 --- Diff: docs/sql-programming-guide.md --- @@ -1002,6 +1002,21 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession

[GitHub] spark issue #22419: [SPARK-23906][SQL] Add built-in UDF TRUNCATE(number)

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22419 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22419: [SPARK-23906][SQL] Add built-in UDF TRUNCATE(number)

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22419 **[Test build #96587 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96587/testReport)** for PR 22419 at commit

[GitHub] spark issue #22419: [SPARK-23906][SQL] Add built-in UDF TRUNCATE(number)

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22419 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22442: [SPARK-25447][SQL] Support JSON options by schema...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22442#discussion_r220407196 --- Diff: python/pyspark/sql/functions.py --- @@ -2328,11 +2328,14 @@ def to_json(col, options={}): @ignore_unicode_prefix @since(2.4)

[GitHub] spark pull request #22528: [SPARK-25513][SQL] Read zipped CSV and JSON

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22528#discussion_r220406916 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CodecStreams.scala --- @@ -41,7 +42,12 @@ object CodecStreams {

[GitHub] spark pull request #22534: [SPARK-25514][SQL] Generating pretty JSON by to_j...

2018-09-25 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22534 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22534: [SPARK-25514][SQL] Generating pretty JSON by to_json

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22534 LGTM Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #22534: [SPARK-25514][SQL] Generating pretty JSON by to_j...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22534#discussion_r220405853 --- Diff: R/pkg/R/functions.R --- @@ -198,8 +198,9 @@ NULL #' } #' @param ... additional argument(s). In \code{to_json} and

[GitHub] spark pull request #22534: [SPARK-25514][SQL] Generating pretty JSON by to_j...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22534#discussion_r220405798 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -3635,6 +3637,8 @@ object functions { * @param e a column

[GitHub] spark pull request #22537: [SPARK-21291][R] add R partitionBy API in DataFra...

2018-09-25 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22537 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22379 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22379 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96573/ Test PASSed. ---

[GitHub] spark pull request #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Con...

2018-09-25 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22540 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22379 **[Test build #96573 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96573/testReport)** for PR 22379 at commit

[GitHub] spark issue #22537: [SPARK-21291][R] add R partitionBy API in DataFrame

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22537 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22504 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22504 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96576/ Test FAILed. ---

[GitHub] spark issue #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Conf to sp...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22540 Merged to master and branch-2.4. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #22548: [SPARK-25534][SQL] Make `SQLHelper` trait

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22548 **[Test build #96586 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96586/testReport)** for PR 22548 at commit

[GitHub] spark issue #22548: [SPARK-25534][SQL] Make `SQLHelper` trait

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22548 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22548: [SPARK-25534][SQL] Make `SQLHelper` trait

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22548 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Conf to sp...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22540 Let me push this into branch-2.4 as well to address renaming concern by @BryanCutler. --- - To unsubscribe, e-mail:

[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22504 **[Test build #96576 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96576/testReport)** for PR 22504 at commit

[GitHub] spark pull request #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Con...

2018-09-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22540#discussion_r220403432 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala --- @@ -131,11 +131,8 @@ object ArrowUtils { } else {

[GitHub] spark issue #22550: [SPARK-25501] Kafka delegation token support

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22550 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22550: [SPARK-25501] Kafka delegation token support

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22550 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22550: [SPARK-25501] Kafka delegation token support

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22550 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22550: [SPARK-25501] Kafka delegation token support

2018-09-25 Thread merlintang
GitHub user merlintang opened a pull request: https://github.com/apache/spark/pull/22550 [SPARK-25501] Kafka delegation token support ## What changes were proposed in this pull request? Kafaka is going to support delegation token, Spark need to read the delegation token

[GitHub] spark issue #22362: [SPARK-25372][YARN][K8S] Deprecate and generalize keytab...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22362 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96572/ Test PASSed. ---

[GitHub] spark issue #22362: [SPARK-25372][YARN][K8S] Deprecate and generalize keytab...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22362 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22455: [SPARK-24572][SPARKR] "eager execution" for R shell, IDE

2018-09-25 Thread falaki
Github user falaki commented on the issue: https://github.com/apache/spark/pull/22455 @adrian555 These all great points. My high-level was enabling other platforms (e.g., Jupyter) to plugin more advanced (custom) functions for displaying SparkDataFrame. If the framework does not set

[GitHub] spark issue #22362: [SPARK-25372][YARN][K8S] Deprecate and generalize keytab...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22362 **[Test build #96572 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96572/testReport)** for PR 22362 at commit

[GitHub] spark issue #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Conf to sp...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22540 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Conf to sp...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22540 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96574/ Test PASSed. ---

[GitHub] spark issue #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Conf to sp...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22540 **[Test build #96574 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96574/testReport)** for PR 22540 at commit

[GitHub] spark pull request #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_...

2018-09-25 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22237#discussion_r220400809 --- Diff: docs/sql-programming-guide.md --- @@ -1877,6 +1877,10 @@ working with timestamps in `pandas_udf`s to get the best performance, see #

[GitHub] spark pull request #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_...

2018-09-25 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22237#discussion_r220400637 --- Diff: docs/sql-programming-guide.md --- @@ -1877,6 +1877,10 @@ working with timestamps in `pandas_udf`s to get the best performance, see #

[GitHub] spark pull request #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_...

2018-09-25 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22237#discussion_r220401132 --- Diff: docs/sql-programming-guide.md --- @@ -1877,6 +1877,10 @@ working with timestamps in `pandas_udf`s to get the best performance, see #

[GitHub] spark pull request #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_...

2018-09-25 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22237#discussion_r220401206 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -554,18 +554,30 @@ case class

[GitHub] spark pull request #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Con...

2018-09-25 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22540#discussion_r220400221 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala --- @@ -131,11 +131,8 @@ object ArrowUtils { } else {

[GitHub] spark issue #22548: [SPARK-25534][SQL] Make `SupportWithSQLConf` trait

2018-09-25 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22548 sure --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22010: [SPARK-21436][CORE] Take advantage of known parti...

2018-09-25 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22010#discussion_r220399074 --- Diff: core/src/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala --- @@ -19,7 +19,7 @@ package org.apache.spark.rdd import

[GitHub] spark pull request #22010: [SPARK-21436][CORE] Take advantage of known parti...

2018-09-25 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22010#discussion_r220399123 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -42,7 +42,8 @@ import org.apache.spark.partial.GroupedCountEvaluator import

[GitHub] spark issue #22548: [SPARK-25534][SQL] Make `SupportWithSQLConf` trait

2018-09-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22548 It turns out that `withTempTable` cannot be here together because it requires `spark.catalog`. ```scala def withTempTable(tableNames: String*)(f: => Unit): Unit = { try f

[GitHub] spark issue #22521: [SPARK-24519][CORE] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIG...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22521 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22521: [SPARK-24519][CORE] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIG...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22521 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22521: [SPARK-24519][CORE] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIG...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22521 **[Test build #96585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96585/testReport)** for PR 22521 at commit

[GitHub] spark pull request #22546: [SPARK-25422][CORE] Don't memory map blocks strea...

2018-09-25 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22546 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22521: [SPARK-24519][CORE] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIG...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22521 **[Test build #96584 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96584/testReport)** for PR 22521 at commit

[GitHub] spark issue #22521: [SPARK-24519][CORE] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIG...

2018-09-25 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22521 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22546: [SPARK-25422][CORE] Don't memory map blocks streamed to ...

2018-09-25 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22546 thanks, merging to master/2.4! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #22521: [SPARK-24519][CORE] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIG...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22521 **[Test build #96571 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96571/testReport)** for PR 22521 at commit

[GitHub] spark issue #22521: [SPARK-24519][CORE] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIG...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22521 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22521: [SPARK-24519][CORE] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIG...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22521 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96571/ Test FAILed. ---

[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...

2018-09-25 Thread shuyang-truex
Github user shuyang-truex commented on the issue: https://github.com/apache/spark/pull/21929 Just like to add that I was able to test this PR in our production environment and it works well, really hope to see it get merged. ---

[GitHub] spark issue #22548: [SPARK-25534][SQL] Make `SupportWithSQLConf` trait

2018-09-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22548 +1, Sure, @cloud-fan ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22010: [SPARK-21436][CORE] Take advantage of known partitioner ...

2018-09-25 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/22010 Did another quick micro benchmark on a small cluster: ```scala import org.apache.spark.util.collection.ExternalAppendOnlyMap def removeDuplicatesInPartition(partition:

[GitHub] spark issue #22548: [SPARK-25534][SQL] Make `SupportWithSQLConf` trait

2018-09-25 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22548 We have duplicated `withTempPath`, `withTempTable` too. Shall we pull them out as well? The trait name can be `SQLHelper` ---

[GitHub] spark issue #22010: [SPARK-21436][CORE] Take advantage of known partitioner ...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22010 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22010: [SPARK-21436][CORE] Take advantage of known partitioner ...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22010 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22255: [SPARK-25102][Spark Core] Write Spark version informatio...

2018-09-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22255 Hi, @npoberezkin . Thank you for your first contribution. Could you update your PR to use custom key-value metadata according to the above advice of @rdblue ? Also, please use tag `[SQL]`

[GitHub] spark issue #22549: [SPARK-25533][CORE][WEBUI]AppSummary should hold the inf...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22549 **[Test build #96582 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96582/testReport)** for PR 22549 at commit

[GitHub] spark issue #22010: [SPARK-21436][CORE] Take advantage of known partitioner ...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22010 **[Test build #96583 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96583/testReport)** for PR 22010 at commit

[GitHub] spark pull request #22549: [SPARK-25533][CORE][WEBUI]AppSummary should hold ...

2018-09-25 Thread shahidki31
Github user shahidki31 commented on a diff in the pull request: https://github.com/apache/spark/pull/22549#discussion_r220385769 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -388,10 +388,11 @@ private[spark] class AppStatusListener(

[GitHub] spark pull request #22549: [SPARK-25533][CORE][WEBUI]AppSummary should hold ...

2018-09-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22549#discussion_r220383364 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -388,10 +388,11 @@ private[spark] class AppStatusListener(

[GitHub] spark issue #22325: [SPARK-25318]. Add exception handling when wrapping the ...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22325 **[Test build #96581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96581/testReport)** for PR 22325 at commit

[GitHub] spark pull request #22414: [SPARK-25424][SQL] Window duration and slide dura...

2018-09-25 Thread raghavgautam
Github user raghavgautam commented on a diff in the pull request: https://github.com/apache/spark/pull/22414#discussion_r220380587 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/TimeWindowSuite.scala --- @@ -122,6 +123,51 @@ class TimeWindowSuite

[GitHub] spark pull request #22414: [SPARK-25424][SQL] Window duration and slide dura...

2018-09-25 Thread raghavgautam
Github user raghavgautam commented on a diff in the pull request: https://github.com/apache/spark/pull/22414#discussion_r220380561 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TimeWindow.scala --- @@ -137,16 +139,42 @@ object TimeWindow {

[GitHub] spark issue #22325: [SPARK-25318]. Add exception handling when wrapping the ...

2018-09-25 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22325 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22547: [SPARK-25528][SQL] data source V2 read side API refactor...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22547 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96570/ Test PASSed. ---

[GitHub] spark issue #22547: [SPARK-25528][SQL] data source V2 read side API refactor...

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22547 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22547: [SPARK-25528][SQL] data source V2 read side API refactor...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22547 **[Test build #96570 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96570/testReport)** for PR 22547 at commit

[GitHub] spark issue #22362: [SPARK-25372][YARN][K8S] Deprecate and generalize keytab...

2018-09-25 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22362 YARN looks happy. LGTM pending tests. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #22549: [SPARK-25533][CORE][WEBUI]AppSummary should hold the inf...

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22549 **[Test build #96580 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96580/testReport)** for PR 22549 at commit

[GitHub] spark issue #22455: [SPARK-24572][SPARKR] "eager execution" for R shell, IDE

2018-09-25 Thread adrian555
Github user adrian555 commented on the issue: https://github.com/apache/spark/pull/22455 Another flavor will be this ```r print.SparkDataFrame <- function(x, ...) { if ((length(list(...)) > 0) && identical(sparkR.conf("spark.sql.repl.eagerEval.enabled", "false")[[1]],

[GitHub] spark issue #22549: [SPARK-25533][CORE][WEBUI]AppSummary should hold the inf...

2018-09-25 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22549 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22455: [SPARK-24572][SPARKR] "eager execution" for R shell, IDE

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22455 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96569/ Test PASSed. ---

[GitHub] spark issue #22455: [SPARK-24572][SPARKR] "eager execution" for R shell, IDE

2018-09-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22455 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22455: [SPARK-24572][SPARKR] "eager execution" for R shell, IDE

2018-09-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22455 **[Test build #96569 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96569/testReport)** for PR 22455 at commit

[GitHub] spark issue #22549: [SPARK-25533][CORE][WEBUI]AppSummary should hold the inf...

2018-09-25 Thread shahidki31
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/22549 cc @vanzin @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

<    1   2   3   4   5   6   7   >