[GitHub] [spark] AmplabJenkins removed a comment on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

AmplabJenkins removed a comment on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-632017229 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

AmplabJenkins commented on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-632017229 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] sarutak commented on a change in pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0

sarutak commented on a change in pull request #28585: URL: https://github.com/apache/spark/pull/28585#discussion_r428578312 ## File path: core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala ## @@ -784,6 +784,7 @@ class UISeleniumSuite extends SparkFunSuite with WebBro

[GitHub] [spark] SparkQA commented on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

SparkQA commented on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-632016717 **[Test build #122927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122927/testReport)** for PR 28583 at commit [`53ebb55`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'

AmplabJenkins commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632013979 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'

AmplabJenkins removed a comment on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632013979 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'

SparkQA commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632013240 **[Test build #122926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122926/testReport)** for PR 28592 at commit [`1d31ed2`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'

AmplabJenkins removed a comment on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632010389 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'

AmplabJenkins commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632010389 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28597: fix typo

AmplabJenkins removed a comment on pull request #28597: URL: https://github.com/apache/spark/pull/28597#issuecomment-632006364 Can one of the admins verify this patch? This is an automated message from the Apache Git Service.

[GitHub] [spark] AmplabJenkins commented on pull request #28597: fix typo

AmplabJenkins commented on pull request #28597: URL: https://github.com/apache/spark/pull/28597#issuecomment-632006901 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] AmplabJenkins commented on pull request #28597: fix typo

AmplabJenkins commented on pull request #28597: URL: https://github.com/apache/spark/pull/28597#issuecomment-632006364 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

AmplabJenkins removed a comment on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-632004289 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

AmplabJenkins removed a comment on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-632004284 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA removed a comment on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

SparkQA removed a comment on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-631947436 **[Test build #122919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122919/testReport)** for PR 28583 at commit [`5851842`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

AmplabJenkins commented on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-632004284 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] 1040979575 opened a new pull request #28597: fix typo

1040979575 opened a new pull request #28597: URL: https://github.com/apache/spark/pull/28597 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

[GitHub] [spark] SparkQA commented on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

SparkQA commented on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-632003894 **[Test build #122919 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122919/testReport)** for PR 28583 at commit [`5851842`](https://github.co

[GitHub] [spark] akshatb1 commented on pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode

akshatb1 commented on pull request #28258: URL: https://github.com/apache/spark/pull/28258#issuecomment-632003872 @Ngone51 @jiangxb1987 Gentle ping, I have addressed your comments. Please review when you get a chance. Thanks. ---

[GitHub] [spark] cfmcgrady commented on a change in pull request #23900: [SPARK-23836][PYTHON] Add support for StructType return in Scalar Pandas UDF

cfmcgrady commented on a change in pull request #23900: URL: https://github.com/apache/spark/pull/23900#discussion_r428563913 ## File path: python/pyspark/sql/types.py ## @@ -1616,6 +1616,12 @@ def to_arrow_type(dt): if type(dt.elementType) == TimestampType:

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

AmplabJenkins removed a comment on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-632000589 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

AmplabJenkins removed a comment on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-632000580 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA removed a comment on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

SparkQA removed a comment on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-631999704 **[Test build #122925 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122925/testReport)** for PR 28525 at commit [`09c498c`](https://gi

[GitHub] [spark] SparkQA commented on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

SparkQA commented on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-632000568 **[Test build #122925 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122925/testReport)** for PR 28525 at commit [`09c498c`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

AmplabJenkins commented on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-632000580 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

SparkQA commented on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-631999704 **[Test build #122925 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122925/testReport)** for PR 28525 at commit [`09c498c`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

AmplabJenkins removed a comment on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-631997557 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122

[GitHub] [spark] SparkQA removed a comment on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

SparkQA removed a comment on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-631996474 **[Test build #122924 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122924/testReport)** for PR 28525 at commit [`73727c4`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

AmplabJenkins removed a comment on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-631997545 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins commented on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

AmplabJenkins commented on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-631997545 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

SparkQA commented on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-631997529 **[Test build #122924 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122924/testReport)** for PR 28525 at commit [`73727c4`](https://github.co

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

AmplabJenkins removed a comment on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-631996898 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] turboFei commented on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

turboFei commented on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-631996915 Thanks for the review. I have modified the solution and save the digests into independent file. This is an aut

[GitHub] [spark] AmplabJenkins commented on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

AmplabJenkins commented on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-631996898 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] HyukjinKwon commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test

HyukjinKwon commented on pull request #28578: URL: https://github.com/apache/spark/pull/28578#issuecomment-631996385 cc @gengliangwang FYI. Looks fine to me too This is an automated message from the Apache Git Service. To res

[GitHub] [spark] HyukjinKwon commented on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0

HyukjinKwon commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-631996492 cc @gengliangwang too FYI. seems okay to me too This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

SparkQA commented on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-631996474 **[Test build #122924 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122924/testReport)** for PR 28525 at commit [`73727c4`](https://github.com

[GitHub] [spark] turboFei commented on a change in pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

turboFei commented on a change in pull request #28525: URL: https://github.com/apache/spark/pull/28525#discussion_r428555017 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ShuffleIndexInformation.java ## @@ -31,17 +34,44 @@ public class Sh

[GitHub] [spark] TJX2014 commented on a change in pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions

TJX2014 commented on a change in pull request #28534: URL: https://github.com/apache/spark/pull/28534#discussion_r428553501 ## File path: sql/core/src/test/resources/sql-tests/results/ansi/datetime.sql.out ## @@ -1,5 +1,65 @@ -- Automatically generated by SQLQueryTestSuite ---

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string

AmplabJenkins removed a comment on pull request #28576: URL: https://github.com/apache/spark/pull/28576#issuecomment-631990533 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string

AmplabJenkins commented on pull request #28576: URL: https://github.com/apache/spark/pull/28576#issuecomment-631990533 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string

SparkQA commented on pull request #28576: URL: https://github.com/apache/spark/pull/28576#issuecomment-631989880 **[Test build #122923 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122923/testReport)** for PR 28576 at commit [`8de3bda`](https://github.com

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28593: [SPARK-31710][SQL] Add two compatibility flag to cast long to timestamp

HyukjinKwon commented on a change in pull request #28593: URL: https://github.com/apache/spark/pull/28593#discussion_r428543986 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2586,6 +2586,22 @@ object SQLConf { .checkValue(_

[GitHub] [spark] GuoPhilipse commented on a change in pull request #28593: [SPARK-31710][SQL] Add two compatibility flag to cast long to timestamp

GuoPhilipse commented on a change in pull request #28593: URL: https://github.com/apache/spark/pull/28593#discussion_r428529561 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2586,6 +2586,22 @@ object SQLConf { .checkValue(_

[GitHub] [spark] SparkQA commented on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

SparkQA commented on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631965191 **[Test build #122922 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122922/testReport)** for PR 28556 at commit [`db601df`](https://github.com

[GitHub] [spark] zhengruifeng commented on pull request #28595: [SPARK-31781][ML][PySpark] Move param k (number of clusters) to shared params

zhengruifeng commented on pull request #28595: URL: https://github.com/apache/spark/pull/28595#issuecomment-631964130 Param `k` is also used in `RankingEvaluator` and `ALS`, is it reasonable to reuse this shared param in them? --

[GitHub] [spark] zhengruifeng commented on pull request #28590: [SPARK-31768][ML] add getMetrics in Evaluators

zhengruifeng commented on pull request #28590: URL: https://github.com/apache/spark/pull/28590#issuecomment-631962737 I do not object this PR. Is there any way to recompute another metric in evaluator without involving mllib.metrics? ---

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

AmplabJenkins removed a comment on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631962223 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] viirya commented on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

viirya commented on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631962148 @maropu I addressed your comments. Could you help take another look? Thanks. This is an automated message from th

[GitHub] [spark] AmplabJenkins commented on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

AmplabJenkins commented on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631962223 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] viirya commented on a change in pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

viirya commented on a change in pull request #28556: URL: https://github.com/apache/spark/pull/28556#discussion_r428520430 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala ## @@ -82,6 +87,8 @@ object NestedColumnAlias

[GitHub] [spark] sarutak commented on a change in pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test

sarutak commented on a change in pull request #28578: URL: https://github.com/apache/spark/pull/28578#discussion_r427076227 ## File path: dev/run-tests.py ## @@ -606,6 +609,8 @@ def main(): print("[info] Found the following changed modules:", ", ".join(x.name fo

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28596: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"

AmplabJenkins removed a comment on pull request #28596: URL: https://github.com/apache/spark/pull/28596#issuecomment-631955788 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions

AmplabJenkins removed a comment on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631955875 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions

AmplabJenkins commented on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631955875 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #28596: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"

AmplabJenkins commented on pull request #28596: URL: https://github.com/apache/spark/pull/28596#issuecomment-631955788 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions

SparkQA commented on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631955288 **[Test build #122921 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122921/testReport)** for PR 28534 at commit [`c1257f2`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #28596: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"

SparkQA commented on pull request #28596: URL: https://github.com/apache/spark/pull/28596#issuecomment-631955215 **[Test build #122920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122920/testReport)** for PR 28596 at commit [`2be2ca7`](https://github.com

[GitHub] [spark] Ngone51 commented on pull request #28584: [SPARK-31730][CORE][TEST] Fix flaky tests in BarrierTaskContextSuite

Ngone51 commented on pull request #28584: URL: https://github.com/apache/spark/pull/28584#issuecomment-631954552 Opened a separate PR https://github.com/apache/spark/pull/28596 to fix the test @WeichenXu123 This is an autom

[GitHub] [spark] TJX2014 commented on a change in pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions

TJX2014 commented on a change in pull request #28534: URL: https://github.com/apache/spark/pull/28534#discussion_r428512148 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -401,6 +401,78 @@ case class DayOfYea

[GitHub] [spark] Ngone51 commented on pull request #28596: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"

Ngone51 commented on pull request #28596: URL: https://github.com/apache/spark/pull/28596#issuecomment-631953628 ping @sarthfrey @WeichenXu123 @jiangxb1987 This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] TJX2014 commented on a change in pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions

TJX2014 commented on a change in pull request #28534: URL: https://github.com/apache/spark/pull/28534#discussion_r428512148 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -401,6 +401,78 @@ case class DayOfYea

[GitHub] [spark] Ngone51 opened a new pull request #28596: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"

Ngone51 opened a new pull request #28596: URL: https://github.com/apache/spark/pull/28596 ### What changes were proposed in this pull request? Change from `messages.toList.iterator` to `Iterator.single(messages.toList)`. ### Why are the changes needed? In

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

AmplabJenkins removed a comment on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-631948059 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

AmplabJenkins commented on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-631948059 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

SparkQA commented on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-631947436 **[Test build #122919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122919/testReport)** for PR 28583 at commit [`5851842`](https://github.com

[GitHub] [spark] sarutak commented on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

sarutak commented on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-63194 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] sarutak commented on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier

sarutak commented on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-631944356 Test seems to be aborted. This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [spark] turboFei commented on a change in pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

turboFei commented on a change in pull request #28525: URL: https://github.com/apache/spark/pull/28525#discussion_r428495266 ## File path: common/network-common/src/main/java/org/apache/spark/network/util/DigestUtils.java ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] cloud-fan commented on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions

cloud-fan commented on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631935225 LGTM except https://github.com/apache/spark/pull/28534/files#r427952417 This is an automated message from the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions

AmplabJenkins removed a comment on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631931016 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions

AmplabJenkins commented on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631931016 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] turboFei commented on a change in pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data

turboFei commented on a change in pull request #28525: URL: https://github.com/apache/spark/pull/28525#discussion_r428490170 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -626,16 +628,61 @@ final class ShuffleBlockFetcherIte

[GitHub] [spark] SparkQA commented on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions

SparkQA commented on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631930528 **[Test build #122918 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122918/testReport)** for PR 28534 at commit [`e4db9a5`](https://github.com

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28593: [SPARK-31710][SQL] Add two compatibility flag to cast long to timestamp

HyukjinKwon commented on a change in pull request #28593: URL: https://github.com/apache/spark/pull/28593#discussion_r428488834 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2586,6 +2586,22 @@ object SQLConf { .checkValue(_

[GitHub] [spark] AmplabJenkins commented on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

AmplabJenkins commented on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631924505 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

AmplabJenkins removed a comment on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631924505 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

SparkQA commented on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631923918 **[Test build #122917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122917/testReport)** for PR 28556 at commit [`b77a1ba`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLISECONDS and TIMESTAMP_MICROSECONDS functions

AmplabJenkins removed a comment on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631922879 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

AmplabJenkins removed a comment on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631922741 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28595: [SPARK-31781][ML][PySpark] Move param k (number of clusters) to shared params

AmplabJenkins removed a comment on pull request #28595: URL: https://github.com/apache/spark/pull/28595#issuecomment-631922905 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122

[GitHub] [spark] viirya commented on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

viirya commented on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631923383 retest this please This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [spark] SparkQA removed a comment on pull request #28595: [SPARK-31781][ML][PySpark] Move param k (number of clusters) to shared params

SparkQA removed a comment on pull request #28595: URL: https://github.com/apache/spark/pull/28595#issuecomment-631892298 **[Test build #122914 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122914/testReport)** for PR 28595 at commit [`2d6e87b`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLISECONDS and TIMESTAMP_MICROSECONDS functions

AmplabJenkins removed a comment on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631922744 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28595: [SPARK-31781][ML][PySpark] Move param k (number of clusters) to shared params

AmplabJenkins removed a comment on pull request #28595: URL: https://github.com/apache/spark/pull/28595#issuecomment-631922899 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

AmplabJenkins removed a comment on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631922732 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA removed a comment on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

SparkQA removed a comment on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631915185 **[Test build #122916 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122916/testReport)** for PR 28556 at commit [`b77a1ba`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

AmplabJenkins commented on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631922732 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #28595: [SPARK-31781][ML][PySpark] Move param k (number of clusters) to shared params

AmplabJenkins commented on pull request #28595: URL: https://github.com/apache/spark/pull/28595#issuecomment-631922899 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA removed a comment on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLISECONDS and TIMESTAMP_MICROSECONDS functions

SparkQA removed a comment on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631892304 This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

[GitHub] [spark] AmplabJenkins commented on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLISECONDS and TIMESTAMP_MICROSECONDS functions

AmplabJenkins commented on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631922744 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28595: [SPARK-31781][ML][PySpark] Move param k (number of clusters) to shared params

SparkQA commented on pull request #28595: URL: https://github.com/apache/spark/pull/28595#issuecomment-631922678 **[Test build #122914 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122914/testReport)** for PR 28595 at commit [`2d6e87b`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLISECONDS and TIMESTAMP_MICROSECONDS functions

AmplabJenkins commented on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631922809 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLISECONDS and TIMESTAMP_MICROSECONDS functions

SparkQA commented on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631922675 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] SparkQA commented on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join

SparkQA commented on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631922680 **[Test build #122916 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122916/testReport)** for PR 28556 at commit [`b77a1ba`](https://github.co

[GitHub] [spark] GuoPhilipse commented on a change in pull request #28593: [SPARK-31710][SQL] Add two compatibility flag to cast long to timestamp

GuoPhilipse commented on a change in pull request #28593: URL: https://github.com/apache/spark/pull/28593#discussion_r428481765 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2586,6 +2586,22 @@ object SQLConf { .checkValue(_

<    1   2   3   4   5   6