[GitHub] [spark] maropu commented on a change in pull request #28830: [SPARK-31990][SQL] Preserves the input order of colNames in dropDuplicates

maropu commented on a change in pull request #28830: URL: https://github.com/apache/spark/pull/28830#discussion_r439892041 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala ## @@ -2548,6 +2548,21 @@ class DataFrameSuite extends QueryTest ass

[GitHub] [spark] maropu commented on pull request #28830: [SPARK-31990][SQL] Preserves the input order of colNames in dropDuplicates

maropu commented on pull request #28830: URL: https://github.com/apache/spark/pull/28830#issuecomment-643856341 cc: @xuanyuanking @HeartSaVioR @srowen @gatorsmile This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-643855961 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28830: [SPARK-31990][SQL] Preserves the input order of colNames in dropDuplicates

AmplabJenkins commented on pull request #28830: URL: https://github.com/apache/spark/pull/28830#issuecomment-643855929 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

AmplabJenkins commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-643855961 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28830: [SPARK-31990][SQL] Preserves the input order of colNames in dropDuplicates

AmplabJenkins removed a comment on pull request #28830: URL: https://github.com/apache/spark/pull/28830#issuecomment-643855929 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

SparkQA commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-643855623 **[Test build #124021 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124021/testReport)** for PR 28710 at commit [`2e6f35c`](https://github.com

[GitHub] [spark] maropu commented on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

maropu commented on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643855656 Probably, I think its better to assign this refactoring to a new JIRA id. This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #28830: [SPARK-31990][SQL] Preserves the input order of colNames in dropDuplicates

SparkQA commented on pull request #28830: URL: https://github.com/apache/spark/pull/28830#issuecomment-643855612 **[Test build #124020 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124020/testReport)** for PR 28830 at commit [`3e02ad6`](https://github.com

[GitHub] [spark] maropu commented on a change in pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

maropu commented on a change in pull request #28825: URL: https://github.com/apache/spark/pull/28825#discussion_r439890770 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala ## @@ -135,8 +134,9 @@ class SQLQueryTestSuite extends QueryTest with S

[GitHub] [spark] dongjoon-hyun commented on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

dongjoon-hyun commented on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-643855155 The situation is the same. There are OOM errors. ``` java.lang.OutOfMemoryError: OOM while handling another exception at org.apache.spark.executor.FetchFailur

[GitHub] [spark] maropu opened a new pull request #28830: [SPARK-31990][SQL] Preserves the input order of colNames in dropDuplicates

maropu opened a new pull request #28830: URL: https://github.com/apache/spark/pull/28830 ### What changes were proposed in this pull request? This PR intends to provide a hot-fix for a bug in `Dataset.dropDuplicates`; we must preserve the input order of `colNames` for `groupC

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

dongjoon-hyun edited a comment on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-643855155 The situation is the same. There are OOM errors. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.4-test-maven-had

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata

AmplabJenkins removed a comment on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-643854260 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

HyukjinKwon commented on a change in pull request #28825: URL: https://github.com/apache/spark/pull/28825#discussion_r439889900 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala ## @@ -135,8 +134,9 @@ class SQLQueryTestSuite extends QueryTest wi

[GitHub] [spark] AmplabJenkins commented on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata

AmplabJenkins commented on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-643854260 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata

AmplabJenkins removed a comment on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-643853892 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata

SparkQA commented on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-643854058 **[Test build #124019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124019/testReport)** for PR 28826 at commit [`0c46105`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata

AmplabJenkins commented on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-643853892 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA removed a comment on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata

SparkQA removed a comment on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-643826770 **[Test build #124014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124014/testReport)** for PR 28616 at commit [`4fd056d`](https://gi

[GitHub] [spark] dongjoon-hyun commented on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

dongjoon-hyun commented on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-643853361 Unfortunately, this seems to break all `branch-2.4` Jenkins jobs again. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.

[GitHub] [spark] SparkQA commented on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata

SparkQA commented on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-643853412 **[Test build #124014 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124014/testReport)** for PR 28616 at commit [`4fd056d`](https://github.co

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata

AmplabJenkins removed a comment on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-643786098 Can one of the admins verify this patch? This is an automated message from the Apache Git Service.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28786: [SPARK-31925][ML] Summary.totalIterations greater than maxIters

AmplabJenkins removed a comment on pull request #28786: URL: https://github.com/apache/spark/pull/28786#issuecomment-643852845 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28786: [SPARK-31925][ML] Summary.totalIterations greater than maxIters

AmplabJenkins commented on pull request #28786: URL: https://github.com/apache/spark/pull/28786#issuecomment-643852845 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] HyukjinKwon commented on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata

HyukjinKwon commented on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-643852790 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [spark] SparkQA commented on pull request #28786: [SPARK-31925][ML] Summary.totalIterations greater than maxIters

SparkQA commented on pull request #28786: URL: https://github.com/apache/spark/pull/28786#issuecomment-643852583 **[Test build #124018 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124018/testReport)** for PR 28786 at commit [`4c4d52b`](https://github.com

[GitHub] [spark] github-actions[bot] closed pull request #27053: [WIP][SPARK-27495][Core][YARN][k8s] Stage Level Scheduling code for reference

github-actions[bot] closed pull request #27053: URL: https://github.com/apache/spark/pull/27053 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[GitHub] [spark] github-actions[bot] closed pull request #27375: [SPARK-30664][Web UI] Add optional metrics to all-stages page

github-actions[bot] closed pull request #27375: URL: https://github.com/apache/spark/pull/27375 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

AmplabJenkins removed a comment on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643840923 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

AmplabJenkins commented on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643840923 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

SparkQA commented on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643840558 **[Test build #124011 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124011/testReport)** for PR 28825 at commit [`248d478`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

SparkQA removed a comment on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643809409 **[Test build #124011 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124011/testReport)** for PR 28825 at commit [`248d478`](https://gi

[GitHub] [spark] maropu commented on pull request #28826: [SPARK-31988] - schema pruning may discard attribute metadata

maropu commented on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-643837728 Could you add some tests first? Thanks! This is an automated message from the Apache Git Service. To respond to t

[GitHub] [spark] maropu commented on a change in pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

maropu commented on a change in pull request #28825: URL: https://github.com/apache/spark/pull/28825#discussion_r439878832 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala ## @@ -135,8 +134,9 @@ class SQLQueryTestSuite extends QueryTest with S

[GitHub] [spark] maropu commented on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

maropu commented on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643837389 For log readability, could you truncate the error messages in the PR description? Also, please add output messages after this PR applied? ---

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643830992 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643830992 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

SparkQA commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643830855 **[Test build #124017 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124017/testReport)** for PR 28828 at commit [`a712b4d`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28829: [WIP][SQL] Benchmark the EXCEPTION rebase mode

AmplabJenkins removed a comment on pull request #28829: URL: https://github.com/apache/spark/pull/28829#issuecomment-643828984 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28829: [WIP][SQL] Benchmark the EXCEPTION rebase mode

AmplabJenkins commented on pull request #28829: URL: https://github.com/apache/spark/pull/28829#issuecomment-643828984 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28829: [WIP][SQL] Benchmark the EXCEPTION rebase mode

SparkQA commented on pull request #28829: URL: https://github.com/apache/spark/pull/28829#issuecomment-643828884 **[Test build #124016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124016/testReport)** for PR 28829 at commit [`16e90be`](https://github.com

[GitHub] [spark] MaxGekk opened a new pull request #28829: [WIP][SQL] Benchmark the EXCEPTION rebase mode

MaxGekk opened a new pull request #28829: URL: https://github.com/apache/spark/pull/28829 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643827609 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643827609 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

SparkQA commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643827509 **[Test build #124015 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124015/testReport)** for PR 28828 at commit [`ca3b3de`](https://github.com

[GitHub] [spark] dongjoon-hyun closed pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

dongjoon-hyun closed pull request #26901: URL: https://github.com/apache/spark/pull/26901 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] SparkQA commented on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata

SparkQA commented on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-643826770 **[Test build #124014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124014/testReport)** for PR 28616 at commit [`4fd056d`](https://github.com

[GitHub] [spark] srowen commented on pull request #28786: [SPARK-31925][ML] Summary.totalIterations greater than maxIters

srowen commented on pull request #28786: URL: https://github.com/apache/spark/pull/28786#issuecomment-643826632 @zhengruifeng if you don't strongly object to https://github.com/apache/spark/pull/28786#discussion_r439608878 I think this one can be merged. Just needs a rebase at your

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826208 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124

[GitHub] [spark] dongjoon-hyun commented on pull request #28787: [SPARK-31959][SQL][test-java11] Fix Gregorian-Julian micros rebasing while switching standard time zone offset

dongjoon-hyun commented on pull request #28787: URL: https://github.com/apache/spark/pull/28787#issuecomment-643826389 Hi, All. As I commented on the backporting PR, this causes a weird consistent failure on `research-jenkins-worker-09`. Could you take a look at that? - https://github.c

[GitHub] [spark] SparkQA removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

SparkQA removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826003 **[Test build #124013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124013/testReport)** for PR 28828 at commit [`cf147aa`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826206 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826206 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826119 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

SparkQA commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826204 **[Test build #124013 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124013/testReport)** for PR 28828 at commit [`cf147aa`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826119 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] srowen commented on pull request #28752: [SPARK-31642][FOLLOWUP] Fix Sorting for duration column and make Status column sortable

srowen commented on pull request #28752: URL: https://github.com/apache/spark/pull/28752#issuecomment-643826026 Oh I see, this was attached to the JIRA for pagination. It does make sense to have a separate JIRA for _that_. Let me retroactively fix up the links. ---

[GitHub] [spark] SparkQA commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

SparkQA commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826003 **[Test build #124013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124013/testReport)** for PR 28828 at commit [`cf147aa`](https://github.com

[GitHub] [spark] srowen commented on pull request #28752: [SPARK-31642][FOLLOWUP] Fix Sorting for duration column and make Status column sortable

srowen commented on pull request #28752: URL: https://github.com/apache/spark/pull/28752#issuecomment-643825958 Merged to master. I'll look at the backport. This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] srowen closed pull request #28752: [SPARK-31642][FOLLOWUP] Fix Sorting for duration column and make Status column sortable

srowen closed pull request #28752: URL: https://github.com/apache/spark/pull/28752 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [spark] HeartSaVioR commented on a change in pull request #28607: [SPARK-24634][SS] Add a new metric regarding number of inputs later than watermark plus allowed delay

HeartSaVioR commented on a change in pull request #28607: URL: https://github.com/apache/spark/pull/28607#discussion_r439870225 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala ## @@ -77,6 +77,8 @@ trait StateStoreWriter ex

[GitHub] [spark] HeartSaVioR commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

HeartSaVioR commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643825765 cc. @zsxwing This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [spark] HeartSaVioR opened a new pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

HeartSaVioR opened a new pull request #28828: URL: https://github.com/apache/spark/pull/28828 ### What changes were proposed in this pull request? This PR renames the variable from "numLateInputs" to "numDropppedRowsByWatermark" so that it becomes self-explanation. ### Why are

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28809: [SPARK-31959][SQL][3.0] Fix Gregorian-Julian micros rebasing while switching standard time zone offset

dongjoon-hyun commented on a change in pull request #28809: URL: https://github.com/apache/spark/pull/28809#discussion_r439869464 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/RebaseDateTimeSuite.scala ## @@ -409,4 +409,40 @@ class RebaseDateTime

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28809: [SPARK-31959][SQL][3.0] Fix Gregorian-Julian micros rebasing while switching standard time zone offset

dongjoon-hyun commented on a change in pull request #28809: URL: https://github.com/apache/spark/pull/28809#discussion_r439869464 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/RebaseDateTimeSuite.scala ## @@ -409,4 +409,40 @@ class RebaseDateTime

[GitHub] [spark] dongjoon-hyun closed pull request #28801: [SPARK-31970][CORE] Make MDC configuration step be consistent between setLocalProperty and log4j.properties

dongjoon-hyun closed pull request #28801: URL: https://github.com/apache/spark/pull/28801 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] SparkQA commented on pull request #27066: [SPARK-31317][SQL] Add withField method to Column

SparkQA commented on pull request #27066: URL: https://github.com/apache/spark/pull/27066#issuecomment-643823823 **[Test build #124012 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124012/testReport)** for PR 27066 at commit [`a54d3fb`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing

AmplabJenkins removed a comment on pull request #24525: URL: https://github.com/apache/spark/pull/24525#issuecomment-643818140 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing

AmplabJenkins commented on pull request #24525: URL: https://github.com/apache/spark/pull/24525#issuecomment-643818140 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA removed a comment on pull request #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing

SparkQA removed a comment on pull request #24525: URL: https://github.com/apache/spark/pull/24525#issuecomment-643786010 **[Test build #124009 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124009/testReport)** for PR 24525 at commit [`fb1d3f3`](https://gi

[GitHub] [spark] SparkQA commented on pull request #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing

SparkQA commented on pull request #24525: URL: https://github.com/apache/spark/pull/24525#issuecomment-643817928 **[Test build #124009 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124009/testReport)** for PR 24525 at commit [`fb1d3f3`](https://github.co

[GitHub] [spark] iRakson commented on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

iRakson commented on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-643816792 #28254 has been closed. This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [spark] iRakson closed pull request #28254: [SPARK-31478][CORE]Call `StopExecutor` before killing executors

iRakson closed pull request #28254: URL: https://github.com/apache/spark/pull/28254 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] iRakson commented on pull request #28254: [SPARK-31478][CORE]Call `StopExecutor` before killing executors

iRakson commented on pull request #28254: URL: https://github.com/apache/spark/pull/28254#issuecomment-643816527 For the time being we are continuing with the current approach only. Closing this PR. This is an automated mess

[GitHub] [spark] iRakson edited a comment on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

iRakson edited a comment on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-642632099 @dongjoon-hyun @vanzin I have tested this a couple of times, older approach of adding shutdownhook is working fine. Last failure is unrelated to this. Also about the oth

[GitHub] [spark] iRakson commented on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

iRakson commented on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-643815823 > Hi, @iRakson . So, this is your final decision and are you going to close #28254 ? > This PR looks like to be reverted into the original patch. In that case, we don't need

[GitHub] [spark] dongjoon-hyun commented on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

dongjoon-hyun commented on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-643814681 Hi, @iRakson . So, this is your final decision and are you going to close #28254 ? This PR looks like to be reverted into the original patch. In that case, we don't nee

[GitHub] [spark] HeartSaVioR commented on a change in pull request #28607: [SPARK-24634][SS] Add a new metric regarding number of inputs later than watermark plus allowed delay

HeartSaVioR commented on a change in pull request #28607: URL: https://github.com/apache/spark/pull/28607#discussion_r439861427 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala ## @@ -77,6 +77,8 @@ trait StateStoreWriter ex

[GitHub] [spark] iRakson commented on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

iRakson commented on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-643811881 @dongjoon-hyun @vanzin Can we get this in 2.4 ? I think now this is fine. It should not break the code. I have already tested this multiple times. ---

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

AmplabJenkins removed a comment on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643809567 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

AmplabJenkins commented on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643809567 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

SparkQA commented on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643809409 **[Test build #124011 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124011/testReport)** for PR 28825 at commit [`248d478`](https://github.com

[GitHub] [spark] agrawaldevesh commented on a change in pull request #28817: [WIP][SPARK-31197][CORE] Exit the executor once all tasks and migrations are finished built on top of on top of spark20629

agrawaldevesh commented on a change in pull request #28817: URL: https://github.com/apache/spark/pull/28817#discussion_r439859171 ## File path: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala ## @@ -258,26 +262,60 @@ private[spark] class Coarse

[GitHub] [spark] holdenk commented on a change in pull request #28817: [WIP][SPARK-31197][CORE] Exit the executor once all tasks and migrations are finished built on top of on top of spark20629

holdenk commented on a change in pull request #28817: URL: https://github.com/apache/spark/pull/28817#discussion_r439857534 ## File path: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala ## @@ -258,26 +262,60 @@ private[spark] class CoarseGraine

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

AmplabJenkins removed a comment on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643806345 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124

[GitHub] [spark] SparkQA removed a comment on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

SparkQA removed a comment on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643804463 **[Test build #124010 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124010/testReport)** for PR 28827 at commit [`900fe2c`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

AmplabJenkins removed a comment on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643806342 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins commented on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

AmplabJenkins commented on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643806342 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

SparkQA commented on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643806320 **[Test build #124010 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124010/testReport)** for PR 28827 at commit [`900fe2c`](https://github.co

[GitHub] [spark] MaxGekk commented on pull request #28816: [SPARK-31986][SQL] Fix Julian-Gregorian micros rebasing of overlapping local timestamps

MaxGekk commented on pull request #28816: URL: https://github.com/apache/spark/pull/28816#issuecomment-643804818 @cloud-fan @HyukjinKwon Please, review the fix. This is an automated message from the Apache Git Service. To res

[GitHub] [spark] MaxGekk commented on pull request #28824: [SPARK-31984][SQL] Make micros rebasing functions via local timestamps pure

MaxGekk commented on pull request #28824: URL: https://github.com/apache/spark/pull/28824#issuecomment-643804694 @cloud-fan @HyukjinKwon Please, review this PR. This is an automated message from the Apache Git Service. To res

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

AmplabJenkins removed a comment on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643804601 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

AmplabJenkins commented on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643804601 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] MaxGekk commented on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

MaxGekk commented on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643804614 @cloud-fan @HyukjinKwon After merging this bug fix https://github.com/apache/spark/pull/28816 and make the rebase functions pure by https://github.com/apache/spark/pull/28824, t

[GitHub] [spark] SparkQA commented on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

SparkQA commented on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643804463 **[Test build #124010 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124010/testReport)** for PR 28827 at commit [`900fe2c`](https://github.com

[GitHub] [spark] MaxGekk opened a new pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

MaxGekk opened a new pull request #28827: URL: https://github.com/apache/spark/pull/28827 ### What changes were proposed in this pull request? 1. Change the max step from 1 week to 30 minutes in the tests `RebaseDateTimeSuite`.`generate 'gregorian-julian-rebase-micros.json'` and `genera

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

AmplabJenkins removed a comment on pull request #28685: URL: https://github.com/apache/spark/pull/28685#issuecomment-643800833 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

AmplabJenkins commented on pull request #28685: URL: https://github.com/apache/spark/pull/28685#issuecomment-643800833 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA removed a comment on pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

SparkQA removed a comment on pull request #28685: URL: https://github.com/apache/spark/pull/28685#issuecomment-643767795 **[Test build #124008 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124008/testReport)** for PR 28685 at commit [`8a8ec04`](https://gi

<    1   2   3   4   5   >